Commit | Line | Data |
---|---|---|
73d98127 CD |
1 | ============= |
2 | Event Tracing | |
3 | ============= | |
abd41443 | 4 | |
73d98127 CD |
5 | :Author: Theodore Ts'o |
6 | :Updated: Li Zefan and Tom Zanussi | |
abd41443 | 7 | |
143c145e LZ |
8 | 1. Introduction |
9 | =============== | |
abd41443 | 10 | |
ec15872d | 11 | Tracepoints (see Documentation/trace/tracepoints.rst) can be used |
abd41443 TT |
12 | without creating custom kernel modules to register probe functions |
13 | using the event tracing infrastructure. | |
14 | ||
15 | Not all tracepoints can be traced using the event tracing system; | |
16 | the kernel developer must provide code snippets which define how the | |
17 | tracing information is saved into the tracing buffer, and how the | |
143c145e | 18 | tracing information should be printed. |
abd41443 | 19 | |
143c145e LZ |
20 | 2. Using Event Tracing |
21 | ====================== | |
22 | ||
23 | 2.1 Via the 'set_event' interface | |
24 | --------------------------------- | |
abd41443 TT |
25 | |
26 | The events which are available for tracing can be found in the file | |
2abfcd29 | 27 | /sys/kernel/tracing/available_events. |
abd41443 TT |
28 | |
29 | To enable a particular event, such as 'sched_wakeup', simply echo it | |
2abfcd29 | 30 | to /sys/kernel/tracing/set_event. For example:: |
abd41443 | 31 | |
2abfcd29 | 32 | # echo sched_wakeup >> /sys/kernel/tracing/set_event |
abd41443 | 33 | |
73d98127 | 34 | .. Note:: '>>' is necessary, otherwise it will firstly disable all the events. |
abd41443 TT |
35 | |
36 | To disable an event, echo the event name to the set_event file prefixed | |
73d98127 | 37 | with an exclamation point:: |
abd41443 | 38 | |
2abfcd29 | 39 | # echo '!sched_wakeup' >> /sys/kernel/tracing/set_event |
143c145e | 40 | |
73d98127 | 41 | To disable all events, echo an empty line to the set_event file:: |
143c145e | 42 | |
2abfcd29 | 43 | # echo > /sys/kernel/tracing/set_event |
abd41443 | 44 | |
6234c7bd | 45 | To enable all events, echo ``*:*`` or ``*:`` to the set_event file:: |
abd41443 | 46 | |
2abfcd29 | 47 | # echo *:* > /sys/kernel/tracing/set_event |
abd41443 TT |
48 | |
49 | The events are organized into subsystems, such as ext4, irq, sched, | |
50 | etc., and a full event name looks like this: <subsystem>:<event>. The | |
51 | subsystem name is optional, but it is displayed in the available_events | |
52 | file. All of the events in a subsystem can be specified via the syntax | |
6234c7bd | 53 | ``<subsystem>:*``; for example, to enable all irq events, you can use the |
73d98127 | 54 | command:: |
abd41443 | 55 | |
2abfcd29 | 56 | # echo 'irq:*' > /sys/kernel/tracing/set_event |
143c145e LZ |
57 | |
58 | 2.2 Via the 'enable' toggle | |
59 | --------------------------- | |
60 | ||
2abfcd29 | 61 | The events available are also listed in /sys/kernel/tracing/events/ hierarchy |
143c145e LZ |
62 | of directories. |
63 | ||
73d98127 | 64 | To enable event 'sched_wakeup':: |
143c145e | 65 | |
2abfcd29 | 66 | # echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enable |
143c145e | 67 | |
73d98127 | 68 | To disable it:: |
143c145e | 69 | |
2abfcd29 | 70 | # echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enable |
143c145e | 71 | |
73d98127 | 72 | To enable all events in sched subsystem:: |
143c145e | 73 | |
2abfcd29 | 74 | # echo 1 > /sys/kernel/tracing/events/sched/enable |
143c145e | 75 | |
73d98127 | 76 | To enable all events:: |
143c145e | 77 | |
2abfcd29 | 78 | # echo 1 > /sys/kernel/tracing/events/enable |
143c145e LZ |
79 | |
80 | When reading one of these enable files, there are four results: | |
81 | ||
73d98127 CD |
82 | - 0 - all events this file affects are disabled |
83 | - 1 - all events this file affects are enabled | |
84 | - X - there is a mixture of events enabled and disabled | |
85 | - ? - this file does not affect any event | |
143c145e | 86 | |
020e5f85 LZ |
87 | 2.3 Boot option |
88 | --------------- | |
89 | ||
73d98127 | 90 | In order to facilitate early boot debugging, use boot option:: |
020e5f85 LZ |
91 | |
92 | trace_event=[event-list] | |
93 | ||
03d646e6 LZ |
94 | event-list is a comma separated list of events. See section 2.1 for event |
95 | format. | |
020e5f85 | 96 | |
143c145e LZ |
97 | 3. Defining an event-enabled tracepoint |
98 | ======================================= | |
99 | ||
100 | See The example provided in samples/trace_events | |
101 | ||
95b69608 TZ |
102 | 4. Event formats |
103 | ================ | |
104 | ||
105 | Each trace event has a 'format' file associated with it that contains | |
106 | a description of each field in a logged event. This information can | |
107 | be used to parse the binary trace stream, and is also the place to | |
108 | find the field names that can be used in event filters (see section 5). | |
109 | ||
110 | It also displays the format string that will be used to print the | |
111 | event in text mode, along with the event name and ID used for | |
112 | profiling. | |
113 | ||
6234c7bd JC |
114 | Every event has a set of ``common`` fields associated with it; these are |
115 | the fields prefixed with ``common_``. The other fields vary between | |
95b69608 TZ |
116 | events and correspond to the fields defined in the TRACE_EVENT |
117 | definition for that event. | |
118 | ||
73d98127 | 119 | Each field in the format has the form:: |
95b69608 TZ |
120 | |
121 | field:field-type field-name; offset:N; size:N; | |
122 | ||
123 | where offset is the offset of the field in the trace record and size | |
124 | is the size of the data item, in bytes. | |
125 | ||
126 | For example, here's the information displayed for the 'sched_wakeup' | |
73d98127 | 127 | event:: |
95b69608 | 128 | |
2abfcd29 | 129 | # cat /sys/kernel/tracing/events/sched/sched_wakeup/format |
95b69608 | 130 | |
73d98127 CD |
131 | name: sched_wakeup |
132 | ID: 60 | |
133 | format: | |
134 | field:unsigned short common_type; offset:0; size:2; | |
135 | field:unsigned char common_flags; offset:2; size:1; | |
136 | field:unsigned char common_preempt_count; offset:3; size:1; | |
137 | field:int common_pid; offset:4; size:4; | |
138 | field:int common_tgid; offset:8; size:4; | |
95b69608 | 139 | |
73d98127 CD |
140 | field:char comm[TASK_COMM_LEN]; offset:12; size:16; |
141 | field:pid_t pid; offset:28; size:4; | |
142 | field:int prio; offset:32; size:4; | |
143 | field:int success; offset:36; size:4; | |
144 | field:int cpu; offset:40; size:4; | |
95b69608 | 145 | |
73d98127 CD |
146 | print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid, |
147 | REC->prio, REC->success, REC->cpu | |
95b69608 TZ |
148 | |
149 | This event contains 10 fields, the first 5 common and the remaining 5 | |
150 | event-specific. All the fields for this event are numeric, except for | |
151 | 'comm' which is a string, a distinction important for event filtering. | |
152 | ||
153 | 5. Event filtering | |
154 | ================== | |
155 | ||
156 | Trace events can be filtered in the kernel by associating boolean | |
157 | 'filter expressions' with them. As soon as an event is logged into | |
158 | the trace buffer, its fields are checked against the filter expression | |
159 | associated with that event type. An event with field values that | |
160 | 'match' the filter will appear in the trace output, and an event whose | |
161 | values don't match will be discarded. An event with no filter | |
162 | associated with it matches everything, and is the default when no | |
163 | filter has been set for an event. | |
164 | ||
165 | 5.1 Expression syntax | |
166 | --------------------- | |
167 | ||
168 | A filter expression consists of one or more 'predicates' that can be | |
169 | combined using the logical operators '&&' and '||'. A predicate is | |
170 | simply a clause that compares the value of a field contained within a | |
171 | logged event with a constant value and returns either 0 or 1 depending | |
73d98127 | 172 | on whether the field value matched (1) or didn't match (0):: |
95b69608 TZ |
173 | |
174 | field-name relational-operator value | |
175 | ||
176 | Parentheses can be used to provide arbitrary logical groupings and | |
177 | double-quotes can be used to prevent the shell from interpreting | |
178 | operators as shell metacharacters. | |
179 | ||
180 | The field-names available for use in filters can be found in the | |
181 | 'format' files for trace events (see section 4). | |
182 | ||
183 | The relational-operators depend on the type of the field being tested: | |
184 | ||
185 | The operators available for numeric fields are: | |
186 | ||
1a891cf1 | 187 | ==, !=, <, <=, >, >=, & |
95b69608 TZ |
188 | |
189 | And for string fields they are: | |
190 | ||
c3e13c7c | 191 | ==, !=, ~ |
95b69608 | 192 | |
6234c7bd | 193 | The glob (~) accepts a wild card character (\*,?) and character classes |
73d98127 | 194 | ([). For example:: |
c3e13c7c SRRH |
195 | |
196 | prev_comm ~ "*sh" | |
197 | prev_comm ~ "sh*" | |
198 | prev_comm ~ "*sh*" | |
60f1d5e3 | 199 | prev_comm ~ "ba*sh" |
95b69608 | 200 | |
f37c3bbc SR |
201 | If the field is a pointer that points into user space (for example |
202 | "filename" from sys_enter_openat), then you have to append ".ustring" to the | |
203 | field name:: | |
204 | ||
205 | filename.ustring ~ "password" | |
206 | ||
207 | As the kernel will have to know how to retrieve the memory that the pointer | |
208 | is at from user space. | |
209 | ||
e6745a4d SRG |
210 | You can convert any long type to a function address and search by function name:: |
211 | ||
212 | call_site.function == security_prepare_creds | |
213 | ||
214 | The above will filter when the field "call_site" falls on the address within | |
215 | "security_prepare_creds". That is, it will compare the value of "call_site" and | |
216 | the filter will return true if it is greater than or equal to the start of | |
217 | the function "security_prepare_creds" and less than the end of that function. | |
218 | ||
219 | The ".function" postfix can only be attached to values of size long, and can only | |
220 | be compared with "==" or "!=". | |
221 | ||
fa828efb VS |
222 | Cpumask fields or scalar fields that encode a CPU number can be filtered using |
223 | a user-provided cpumask in cpulist format. The format is as follows:: | |
224 | ||
225 | CPUS{$cpulist} | |
226 | ||
227 | Operators available to cpumask filtering are: | |
228 | ||
229 | & (intersection), ==, != | |
230 | ||
231 | For example, this will filter events that have their .target_cpu field present | |
232 | in the given cpumask:: | |
233 | ||
234 | target_cpu & CPUS{17-42} | |
235 | ||
95b69608 TZ |
236 | 5.2 Setting filters |
237 | ------------------- | |
238 | ||
239 | A filter for an individual event is set by writing a filter expression | |
240 | to the 'filter' file for the given event. | |
241 | ||
73d98127 | 242 | For example:: |
95b69608 | 243 | |
2abfcd29 | 244 | # cd /sys/kernel/tracing/events/sched/sched_wakeup |
73d98127 | 245 | # echo "common_preempt_count > 4" > filter |
95b69608 | 246 | |
73d98127 | 247 | A slightly more involved example:: |
95b69608 | 248 | |
2abfcd29 | 249 | # cd /sys/kernel/tracing/events/signal/signal_generate |
73d98127 | 250 | # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter |
95b69608 TZ |
251 | |
252 | If there is an error in the expression, you'll get an 'Invalid | |
253 | argument' error when setting it, and the erroneous string along with | |
73d98127 | 254 | an error message can be seen by looking at the filter e.g.:: |
95b69608 | 255 | |
2abfcd29 | 256 | # cd /sys/kernel/tracing/events/signal/signal_generate |
73d98127 CD |
257 | # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter |
258 | -bash: echo: write error: Invalid argument | |
259 | # cat filter | |
260 | ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash | |
261 | ^ | |
262 | parse_error: Field not found | |
95b69608 TZ |
263 | |
264 | Currently the caret ('^') for an error always appears at the beginning of | |
265 | the filter string; the error message should still be useful though | |
266 | even without more accurate position info. | |
267 | ||
77360f9b SR |
268 | 5.2.1 Filter limitations |
269 | ------------------------ | |
270 | ||
271 | If a filter is placed on a string pointer ``(char *)`` that does not point | |
272 | to a string on the ring buffer, but instead points to kernel or user space | |
273 | memory, then, for safety reasons, at most 1024 bytes of the content is | |
274 | copied onto a temporary buffer to do the compare. If the copy of the memory | |
275 | faults (the pointer points to memory that should not be accessed), then the | |
276 | string compare will be treated as not matching. | |
277 | ||
95b69608 TZ |
278 | 5.3 Clearing filters |
279 | -------------------- | |
280 | ||
281 | To clear the filter for an event, write a '0' to the event's filter | |
282 | file. | |
283 | ||
284 | To clear the filters for all events in a subsystem, write a '0' to the | |
285 | subsystem's filter file. | |
286 | ||
71240f94 | 287 | 5.4 Subsystem filters |
95b69608 TZ |
288 | --------------------- |
289 | ||
290 | For convenience, filters for every event in a subsystem can be set or | |
291 | cleared as a group by writing a filter expression into the filter file | |
88393161 | 292 | at the root of the subsystem. Note however, that if a filter for any |
95b69608 TZ |
293 | event within the subsystem lacks a field specified in the subsystem |
294 | filter, or if the filter can't be applied for any other reason, the | |
295 | filter for that event will retain its previous setting. This can | |
296 | result in an unintended mixture of filters which could lead to | |
297 | confusing (to the user who might think different filters are in | |
298 | effect) trace output. Only filters that reference just the common | |
299 | fields can be guaranteed to propagate successfully to all events. | |
300 | ||
301 | Here are a few subsystem filter examples that also illustrate the | |
302 | above points: | |
303 | ||
73d98127 | 304 | Clear the filters on all events in the sched subsystem:: |
95b69608 | 305 | |
2abfcd29 | 306 | # cd /sys/kernel/tracing/events/sched |
73d98127 CD |
307 | # echo 0 > filter |
308 | # cat sched_switch/filter | |
309 | none | |
310 | # cat sched_wakeup/filter | |
311 | none | |
95b69608 TZ |
312 | |
313 | Set a filter using only common fields for all events in the sched | |
73d98127 | 314 | subsystem (all events end up with the same filter):: |
95b69608 | 315 | |
2abfcd29 | 316 | # cd /sys/kernel/tracing/events/sched |
73d98127 CD |
317 | # echo common_pid == 0 > filter |
318 | # cat sched_switch/filter | |
319 | common_pid == 0 | |
320 | # cat sched_wakeup/filter | |
321 | common_pid == 0 | |
95b69608 TZ |
322 | |
323 | Attempt to set a filter using a non-common field for all events in the | |
88393161 | 324 | sched subsystem (all events but those that have a prev_pid field retain |
73d98127 | 325 | their old filters):: |
95b69608 | 326 | |
2abfcd29 | 327 | # cd /sys/kernel/tracing/events/sched |
73d98127 CD |
328 | # echo prev_pid == 0 > filter |
329 | # cat sched_switch/filter | |
330 | prev_pid == 0 | |
331 | # cat sched_wakeup/filter | |
332 | common_pid == 0 | |
ac38fb85 | 333 | |
71240f94 | 334 | 5.5 PID filtering |
627645fd SRRH |
335 | ----------------- |
336 | ||
337 | The set_event_pid file in the same directory as the top events directory | |
338 | exists, will filter all events from tracing any task that does not have the | |
339 | PID listed in the set_event_pid file. | |
73d98127 | 340 | :: |
627645fd | 341 | |
2abfcd29 | 342 | # cd /sys/kernel/tracing |
73d98127 CD |
343 | # echo $$ > set_event_pid |
344 | # echo 1 > events/enable | |
627645fd SRRH |
345 | |
346 | Will only trace events for the current task. | |
347 | ||
348 | To add more PIDs without losing the PIDs already included, use '>>'. | |
73d98127 | 349 | :: |
627645fd | 350 | |
73d98127 | 351 | # echo 123 244 1 >> set_event_pid |
627645fd SRRH |
352 | |
353 | ||
ac38fb85 TZ |
354 | 6. Event triggers |
355 | ================= | |
356 | ||
357 | Trace events can be made to conditionally invoke trigger 'commands' | |
358 | which can take various forms and are described in detail below; | |
359 | examples would be enabling or disabling other trace events or invoking | |
360 | a stack trace whenever the trace event is hit. Whenever a trace event | |
361 | with attached triggers is invoked, the set of trigger commands | |
362 | associated with that event is invoked. Any given trigger can | |
363 | additionally have an event filter of the same form as described in | |
364 | section 5 (Event filtering) associated with it - the command will only | |
365 | be invoked if the event being invoked passes the associated filter. | |
366 | If no filter is associated with the trigger, it always passes. | |
367 | ||
368 | Triggers are added to and removed from a particular event by writing | |
369 | trigger expressions to the 'trigger' file for the given event. | |
370 | ||
371 | A given event can have any number of triggers associated with it, | |
372 | subject to any restrictions that individual commands may have in that | |
373 | regard. | |
374 | ||
375 | Event triggers are implemented on top of "soft" mode, which means that | |
376 | whenever a trace event has one or more triggers associated with it, | |
377 | the event is activated even if it isn't actually enabled, but is | |
378 | disabled in a "soft" mode. That is, the tracepoint will be called, | |
379 | but just will not be traced, unless of course it's actually enabled. | |
380 | This scheme allows triggers to be invoked even for events that aren't | |
381 | enabled, and also allows the current event filter implementation to be | |
382 | used for conditionally invoking triggers. | |
383 | ||
384 | The syntax for event triggers is roughly based on the syntax for | |
385 | set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands' | |
d3439f9d | 386 | section of Documentation/trace/ftrace.rst), but there are major |
ac38fb85 TZ |
387 | differences and the implementation isn't currently tied to it in any |
388 | way, so beware about making generalizations between the two. | |
389 | ||
8206de7d MCC |
390 | .. Note:: |
391 | Writing into trace_marker (See Documentation/trace/ftrace.rst) | |
d3439f9d SRV |
392 | can also enable triggers that are written into |
393 | /sys/kernel/tracing/events/ftrace/print/trigger | |
394 | ||
ac38fb85 TZ |
395 | 6.1 Expression syntax |
396 | --------------------- | |
397 | ||
73d98127 | 398 | Triggers are added by echoing the command to the 'trigger' file:: |
ac38fb85 TZ |
399 | |
400 | # echo 'command[:count] [if filter]' > trigger | |
401 | ||
402 | Triggers are removed by echoing the same command but starting with '!' | |
73d98127 | 403 | to the 'trigger' file:: |
ac38fb85 TZ |
404 | |
405 | # echo '!command[:count] [if filter]' > trigger | |
406 | ||
407 | The [if filter] part isn't used in matching commands when removing, so | |
408 | leaving that off in a '!' command will accomplish the same thing as | |
409 | having it in. | |
410 | ||
411 | The filter syntax is the same as that described in the 'Event | |
412 | filtering' section above. | |
413 | ||
414 | For ease of use, writing to the trigger file using '>' currently just | |
415 | adds or removes a single trigger and there's no explicit '>>' support | |
416 | ('>' actually behaves like '>>') or truncation support to remove all | |
417 | triggers (you have to use '!' for each one added.) | |
418 | ||
419 | 6.2 Supported trigger commands | |
420 | ------------------------------ | |
421 | ||
422 | The following commands are supported: | |
423 | ||
424 | - enable_event/disable_event | |
425 | ||
426 | These commands can enable or disable another trace event whenever | |
427 | the triggering event is hit. When these commands are registered, | |
428 | the other trace event is activated, but disabled in a "soft" mode. | |
429 | That is, the tracepoint will be called, but just will not be traced. | |
430 | The event tracepoint stays in this mode as long as there's a trigger | |
431 | in effect that can trigger it. | |
432 | ||
433 | For example, the following trigger causes kmalloc events to be | |
434 | traced when a read system call is entered, and the :1 at the end | |
73d98127 | 435 | specifies that this enablement happens only once:: |
ac38fb85 | 436 | |
73d98127 | 437 | # echo 'enable_event:kmem:kmalloc:1' > \ |
2abfcd29 | 438 | /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger |
ac38fb85 TZ |
439 | |
440 | The following trigger causes kmalloc events to stop being traced | |
441 | when a read system call exits. This disablement happens on every | |
73d98127 | 442 | read system call exit:: |
ac38fb85 | 443 | |
73d98127 | 444 | # echo 'disable_event:kmem:kmalloc' > \ |
2abfcd29 | 445 | /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger |
ac38fb85 | 446 | |
73d98127 | 447 | The format is:: |
ac38fb85 TZ |
448 | |
449 | enable_event:<system>:<event>[:count] | |
450 | disable_event:<system>:<event>[:count] | |
451 | ||
73d98127 | 452 | To remove the above commands:: |
ac38fb85 | 453 | |
73d98127 | 454 | # echo '!enable_event:kmem:kmalloc:1' > \ |
2abfcd29 | 455 | /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger |
ac38fb85 | 456 | |
73d98127 | 457 | # echo '!disable_event:kmem:kmalloc' > \ |
2abfcd29 | 458 | /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger |
ac38fb85 TZ |
459 | |
460 | Note that there can be any number of enable/disable_event triggers | |
461 | per triggering event, but there can only be one trigger per | |
462 | triggered event. e.g. sys_enter_read can have triggers enabling both | |
463 | kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc | |
464 | versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if | |
465 | bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they | |
466 | could be combined into a single filter on kmem:kmalloc though). | |
467 | ||
468 | - stacktrace | |
469 | ||
470 | This command dumps a stacktrace in the trace buffer whenever the | |
471 | triggering event occurs. | |
472 | ||
473 | For example, the following trigger dumps a stacktrace every time the | |
73d98127 | 474 | kmalloc tracepoint is hit:: |
ac38fb85 | 475 | |
73d98127 | 476 | # echo 'stacktrace' > \ |
2abfcd29 | 477 | /sys/kernel/tracing/events/kmem/kmalloc/trigger |
ac38fb85 TZ |
478 | |
479 | The following trigger dumps a stacktrace the first 5 times a kmalloc | |
73d98127 | 480 | request happens with a size >= 64K:: |
ac38fb85 | 481 | |
73d98127 | 482 | # echo 'stacktrace:5 if bytes_req >= 65536' > \ |
2abfcd29 | 483 | /sys/kernel/tracing/events/kmem/kmalloc/trigger |
ac38fb85 | 484 | |
73d98127 | 485 | The format is:: |
ac38fb85 TZ |
486 | |
487 | stacktrace[:count] | |
488 | ||
73d98127 | 489 | To remove the above commands:: |
ac38fb85 | 490 | |
73d98127 | 491 | # echo '!stacktrace' > \ |
2abfcd29 | 492 | /sys/kernel/tracing/events/kmem/kmalloc/trigger |
ac38fb85 | 493 | |
73d98127 | 494 | # echo '!stacktrace:5 if bytes_req >= 65536' > \ |
2abfcd29 | 495 | /sys/kernel/tracing/events/kmem/kmalloc/trigger |
ac38fb85 TZ |
496 | |
497 | The latter can also be removed more simply by the following (without | |
73d98127 | 498 | the filter):: |
ac38fb85 | 499 | |
73d98127 | 500 | # echo '!stacktrace:5' > \ |
2abfcd29 | 501 | /sys/kernel/tracing/events/kmem/kmalloc/trigger |
ac38fb85 TZ |
502 | |
503 | Note that there can be only one stacktrace trigger per triggering | |
504 | event. | |
505 | ||
506 | - snapshot | |
507 | ||
508 | This command causes a snapshot to be triggered whenever the | |
509 | triggering event occurs. | |
510 | ||
511 | The following command creates a snapshot every time a block request | |
512 | queue is unplugged with a depth > 1. If you were tracing a set of | |
513 | events or functions at the time, the snapshot trace buffer would | |
73d98127 | 514 | capture those events when the trigger event occurred:: |
ac38fb85 | 515 | |
73d98127 | 516 | # echo 'snapshot if nr_rq > 1' > \ |
2abfcd29 | 517 | /sys/kernel/tracing/events/block/block_unplug/trigger |
ac38fb85 | 518 | |
73d98127 | 519 | To only snapshot once:: |
ac38fb85 | 520 | |
73d98127 | 521 | # echo 'snapshot:1 if nr_rq > 1' > \ |
2abfcd29 | 522 | /sys/kernel/tracing/events/block/block_unplug/trigger |
ac38fb85 | 523 | |
73d98127 | 524 | To remove the above commands:: |
ac38fb85 | 525 | |
73d98127 | 526 | # echo '!snapshot if nr_rq > 1' > \ |
2abfcd29 | 527 | /sys/kernel/tracing/events/block/block_unplug/trigger |
ac38fb85 | 528 | |
73d98127 | 529 | # echo '!snapshot:1 if nr_rq > 1' > \ |
2abfcd29 | 530 | /sys/kernel/tracing/events/block/block_unplug/trigger |
ac38fb85 TZ |
531 | |
532 | Note that there can be only one snapshot trigger per triggering | |
533 | event. | |
534 | ||
535 | - traceon/traceoff | |
536 | ||
537 | These commands turn tracing on and off when the specified events are | |
538 | hit. The parameter determines how many times the tracing system is | |
539 | turned on and off. If unspecified, there is no limit. | |
540 | ||
541 | The following command turns tracing off the first time a block | |
542 | request queue is unplugged with a depth > 1. If you were tracing a | |
543 | set of events or functions at the time, you could then examine the | |
544 | trace buffer to see the sequence of events that led up to the | |
73d98127 | 545 | trigger event:: |
ac38fb85 | 546 | |
73d98127 | 547 | # echo 'traceoff:1 if nr_rq > 1' > \ |
2abfcd29 | 548 | /sys/kernel/tracing/events/block/block_unplug/trigger |
ac38fb85 | 549 | |
73d98127 | 550 | To always disable tracing when nr_rq > 1:: |
ac38fb85 | 551 | |
73d98127 | 552 | # echo 'traceoff if nr_rq > 1' > \ |
2abfcd29 | 553 | /sys/kernel/tracing/events/block/block_unplug/trigger |
ac38fb85 | 554 | |
73d98127 | 555 | To remove the above commands:: |
ac38fb85 | 556 | |
73d98127 | 557 | # echo '!traceoff:1 if nr_rq > 1' > \ |
2abfcd29 | 558 | /sys/kernel/tracing/events/block/block_unplug/trigger |
ac38fb85 | 559 | |
73d98127 | 560 | # echo '!traceoff if nr_rq > 1' > \ |
2abfcd29 | 561 | /sys/kernel/tracing/events/block/block_unplug/trigger |
ac38fb85 TZ |
562 | |
563 | Note that there can be only one traceon or traceoff trigger per | |
564 | triggering event. | |
0fc3813c TZ |
565 | |
566 | - hist | |
567 | ||
568 | This command aggregates event hits into a hash table keyed on one or | |
569 | more trace event format fields (or stacktrace) and a set of running | |
570 | totals derived from one or more trace event format fields and/or | |
571 | event counts (hitcount). | |
572 | ||
ea272257 | 573 | See Documentation/trace/histogram.rst for details and examples. |
34ed6357 | 574 | |
b8170fad TZ |
575 | 7. In-kernel trace event API |
576 | ============================ | |
34ed6357 TZ |
577 | |
578 | In most cases, the command-line interface to trace events is more than | |
579 | sufficient. Sometimes, however, applications might find the need for | |
580 | more complex relationships than can be expressed through a simple | |
581 | series of linked command-line expressions, or putting together sets of | |
582 | commands may be simply too cumbersome. An example might be an | |
583 | application that needs to 'listen' to the trace stream in order to | |
584 | maintain an in-kernel state machine detecting, for instance, when an | |
585 | illegal kernel state occurs in the scheduler. | |
586 | ||
587 | The trace event subsystem provides an in-kernel API allowing modules | |
588 | or other kernel code to generate user-defined 'synthetic' events at | |
589 | will, which can be used to either augment the existing trace stream | |
590 | and/or signal that a particular important state has occurred. | |
591 | ||
592 | A similar in-kernel API is also available for creating kprobe and | |
593 | kretprobe events. | |
594 | ||
595 | Both the synthetic event and k/ret/probe event APIs are built on top | |
596 | of a lower-level "dynevent_cmd" event command API, which is also | |
597 | available for more specialized applications, or as the basis of other | |
598 | higher-level trace event APIs. | |
599 | ||
600 | The API provided for these purposes is describe below and allows the | |
601 | following: | |
602 | ||
603 | - dynamically creating synthetic event definitions | |
604 | - dynamically creating kprobe and kretprobe event definitions | |
605 | - tracing synthetic events from in-kernel code | |
606 | - the low-level "dynevent_cmd" API | |
607 | ||
b8170fad TZ |
608 | 7.1 Dyamically creating synthetic event definitions |
609 | --------------------------------------------------- | |
34ed6357 TZ |
610 | |
611 | There are a couple ways to create a new synthetic event from a kernel | |
612 | module or other kernel code. | |
613 | ||
614 | The first creates the event in one step, using synth_event_create(). | |
615 | In this method, the name of the event to create and an array defining | |
616 | the fields is supplied to synth_event_create(). If successful, a | |
617 | synthetic event with that name and fields will exist following that | |
8206de7d | 618 | call. For example, to create a new "schedtest" synthetic event:: |
34ed6357 TZ |
619 | |
620 | ret = synth_event_create("schedtest", sched_fields, | |
621 | ARRAY_SIZE(sched_fields), THIS_MODULE); | |
622 | ||
623 | The sched_fields param in this example points to an array of struct | |
624 | synth_field_desc, each of which describes an event field by type and | |
8206de7d | 625 | name:: |
34ed6357 TZ |
626 | |
627 | static struct synth_field_desc sched_fields[] = { | |
628 | { .type = "pid_t", .name = "next_pid_field" }, | |
629 | { .type = "char[16]", .name = "next_comm_field" }, | |
630 | { .type = "u64", .name = "ts_ns" }, | |
631 | { .type = "u64", .name = "ts_ms" }, | |
632 | { .type = "unsigned int", .name = "cpu" }, | |
633 | { .type = "char[64]", .name = "my_string_field" }, | |
634 | { .type = "int", .name = "my_int_field" }, | |
635 | }; | |
636 | ||
bd82631d TZ |
637 | See synth_field_size() for available types. |
638 | ||
639 | If field_name contains [n], the field is considered to be a static array. | |
640 | ||
641 | If field_names contains[] (no subscript), the field is considered to | |
642 | be a dynamic array, which will only take as much space in the event as | |
643 | is required to hold the array. | |
644 | ||
645 | Because space for an event is reserved before assigning field values | |
646 | to the event, using dynamic arrays implies that the piecewise | |
647 | in-kernel API described below can't be used with dynamic arrays. The | |
648 | other non-piecewise in-kernel APIs can, however, be used with dynamic | |
649 | arrays. | |
34ed6357 TZ |
650 | |
651 | If the event is created from within a module, a pointer to the module | |
652 | must be passed to synth_event_create(). This will ensure that the | |
653 | trace buffer won't contain unreadable events when the module is | |
654 | removed. | |
655 | ||
656 | At this point, the event object is ready to be used for generating new | |
657 | events. | |
658 | ||
659 | In the second method, the event is created in several steps. This | |
660 | allows events to be created dynamically and without the need to create | |
661 | and populate an array of fields beforehand. | |
662 | ||
663 | To use this method, an empty or partially empty synthetic event should | |
664 | first be created using synth_event_gen_cmd_start() or | |
665 | synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(), | |
666 | the name of the event along with one or more pairs of args each pair | |
667 | representing a 'type field_name;' field specification should be | |
668 | supplied. For synth_event_gen_cmd_array_start(), the name of the | |
669 | event along with an array of struct synth_field_desc should be | |
670 | supplied. Before calling synth_event_gen_cmd_start() or | |
671 | synth_event_gen_cmd_array_start(), the user should create and | |
672 | initialize a dynevent_cmd object using synth_event_cmd_init(). | |
673 | ||
674 | For example, to create a new "schedtest" synthetic event with two | |
8206de7d | 675 | fields:: |
34ed6357 TZ |
676 | |
677 | struct dynevent_cmd cmd; | |
678 | char *buf; | |
679 | ||
680 | /* Create a buffer to hold the generated command */ | |
681 | buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); | |
682 | ||
683 | /* Before generating the command, initialize the cmd object */ | |
684 | synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN); | |
685 | ||
686 | ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE, | |
687 | "pid_t", "next_pid_field", | |
688 | "u64", "ts_ns"); | |
689 | ||
690 | Alternatively, using an array of struct synth_field_desc fields | |
8206de7d | 691 | containing the same information:: |
34ed6357 TZ |
692 | |
693 | ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE, | |
694 | fields, n_fields); | |
695 | ||
696 | Once the synthetic event object has been created, it can then be | |
697 | populated with more fields. Fields are added one by one using | |
698 | synth_event_add_field(), supplying the dynevent_cmd object, a field | |
699 | type, and a field name. For example, to add a new int field named | |
8206de7d | 700 | "intfield", the following call should be made:: |
34ed6357 TZ |
701 | |
702 | ret = synth_event_add_field(&cmd, "int", "intfield"); | |
703 | ||
704 | See synth_field_size() for available types. If field_name contains [n] | |
705 | the field is considered to be an array. | |
706 | ||
707 | A group of fields can also be added all at once using an array of | |
708 | synth_field_desc with add_synth_fields(). For example, this would add | |
8206de7d | 709 | just the first four sched_fields:: |
34ed6357 TZ |
710 | |
711 | ret = synth_event_add_fields(&cmd, sched_fields, 4); | |
712 | ||
713 | If you already have a string of the form 'type field_name', | |
714 | synth_event_add_field_str() can be used to add it as-is; it will | |
715 | also automatically append a ';' to the string. | |
716 | ||
717 | Once all the fields have been added, the event should be finalized and | |
8206de7d | 718 | registered by calling the synth_event_gen_cmd_end() function:: |
34ed6357 TZ |
719 | |
720 | ret = synth_event_gen_cmd_end(&cmd); | |
721 | ||
722 | At this point, the event object is ready to be used for tracing new | |
723 | events. | |
724 | ||
b8170fad TZ |
725 | 7.2 Tracing synthetic events from in-kernel code |
726 | ------------------------------------------------ | |
34ed6357 TZ |
727 | |
728 | To trace a synthetic event, there are several options. The first | |
729 | option is to trace the event in one call, using synth_event_trace() | |
730 | with a variable number of values, or synth_event_trace_array() with an | |
731 | array of values to be set. A second option can be used to avoid the | |
732 | need for a pre-formed array of values or list of arguments, via | |
733 | synth_event_trace_start() and synth_event_trace_end() along with | |
734 | synth_event_add_next_val() or synth_event_add_val() to add the values | |
735 | piecewise. | |
736 | ||
b8170fad TZ |
737 | 7.2.1 Tracing a synthetic event all at once |
738 | ------------------------------------------- | |
34ed6357 TZ |
739 | |
740 | To trace a synthetic event all at once, the synth_event_trace() or | |
741 | synth_event_trace_array() functions can be used. | |
742 | ||
743 | The synth_event_trace() function is passed the trace_event_file | |
744 | representing the synthetic event (which can be retrieved using | |
745 | trace_get_event_file() using the synthetic event name, "synthetic" as | |
746 | the system name, and the trace instance name (NULL if using the global | |
747 | trace array)), along with an variable number of u64 args, one for each | |
748 | synthetic event field, and the number of values being passed. | |
749 | ||
750 | So, to trace an event corresponding to the synthetic event definition | |
8206de7d | 751 | above, code like the following could be used:: |
34ed6357 TZ |
752 | |
753 | ret = synth_event_trace(create_synth_test, 7, /* number of values */ | |
754 | 444, /* next_pid_field */ | |
755 | (u64)"clackers", /* next_comm_field */ | |
756 | 1000000, /* ts_ns */ | |
757 | 1000, /* ts_ms */ | |
758 | smp_processor_id(),/* cpu */ | |
759 | (u64)"Thneed", /* my_string_field */ | |
760 | 999); /* my_int_field */ | |
761 | ||
762 | All vals should be cast to u64, and string vals are just pointers to | |
763 | strings, cast to u64. Strings will be copied into space reserved in | |
764 | the event for the string, using these pointers. | |
765 | ||
766 | Alternatively, the synth_event_trace_array() function can be used to | |
767 | accomplish the same thing. It is passed the trace_event_file | |
768 | representing the synthetic event (which can be retrieved using | |
769 | trace_get_event_file() using the synthetic event name, "synthetic" as | |
770 | the system name, and the trace instance name (NULL if using the global | |
771 | trace array)), along with an array of u64, one for each synthetic | |
772 | event field. | |
773 | ||
774 | To trace an event corresponding to the synthetic event definition | |
8206de7d | 775 | above, code like the following could be used:: |
34ed6357 TZ |
776 | |
777 | u64 vals[7]; | |
778 | ||
779 | vals[0] = 777; /* next_pid_field */ | |
780 | vals[1] = (u64)"tiddlywinks"; /* next_comm_field */ | |
781 | vals[2] = 1000000; /* ts_ns */ | |
782 | vals[3] = 1000; /* ts_ms */ | |
783 | vals[4] = smp_processor_id(); /* cpu */ | |
784 | vals[5] = (u64)"thneed"; /* my_string_field */ | |
785 | vals[6] = 398; /* my_int_field */ | |
786 | ||
787 | The 'vals' array is just an array of u64, the number of which must | |
788 | match the number of field in the synthetic event, and which must be in | |
789 | the same order as the synthetic event fields. | |
790 | ||
791 | All vals should be cast to u64, and string vals are just pointers to | |
792 | strings, cast to u64. Strings will be copied into space reserved in | |
793 | the event for the string, using these pointers. | |
794 | ||
795 | In order to trace a synthetic event, a pointer to the trace event file | |
796 | is needed. The trace_get_event_file() function can be used to get | |
797 | it - it will find the file in the given trace instance (in this case | |
798 | NULL since the top trace array is being used) while at the same time | |
8206de7d | 799 | preventing the instance containing it from going away:: |
34ed6357 TZ |
800 | |
801 | schedtest_event_file = trace_get_event_file(NULL, "synthetic", | |
802 | "schedtest"); | |
803 | ||
804 | Before tracing the event, it should be enabled in some way, otherwise | |
805 | the synthetic event won't actually show up in the trace buffer. | |
806 | ||
807 | To enable a synthetic event from the kernel, trace_array_set_clr_event() | |
808 | can be used (which is not specific to synthetic events, so does need | |
809 | the "synthetic" system name to be specified explicitly). | |
810 | ||
8206de7d | 811 | To enable the event, pass 'true' to it:: |
34ed6357 TZ |
812 | |
813 | trace_array_set_clr_event(schedtest_event_file->tr, | |
814 | "synthetic", "schedtest", true); | |
815 | ||
8206de7d | 816 | To disable it pass false:: |
34ed6357 TZ |
817 | |
818 | trace_array_set_clr_event(schedtest_event_file->tr, | |
819 | "synthetic", "schedtest", false); | |
820 | ||
821 | Finally, synth_event_trace_array() can be used to actually trace the | |
8206de7d | 822 | event, which should be visible in the trace buffer afterwards:: |
34ed6357 TZ |
823 | |
824 | ret = synth_event_trace_array(schedtest_event_file, vals, | |
825 | ARRAY_SIZE(vals)); | |
826 | ||
827 | To remove the synthetic event, the event should be disabled, and the | |
8206de7d | 828 | trace instance should be 'put' back using trace_put_event_file():: |
34ed6357 TZ |
829 | |
830 | trace_array_set_clr_event(schedtest_event_file->tr, | |
831 | "synthetic", "schedtest", false); | |
832 | trace_put_event_file(schedtest_event_file); | |
833 | ||
834 | If those have been successful, synth_event_delete() can be called to | |
8206de7d | 835 | remove the event:: |
34ed6357 TZ |
836 | |
837 | ret = synth_event_delete("schedtest"); | |
838 | ||
b8170fad TZ |
839 | 7.2.2 Tracing a synthetic event piecewise |
840 | ----------------------------------------- | |
34ed6357 TZ |
841 | |
842 | To trace a synthetic using the piecewise method described above, the | |
843 | synth_event_trace_start() function is used to 'open' the synthetic | |
8206de7d | 844 | event trace:: |
34ed6357 | 845 | |
301de546 | 846 | struct synth_event_trace_state trace_state; |
34ed6357 TZ |
847 | |
848 | ret = synth_event_trace_start(schedtest_event_file, &trace_state); | |
849 | ||
850 | It's passed the trace_event_file representing the synthetic event | |
851 | using the same methods as described above, along with a pointer to a | |
301de546 | 852 | struct synth_event_trace_state object, which will be zeroed before use and |
34ed6357 TZ |
853 | used to maintain state between this and following calls. |
854 | ||
855 | Once the event has been opened, which means space for it has been | |
856 | reserved in the trace buffer, the individual fields can be set. There | |
857 | are two ways to do that, either one after another for each field in | |
858 | the event, which requires no lookups, or by name, which does. The | |
859 | tradeoff is flexibility in doing the assignments vs the cost of a | |
860 | lookup per field. | |
861 | ||
862 | To assign the values one after the other without lookups, | |
863 | synth_event_add_next_val() should be used. Each call is passed the | |
301de546 | 864 | same synth_event_trace_state object used in the synth_event_trace_start(), |
34ed6357 TZ |
865 | along with the value to set the next field in the event. After each |
866 | field is set, the 'cursor' points to the next field, which will be set | |
867 | by the subsequent call, continuing until all the fields have been set | |
868 | in order. The same sequence of calls as in the above examples using | |
8206de7d | 869 | this method would be (without error-handling code):: |
34ed6357 TZ |
870 | |
871 | /* next_pid_field */ | |
872 | ret = synth_event_add_next_val(777, &trace_state); | |
873 | ||
874 | /* next_comm_field */ | |
875 | ret = synth_event_add_next_val((u64)"slinky", &trace_state); | |
876 | ||
877 | /* ts_ns */ | |
878 | ret = synth_event_add_next_val(1000000, &trace_state); | |
879 | ||
880 | /* ts_ms */ | |
881 | ret = synth_event_add_next_val(1000, &trace_state); | |
882 | ||
883 | /* cpu */ | |
884 | ret = synth_event_add_next_val(smp_processor_id(), &trace_state); | |
885 | ||
886 | /* my_string_field */ | |
887 | ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state); | |
888 | ||
889 | /* my_int_field */ | |
890 | ret = synth_event_add_next_val(395, &trace_state); | |
891 | ||
892 | To assign the values in any order, synth_event_add_val() should be | |
301de546 | 893 | used. Each call is passed the same synth_event_trace_state object used in |
34ed6357 TZ |
894 | the synth_event_trace_start(), along with the field name of the field |
895 | to set and the value to set it to. The same sequence of calls as in | |
896 | the above examples using this method would be (without error-handling | |
8206de7d | 897 | code):: |
34ed6357 TZ |
898 | |
899 | ret = synth_event_add_val("next_pid_field", 777, &trace_state); | |
900 | ret = synth_event_add_val("next_comm_field", (u64)"silly putty", | |
901 | &trace_state); | |
902 | ret = synth_event_add_val("ts_ns", 1000000, &trace_state); | |
903 | ret = synth_event_add_val("ts_ms", 1000, &trace_state); | |
904 | ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state); | |
905 | ret = synth_event_add_val("my_string_field", (u64)"thneed_9", | |
906 | &trace_state); | |
907 | ret = synth_event_add_val("my_int_field", 3999, &trace_state); | |
908 | ||
909 | Note that synth_event_add_next_val() and synth_event_add_val() are | |
910 | incompatible if used within the same trace of an event - either one | |
911 | can be used but not both at the same time. | |
912 | ||
913 | Finally, the event won't be actually traced until it's 'closed', | |
914 | which is done using synth_event_trace_end(), which takes only the | |
301de546 | 915 | struct synth_event_trace_state object used in the previous calls:: |
34ed6357 TZ |
916 | |
917 | ret = synth_event_trace_end(&trace_state); | |
918 | ||
919 | Note that synth_event_trace_end() must be called at the end regardless | |
920 | of whether any of the add calls failed (say due to a bad field name | |
921 | being passed in). | |
922 | ||
b8170fad TZ |
923 | 7.3 Dyamically creating kprobe and kretprobe event definitions |
924 | -------------------------------------------------------------- | |
34ed6357 TZ |
925 | |
926 | To create a kprobe or kretprobe trace event from kernel code, the | |
927 | kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start() | |
928 | functions can be used. | |
929 | ||
930 | To create a kprobe event, an empty or partially empty kprobe event | |
931 | should first be created using kprobe_event_gen_cmd_start(). The name | |
d56b699d | 932 | of the event and the probe location should be specified along with one |
34ed6357 TZ |
933 | or args each representing a probe field should be supplied to this |
934 | function. Before calling kprobe_event_gen_cmd_start(), the user | |
935 | should create and initialize a dynevent_cmd object using | |
936 | kprobe_event_cmd_init(). | |
937 | ||
8206de7d | 938 | For example, to create a new "schedtest" kprobe event with two fields:: |
34ed6357 TZ |
939 | |
940 | struct dynevent_cmd cmd; | |
941 | char *buf; | |
942 | ||
943 | /* Create a buffer to hold the generated command */ | |
944 | buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); | |
945 | ||
946 | /* Before generating the command, initialize the cmd object */ | |
947 | kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN); | |
948 | ||
949 | /* | |
950 | * Define the gen_kprobe_test event with the first 2 kprobe | |
951 | * fields. | |
952 | */ | |
953 | ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open", | |
954 | "dfd=%ax", "filename=%dx"); | |
955 | ||
956 | Once the kprobe event object has been created, it can then be | |
957 | populated with more fields. Fields can be added using | |
958 | kprobe_event_add_fields(), supplying the dynevent_cmd object along | |
959 | with a variable arg list of probe fields. For example, to add a | |
8206de7d | 960 | couple additional fields, the following call could be made:: |
34ed6357 TZ |
961 | |
962 | ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)"); | |
963 | ||
964 | Once all the fields have been added, the event should be finalized and | |
965 | registered by calling the kprobe_event_gen_cmd_end() or | |
966 | kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe | |
8206de7d | 967 | or kretprobe command was started:: |
34ed6357 TZ |
968 | |
969 | ret = kprobe_event_gen_cmd_end(&cmd); | |
970 | ||
8206de7d | 971 | or:: |
34ed6357 TZ |
972 | |
973 | ret = kretprobe_event_gen_cmd_end(&cmd); | |
974 | ||
975 | At this point, the event object is ready to be used for tracing new | |
976 | events. | |
977 | ||
978 | Similarly, a kretprobe event can be created using | |
979 | kretprobe_event_gen_cmd_start() with a probe name and location and | |
8206de7d | 980 | additional params such as $retval:: |
34ed6357 TZ |
981 | |
982 | ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test", | |
983 | "do_sys_open", "$retval"); | |
984 | ||
985 | Similar to the synthetic event case, code like the following can be | |
8206de7d | 986 | used to enable the newly created kprobe event:: |
34ed6357 TZ |
987 | |
988 | gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test"); | |
989 | ||
990 | ret = trace_array_set_clr_event(gen_kprobe_test->tr, | |
991 | "kprobes", "gen_kprobe_test", true); | |
992 | ||
993 | Finally, also similar to synthetic events, the following code can be | |
8206de7d | 994 | used to give the kprobe event file back and delete the event:: |
34ed6357 TZ |
995 | |
996 | trace_put_event_file(gen_kprobe_test); | |
997 | ||
998 | ret = kprobe_event_delete("gen_kprobe_test"); | |
999 | ||
b8170fad TZ |
1000 | 7.4 The "dynevent_cmd" low-level API |
1001 | ------------------------------------ | |
34ed6357 TZ |
1002 | |
1003 | Both the in-kernel synthetic event and kprobe interfaces are built on | |
1004 | top of a lower-level "dynevent_cmd" interface. This interface is | |
1005 | meant to provide the basis for higher-level interfaces such as the | |
1006 | synthetic and kprobe interfaces, which can be used as examples. | |
1007 | ||
1008 | The basic idea is simple and amounts to providing a general-purpose | |
1009 | layer that can be used to generate trace event commands. The | |
1010 | generated command strings can then be passed to the command-parsing | |
1011 | and event creation code that already exists in the trace event | |
d56b699d | 1012 | subsystem for creating the corresponding trace events. |
34ed6357 TZ |
1013 | |
1014 | In a nutshell, the way it works is that the higher-level interface | |
1015 | code creates a struct dynevent_cmd object, then uses a couple | |
1016 | functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up | |
1017 | a command string, which finally causes the command to be executed | |
1018 | using the dynevent_create() function. The details of the interface | |
1019 | are described below. | |
1020 | ||
1021 | The first step in building a new command string is to create and | |
1022 | initialize an instance of a dynevent_cmd. Here, for instance, we | |
8206de7d | 1023 | create a dynevent_cmd on the stack and initialize it:: |
34ed6357 TZ |
1024 | |
1025 | struct dynevent_cmd cmd; | |
1026 | char *buf; | |
1027 | int ret; | |
1028 | ||
1029 | buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL); | |
1030 | ||
1031 | dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO, | |
1032 | foo_event_run_command); | |
1033 | ||
1034 | The dynevent_cmd initialization needs to be given a user-specified | |
1035 | buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used | |
1036 | for this purpose - at 2k it's generally too big to be comfortably put | |
1037 | on the stack, so is dynamically allocated), a dynevent type id, which | |
1038 | is meant to be used to check that further API calls are for the | |
1039 | correct command type, and a pointer to an event-specific run_command() | |
1040 | callback that will be called to actually execute the event-specific | |
1041 | command function. | |
1042 | ||
1043 | Once that's done, the command string can by built up by successive | |
1044 | calls to argument-adding functions. | |
1045 | ||
1046 | To add a single argument, define and initialize a struct dynevent_arg | |
1047 | or struct dynevent_arg_pair object. Here's an example of the simplest | |
1048 | possible arg addition, which is simply to append the given string as | |
8206de7d | 1049 | a whitespace-separated argument to the command:: |
34ed6357 TZ |
1050 | |
1051 | struct dynevent_arg arg; | |
1052 | ||
1053 | dynevent_arg_init(&arg, NULL, 0); | |
1054 | ||
1055 | arg.str = name; | |
1056 | ||
1057 | ret = dynevent_arg_add(cmd, &arg); | |
1058 | ||
1059 | The arg object is first initialized using dynevent_arg_init() and in | |
1060 | this case the parameters are NULL or 0, which means there's no | |
1061 | optional sanity-checking function or separator appended to the end of | |
1062 | the arg. | |
1063 | ||
1064 | Here's another more complicated example using an 'arg pair', which is | |
1065 | used to create an argument that consists of a couple components added | |
1066 | together as a unit, for example, a 'type field_name;' arg or a simple | |
8206de7d | 1067 | expression arg e.g. 'flags=%cx':: |
34ed6357 TZ |
1068 | |
1069 | struct dynevent_arg_pair arg_pair; | |
1070 | ||
1071 | dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';'); | |
1072 | ||
1073 | arg_pair.lhs = type; | |
1074 | arg_pair.rhs = name; | |
1075 | ||
1076 | ret = dynevent_arg_pair_add(cmd, &arg_pair); | |
1077 | ||
1078 | Again, the arg_pair is first initialized, in this case with a callback | |
1079 | function used to check the sanity of the args (for example, that | |
1080 | neither part of the pair is NULL), along with a character to be used | |
1081 | to add an operator between the pair (here none) and a separator to be | |
1082 | appended onto the end of the arg pair (here ';'). | |
1083 | ||
1084 | There's also a dynevent_str_add() function that can be used to simply | |
d56b699d | 1085 | add a string as-is, with no spaces, delimiters, or arg check. |
34ed6357 TZ |
1086 | |
1087 | Any number of dynevent_*_add() calls can be made to build up the string | |
1088 | (until its length surpasses cmd->maxlen). When all the arguments have | |
1089 | been added and the command string is complete, the only thing left to | |
1090 | do is run the command, which happens by simply calling | |
8206de7d | 1091 | dynevent_create():: |
34ed6357 TZ |
1092 | |
1093 | ret = dynevent_create(&cmd); | |
1094 | ||
1095 | At that point, if the return value is 0, the dynamic event has been | |
1096 | created and is ready to use. | |
1097 | ||
1098 | See the dynevent_cmd function definitions themselves for the details | |
1099 | of the API. |