Commit | Line | Data |
---|---|---|
263ee775 CD |
1 | ========================== |
2 | Kprobe-based Event Tracing | |
3 | ========================== | |
d8ec9185 | 4 | |
263ee775 | 5 | :Author: Masami Hiramatsu |
d8ec9185 MH |
6 | |
7 | Overview | |
8 | -------- | |
77b44d1b MH |
9 | These events are similar to tracepoint based events. Instead of Tracepoint, |
10 | this is based on kprobes (kprobe and kretprobe). So it can probe wherever | |
c1ac094d NR |
11 | kprobes can probe (this means, all functions except those with |
12 | __kprobes/nokprobe_inline annotation and those marked NOKPROBE_SYMBOL). | |
13 | Unlike the Tracepoint based event, this can be added and removed | |
77b44d1b | 14 | dynamically, on the fly. |
d8ec9185 | 15 | |
6b0b7551 | 16 | To enable this feature, build your kernel with CONFIG_KPROBE_EVENTS=y. |
d8ec9185 | 17 | |
77b44d1b MH |
18 | Similar to the events tracer, this doesn't need to be activated via |
19 | current_tracer. Instead of that, add probe points via | |
20 | /sys/kernel/debug/tracing/kprobe_events, and enable it via | |
e50891d6 | 21 | /sys/kernel/debug/tracing/events/kprobes/<EVENT>/enable. |
d8ec9185 | 22 | |
6212dd29 MH |
23 | You can also use /sys/kernel/debug/tracing/dynamic_events instead of |
24 | kprobe_events. That interface will provide unified access to other | |
25 | dynamic events too. | |
d8ec9185 MH |
26 | |
27 | Synopsis of kprobe_events | |
28 | ------------------------- | |
263ee775 CD |
29 | :: |
30 | ||
61424318 | 31 | p[:[GRP/]EVENT] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS] : Set a probe |
696ced4f | 32 | r[MAXACTIVE][:[GRP/]EVENT] [MOD:]SYM[+0] [FETCHARGS] : Set a return probe |
638e476d | 33 | p:[GRP/]EVENT] [MOD:]SYM[+0]%return [FETCHARGS] : Set a return probe |
df3ab708 | 34 | -:[GRP/]EVENT : Clear a probe |
d8ec9185 | 35 | |
f52487e9 | 36 | GRP : Group name. If omitted, use "kprobes" for it. |
2fba0c88 | 37 | EVENT : Event name. If omitted, the event name is generated |
61424318 MH |
38 | based on SYM+offs or MEMADDR. |
39 | MOD : Module name which has given SYM. | |
40 | SYM[+offs] : Symbol+offset where the probe is inserted. | |
638e476d | 41 | SYM%return : Return address of the symbol |
2fba0c88 | 42 | MEMADDR : Address where the probe is inserted. |
696ced4f AC |
43 | MAXACTIVE : Maximum number of instances of the specified function that |
44 | can be probed simultaneously, or 0 for the default value | |
7f9a2357 | 45 | as defined in Documentation/trace/kprobes.rst section 1.3.1. |
d8ec9185 | 46 | |
2fba0c88 | 47 | FETCHARGS : Arguments. Each probe can have up to 128 args. |
2e06ff63 MH |
48 | %REG : Fetch register REG |
49 | @ADDR : Fetch memory at ADDR (ADDR should be in kernel) | |
d8ec9185 | 50 | @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol) |
2e06ff63 MH |
51 | $stackN : Fetch Nth entry of stack (N >= 0) |
52 | $stack : Fetch stack address. | |
a1303af5 MH |
53 | $argN : Fetch the Nth function argument. (N >= 1) (\*1) |
54 | $retval : Fetch return value.(\*2) | |
35abb67d | 55 | $comm : Fetch current task comm. |
e65f7ae7 | 56 | +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4) |
6218bf9f | 57 | \IMM : Store an immediate value to the argument. |
93ccae7a MH |
58 | NAME=FETCHARG : Set NAME as the argument name of FETCHARG. |
59 | FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types | |
17ce3dc7 | 60 | (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types |
88903c46 MH |
61 | (x8/x16/x32/x64), "string", "ustring" and bitfield |
62 | are supported. | |
d8ec9185 | 63 | |
a1303af5 MH |
64 | (\*1) only for the probe on function entry (offs == 0). |
65 | (\*2) only for return probe. | |
66 | (\*3) this is useful for fetching a field of data structures. | |
e65f7ae7 | 67 | (\*4) "u" means user-space dereference. See :ref:`user_mem_access`. |
d8ec9185 | 68 | |
1ff511e3 MH |
69 | Types |
70 | ----- | |
71 | Several types are supported for fetch-args. Kprobe tracer will access memory | |
72 | by given type. Prefix 's' and 'u' means those types are signed and unsigned | |
bdca79c2 MH |
73 | respectively. 'x' prefix implies it is unsigned. Traced arguments are shown |
74 | in decimal ('s' and 'u') or hexadecimal ('x'). Without type casting, 'x32' | |
75 | or 'x64' is used depends on the architecture (e.g. x86-32 uses x32, and | |
76 | x86-64 uses x64). | |
40b53b77 MH |
77 | These value types can be an array. To record array data, you can add '[N]' |
78 | (where N is a fixed number, less than 64) to the base type. | |
79 | E.g. 'x16[4]' means an array of x16 (2bytes hex) with 4 elements. | |
80 | Note that the array can be applied to memory type fetchargs, you can not | |
81 | apply it to registers/stack-entries etc. (for example, '$stack1:x8[8]' is | |
82 | wrong, but '+8($stack):x8[8]' is OK.) | |
1ff511e3 MH |
83 | String type is a special type, which fetches a "null-terminated" string from |
84 | kernel space. This means it will fail and store NULL if the string container | |
88903c46 | 85 | has been paged out. "ustring" type is an alternative of string for user-space. |
e65f7ae7 | 86 | See :ref:`user_mem_access` for more info.. |
40b53b77 MH |
87 | The string array type is a bit different from other types. For other base |
88 | types, <base-type>[1] is equal to <base-type> (e.g. +0(%di):x32[1] is same | |
89 | as +0(%di):x32.) But string[1] is not equal to string. The string type itself | |
90 | represents "char array", but string array type represents "char * array". | |
91 | So, for example, +0(%di):string[1] is equal to +0(+0(%di)):string. | |
1ff511e3 | 92 | Bitfield is another special type, which takes 3 parameters, bit-width, bit- |
263ee775 | 93 | offset, and container-size (usually 32). The syntax is:: |
1ff511e3 MH |
94 | |
95 | b<bit-width>@<bit-offset>/<container-size> | |
96 | ||
60c2e0ce MH |
97 | Symbol type('symbol') is an alias of u32 or u64 type (depends on BITS_PER_LONG) |
98 | which shows given pointer in "symbol+offset" style. | |
35abb67d OS |
99 | For $comm, the default type is "string"; any other type is invalid. |
100 | ||
e65f7ae7 | 101 | .. _user_mem_access: |
ff1e81a7 | 102 | |
e65f7ae7 MH |
103 | User Memory Access |
104 | ------------------ | |
105 | Kprobe events supports user-space memory access. For that purpose, you can use | |
106 | either user-space dereference syntax or 'ustring' type. | |
107 | ||
108 | The user-space dereference syntax allows you to access a field of a data | |
109 | structure in user-space. This is done by adding the "u" prefix to the | |
110 | dereference syntax. For example, +u4(%si) means it will read memory from the | |
111 | address in the register %si offset by 4, and the memory is expected to be in | |
112 | user-space. You can use this for strings too, e.g. +u0(%si):string will read | |
113 | a string from the address in the register %si that is expected to be in user- | |
114 | space. 'ustring' is a shortcut way of performing the same task. That is, | |
115 | +0(%si):ustring is equivalent to +u0(%si):string. | |
116 | ||
117 | Note that kprobe-event provides the user-memory access syntax but it doesn't | |
118 | use it transparently. This means if you use normal dereference or string type | |
119 | for user memory, it might fail, and may always fail on some archs. The user | |
120 | has to carefully check if the target data is in kernel or user space. | |
d8ec9185 MH |
121 | |
122 | Per-Probe Event Filtering | |
123 | ------------------------- | |
263ee775 | 124 | Per-probe event filtering feature allows you to set different filter on each |
d8ec9185 | 125 | probe and gives you what arguments will be shown in trace buffer. If an event |
77b44d1b MH |
126 | name is specified right after 'p:' or 'r:' in kprobe_events, it adds an event |
127 | under tracing/events/kprobes/<EVENT>, at the directory you can see 'id', | |
31130c8e | 128 | 'enable', 'format', 'filter' and 'trigger'. |
d8ec9185 | 129 | |
e50891d6 | 130 | enable: |
d8ec9185 MH |
131 | You can enable/disable the probe by writing 1 or 0 on it. |
132 | ||
133 | format: | |
eca0d916 | 134 | This shows the format of this probe event. |
d8ec9185 MH |
135 | |
136 | filter: | |
eca0d916 | 137 | You can write filtering rules of this event. |
d8ec9185 | 138 | |
e08d1c65 MH |
139 | id: |
140 | This shows the id of this probe event. | |
d8ec9185 | 141 | |
31130c8e AZ |
142 | trigger: |
143 | This allows to install trigger commands which are executed when the event is | |
144 | hit (for details, see Documentation/trace/events.rst, section 6). | |
77b44d1b | 145 | |
cd7e7bd5 MH |
146 | Event Profiling |
147 | --------------- | |
263ee775 | 148 | You can check the total number of probe hits and probe miss-hits via |
cd7e7bd5 | 149 | /sys/kernel/debug/tracing/kprobe_profile. |
263ee775 | 150 | The first column is event name, the second is the number of probe hits, |
cd7e7bd5 MH |
151 | the third is the number of probe miss-hits. |
152 | ||
970988e1 MH |
153 | Kernel Boot Parameter |
154 | --------------------- | |
155 | You can add and enable new kprobe events when booting up the kernel by | |
156 | "kprobe_event=" parameter. The parameter accepts a semicolon-delimited | |
157 | kprobe events, which format is similar to the kprobe_events. | |
158 | The difference is that the probe definition parameters are comma-delimited | |
159 | instead of space. For example, adding myprobe event on do_sys_open like below | |
160 | ||
161 | p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack) | |
162 | ||
163 | should be below for kernel boot parameter (just replace spaces with comma) | |
164 | ||
165 | p:myprobe,do_sys_open,dfd=%ax,filename=%dx,flags=%cx,mode=+4($stack) | |
166 | ||
cd7e7bd5 | 167 | |
d8ec9185 MH |
168 | Usage examples |
169 | -------------- | |
170 | To add a probe as a new event, write a new definition to kprobe_events | |
263ee775 | 171 | as below:: |
d8ec9185 | 172 | |
580d9e00 | 173 | echo 'p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)' > /sys/kernel/debug/tracing/kprobe_events |
d8ec9185 | 174 | |
263ee775 | 175 | This sets a kprobe on the top of do_sys_open() function with recording |
14640106 MH |
176 | 1st to 4th arguments as "myprobe" event. Note, which register/stack entry is |
177 | assigned to each function argument depends on arch-specific ABI. If you unsure | |
178 | the ABI, please try to use probe subcommand of perf-tools (you can find it | |
179 | under tools/perf/). | |
180 | As this example shows, users can choose more familiar names for each arguments. | |
263ee775 | 181 | :: |
d8ec9185 | 182 | |
580d9e00 | 183 | echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events |
d8ec9185 | 184 | |
263ee775 | 185 | This sets a kretprobe on the return point of do_sys_open() function with |
99329c44 | 186 | recording return value as "myretprobe" event. |
263ee775 | 187 | You can see the format of these events via |
d8ec9185 | 188 | /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format. |
263ee775 | 189 | :: |
d8ec9185 MH |
190 | |
191 | cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format | |
263ee775 CD |
192 | name: myprobe |
193 | ID: 780 | |
194 | format: | |
195 | field:unsigned short common_type; offset:0; size:2; signed:0; | |
196 | field:unsigned char common_flags; offset:2; size:1; signed:0; | |
197 | field:unsigned char common_preempt_count; offset:3; size:1;signed:0; | |
198 | field:int common_pid; offset:4; size:4; signed:1; | |
ec3a9039 | 199 | |
263ee775 CD |
200 | field:unsigned long __probe_ip; offset:12; size:4; signed:0; |
201 | field:int __probe_nargs; offset:16; size:4; signed:1; | |
202 | field:unsigned long dfd; offset:20; size:4; signed:0; | |
203 | field:unsigned long filename; offset:24; size:4; signed:0; | |
204 | field:unsigned long flags; offset:28; size:4; signed:0; | |
205 | field:unsigned long mode; offset:32; size:4; signed:0; | |
ec3a9039 MH |
206 | |
207 | ||
263ee775 CD |
208 | print fmt: "(%lx) dfd=%lx filename=%lx flags=%lx mode=%lx", REC->__probe_ip, |
209 | REC->dfd, REC->filename, REC->flags, REC->mode | |
d8ec9185 | 210 | |
263ee775 CD |
211 | You can see that the event has 4 arguments as in the expressions you specified. |
212 | :: | |
d8ec9185 MH |
213 | |
214 | echo > /sys/kernel/debug/tracing/kprobe_events | |
215 | ||
263ee775 | 216 | This clears all probe points. |
5a0d9050 | 217 | |
263ee775 CD |
218 | Or, |
219 | :: | |
df3ab708 MK |
220 | |
221 | echo -:myprobe >> kprobe_events | |
222 | ||
263ee775 | 223 | This clears probe points selectively. |
df3ab708 | 224 | |
263ee775 | 225 | Right after definition, each event is disabled by default. For tracing these |
5a0d9050 | 226 | events, you need to enable it. |
263ee775 | 227 | :: |
5a0d9050 MH |
228 | |
229 | echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable | |
230 | echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable | |
231 | ||
78a89463 LC |
232 | Use the following command to start tracing in an interval. |
233 | :: | |
7e6294cd | 234 | |
78a89463 LC |
235 | # echo 1 > tracing_on |
236 | Open something... | |
237 | # echo 0 > tracing_on | |
238 | ||
263ee775 CD |
239 | And you can see the traced information via /sys/kernel/debug/tracing/trace. |
240 | :: | |
d8ec9185 MH |
241 | |
242 | cat /sys/kernel/debug/tracing/trace | |
263ee775 CD |
243 | # tracer: nop |
244 | # | |
245 | # TASK-PID CPU# TIMESTAMP FUNCTION | |
246 | # | | | | | | |
247 | <...>-1447 [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0 | |
248 | <...>-1447 [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $retval=fffffffffffffffe | |
249 | <...>-1447 [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6 | |
250 | <...>-1447 [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3 | |
251 | <...>-1447 [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10 | |
252 | <...>-1447 [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3 | |
253 | ||
254 | ||
255 | Each line shows when the kernel hits an event, and <- SYMBOL means kernel | |
d8ec9185 MH |
256 | returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel |
257 | returns from do_sys_open to sys_open+0x1b). |