Commit | Line | Data |
---|---|---|
77b44d1b MH |
1 | Kprobe-based Event Tracing |
2 | ========================== | |
d8ec9185 MH |
3 | |
4 | Documentation is written by Masami Hiramatsu | |
5 | ||
6 | ||
7 | Overview | |
8 | -------- | |
77b44d1b MH |
9 | These events are similar to tracepoint based events. Instead of Tracepoint, |
10 | this is based on kprobes (kprobe and kretprobe). So it can probe wherever | |
c1ac094d NR |
11 | kprobes can probe (this means, all functions except those with |
12 | __kprobes/nokprobe_inline annotation and those marked NOKPROBE_SYMBOL). | |
13 | Unlike the Tracepoint based event, this can be added and removed | |
77b44d1b | 14 | dynamically, on the fly. |
d8ec9185 | 15 | |
6b0b7551 | 16 | To enable this feature, build your kernel with CONFIG_KPROBE_EVENTS=y. |
d8ec9185 | 17 | |
77b44d1b MH |
18 | Similar to the events tracer, this doesn't need to be activated via |
19 | current_tracer. Instead of that, add probe points via | |
20 | /sys/kernel/debug/tracing/kprobe_events, and enable it via | |
21 | /sys/kernel/debug/tracing/events/kprobes/<EVENT>/enabled. | |
d8ec9185 MH |
22 | |
23 | ||
24 | Synopsis of kprobe_events | |
25 | ------------------------- | |
61424318 | 26 | p[:[GRP/]EVENT] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS] : Set a probe |
696ced4f | 27 | r[MAXACTIVE][:[GRP/]EVENT] [MOD:]SYM[+0] [FETCHARGS] : Set a return probe |
df3ab708 | 28 | -:[GRP/]EVENT : Clear a probe |
d8ec9185 | 29 | |
f52487e9 | 30 | GRP : Group name. If omitted, use "kprobes" for it. |
2fba0c88 | 31 | EVENT : Event name. If omitted, the event name is generated |
61424318 MH |
32 | based on SYM+offs or MEMADDR. |
33 | MOD : Module name which has given SYM. | |
34 | SYM[+offs] : Symbol+offset where the probe is inserted. | |
2fba0c88 | 35 | MEMADDR : Address where the probe is inserted. |
696ced4f AC |
36 | MAXACTIVE : Maximum number of instances of the specified function that |
37 | can be probed simultaneously, or 0 for the default value | |
38 | as defined in Documentation/kprobes.txt section 1.3.1. | |
d8ec9185 | 39 | |
2fba0c88 | 40 | FETCHARGS : Arguments. Each probe can have up to 128 args. |
2e06ff63 MH |
41 | %REG : Fetch register REG |
42 | @ADDR : Fetch memory at ADDR (ADDR should be in kernel) | |
d8ec9185 | 43 | @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol) |
2e06ff63 MH |
44 | $stackN : Fetch Nth entry of stack (N >= 0) |
45 | $stack : Fetch stack address. | |
14640106 | 46 | $retval : Fetch return value.(*) |
35abb67d | 47 | $comm : Fetch current task comm. |
14640106 | 48 | +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(**) |
93ccae7a MH |
49 | NAME=FETCHARG : Set NAME as the argument name of FETCHARG. |
50 | FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types | |
17ce3dc7 MH |
51 | (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types |
52 | (x8/x16/x32/x64), "string" and bitfield are supported. | |
d8ec9185 | 53 | |
14640106 MH |
54 | (*) only for return probe. |
55 | (**) this is useful for fetching a field of data structures. | |
d8ec9185 | 56 | |
1ff511e3 MH |
57 | Types |
58 | ----- | |
59 | Several types are supported for fetch-args. Kprobe tracer will access memory | |
60 | by given type. Prefix 's' and 'u' means those types are signed and unsigned | |
bdca79c2 MH |
61 | respectively. 'x' prefix implies it is unsigned. Traced arguments are shown |
62 | in decimal ('s' and 'u') or hexadecimal ('x'). Without type casting, 'x32' | |
63 | or 'x64' is used depends on the architecture (e.g. x86-32 uses x32, and | |
64 | x86-64 uses x64). | |
1ff511e3 MH |
65 | String type is a special type, which fetches a "null-terminated" string from |
66 | kernel space. This means it will fail and store NULL if the string container | |
67 | has been paged out. | |
68 | Bitfield is another special type, which takes 3 parameters, bit-width, bit- | |
69 | offset, and container-size (usually 32). The syntax is; | |
70 | ||
71 | b<bit-width>@<bit-offset>/<container-size> | |
72 | ||
35abb67d OS |
73 | For $comm, the default type is "string"; any other type is invalid. |
74 | ||
d8ec9185 MH |
75 | |
76 | Per-Probe Event Filtering | |
77 | ------------------------- | |
78 | Per-probe event filtering feature allows you to set different filter on each | |
79 | probe and gives you what arguments will be shown in trace buffer. If an event | |
77b44d1b MH |
80 | name is specified right after 'p:' or 'r:' in kprobe_events, it adds an event |
81 | under tracing/events/kprobes/<EVENT>, at the directory you can see 'id', | |
82 | 'enabled', 'format' and 'filter'. | |
d8ec9185 MH |
83 | |
84 | enabled: | |
85 | You can enable/disable the probe by writing 1 or 0 on it. | |
86 | ||
87 | format: | |
eca0d916 | 88 | This shows the format of this probe event. |
d8ec9185 MH |
89 | |
90 | filter: | |
eca0d916 | 91 | You can write filtering rules of this event. |
d8ec9185 | 92 | |
e08d1c65 MH |
93 | id: |
94 | This shows the id of this probe event. | |
d8ec9185 | 95 | |
77b44d1b | 96 | |
cd7e7bd5 MH |
97 | Event Profiling |
98 | --------------- | |
99 | You can check the total number of probe hits and probe miss-hits via | |
100 | /sys/kernel/debug/tracing/kprobe_profile. | |
101 | The first column is event name, the second is the number of probe hits, | |
102 | the third is the number of probe miss-hits. | |
103 | ||
104 | ||
d8ec9185 MH |
105 | Usage examples |
106 | -------------- | |
107 | To add a probe as a new event, write a new definition to kprobe_events | |
108 | as below. | |
109 | ||
580d9e00 | 110 | echo 'p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)' > /sys/kernel/debug/tracing/kprobe_events |
d8ec9185 MH |
111 | |
112 | This sets a kprobe on the top of do_sys_open() function with recording | |
14640106 MH |
113 | 1st to 4th arguments as "myprobe" event. Note, which register/stack entry is |
114 | assigned to each function argument depends on arch-specific ABI. If you unsure | |
115 | the ABI, please try to use probe subcommand of perf-tools (you can find it | |
116 | under tools/perf/). | |
117 | As this example shows, users can choose more familiar names for each arguments. | |
d8ec9185 | 118 | |
580d9e00 | 119 | echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events |
d8ec9185 MH |
120 | |
121 | This sets a kretprobe on the return point of do_sys_open() function with | |
99329c44 | 122 | recording return value as "myretprobe" event. |
d8ec9185 MH |
123 | You can see the format of these events via |
124 | /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format. | |
125 | ||
126 | cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format | |
127 | name: myprobe | |
ec3a9039 | 128 | ID: 780 |
d8ec9185 | 129 | format: |
ec3a9039 MH |
130 | field:unsigned short common_type; offset:0; size:2; signed:0; |
131 | field:unsigned char common_flags; offset:2; size:1; signed:0; | |
132 | field:unsigned char common_preempt_count; offset:3; size:1;signed:0; | |
133 | field:int common_pid; offset:4; size:4; signed:1; | |
ec3a9039 MH |
134 | |
135 | field:unsigned long __probe_ip; offset:12; size:4; signed:0; | |
136 | field:int __probe_nargs; offset:16; size:4; signed:1; | |
137 | field:unsigned long dfd; offset:20; size:4; signed:0; | |
138 | field:unsigned long filename; offset:24; size:4; signed:0; | |
139 | field:unsigned long flags; offset:28; size:4; signed:0; | |
140 | field:unsigned long mode; offset:32; size:4; signed:0; | |
141 | ||
142 | ||
143 | print fmt: "(%lx) dfd=%lx filename=%lx flags=%lx mode=%lx", REC->__probe_ip, | |
144 | REC->dfd, REC->filename, REC->flags, REC->mode | |
d8ec9185 | 145 | |
eca0d916 | 146 | You can see that the event has 4 arguments as in the expressions you specified. |
d8ec9185 MH |
147 | |
148 | echo > /sys/kernel/debug/tracing/kprobe_events | |
149 | ||
5a0d9050 MH |
150 | This clears all probe points. |
151 | ||
df3ab708 MK |
152 | Or, |
153 | ||
154 | echo -:myprobe >> kprobe_events | |
155 | ||
156 | This clears probe points selectively. | |
157 | ||
5a0d9050 MH |
158 | Right after definition, each event is disabled by default. For tracing these |
159 | events, you need to enable it. | |
160 | ||
161 | echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable | |
162 | echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable | |
163 | ||
164 | And you can see the traced information via /sys/kernel/debug/tracing/trace. | |
d8ec9185 MH |
165 | |
166 | cat /sys/kernel/debug/tracing/trace | |
167 | # tracer: nop | |
168 | # | |
169 | # TASK-PID CPU# TIMESTAMP FUNCTION | |
170 | # | | | | | | |
6e9f23d1 | 171 | <...>-1447 [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0 |
2e06ff63 | 172 | <...>-1447 [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $retval=fffffffffffffffe |
6e9f23d1 | 173 | <...>-1447 [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6 |
2e06ff63 | 174 | <...>-1447 [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3 |
6e9f23d1 | 175 | <...>-1447 [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10 |
2e06ff63 | 176 | <...>-1447 [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3 |
d8ec9185 MH |
177 | |
178 | ||
6e9f23d1 | 179 | Each line shows when the kernel hits an event, and <- SYMBOL means kernel |
d8ec9185 MH |
180 | returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel |
181 | returns from do_sys_open to sys_open+0x1b). | |
182 |