Commit | Line | Data |
---|---|---|
47781947 MH |
1 | .. SPDX-License-Identifier: GPL-2.0 |
2 | ||
3 | ================= | |
4 | Boot-time tracing | |
5 | ================= | |
6 | ||
7 | :Author: Masami Hiramatsu <mhiramat@kernel.org> | |
8 | ||
9 | Overview | |
10 | ======== | |
11 | ||
12 | Boot-time tracing allows users to trace boot-time process including | |
13 | device initialization with full features of ftrace including per-event | |
14 | filter and actions, histograms, kprobe-events and synthetic-events, | |
15 | and trace instances. | |
b8381ce7 | 16 | Since kernel command line is not enough to control these complex features, |
47781947 MH |
17 | this uses bootconfig file to describe tracing feature programming. |
18 | ||
19 | Options in the Boot Config | |
20 | ========================== | |
21 | ||
22 | Here is the list of available options list for boot time tracing in | |
23 | boot config file [1]_. All options are under "ftrace." or "kernel." | |
b8381ce7 | 24 | prefix. See kernel parameters for the options which starts |
47781947 MH |
25 | with "kernel." prefix [2]_. |
26 | ||
27 | .. [1] See :ref:`Documentation/admin-guide/bootconfig.rst <bootconfig>` | |
28 | .. [2] See :ref:`Documentation/admin-guide/kernel-parameters.rst <kernelparameters>` | |
29 | ||
30 | Ftrace Global Options | |
31 | --------------------- | |
32 | ||
33 | Ftrace global options have "kernel." prefix in boot config, which means | |
34 | these options are passed as a part of kernel legacy command line. | |
35 | ||
36 | kernel.tp_printk | |
37 | Output trace-event data on printk buffer too. | |
38 | ||
39 | kernel.dump_on_oops [= MODE] | |
40 | Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer | |
41 | on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops. | |
42 | ||
43 | kernel.traceoff_on_warning | |
44 | Stop tracing if WARN_ON() occurs. | |
45 | ||
46 | kernel.fgraph_max_depth = MAX_DEPTH | |
47 | Set MAX_DEPTH to maximum depth of fgraph tracer. | |
48 | ||
49 | kernel.fgraph_filters = FILTER[, FILTER2...] | |
50 | Add fgraph tracing function filters. | |
51 | ||
52 | kernel.fgraph_notraces = FILTER[, FILTER2...] | |
b8381ce7 | 53 | Add fgraph non-tracing function filters. |
47781947 MH |
54 | |
55 | ||
56 | Ftrace Per-instance Options | |
57 | --------------------------- | |
58 | ||
59 | These options can be used for each instance including global ftrace node. | |
60 | ||
61 | ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]] | |
62 | Enable given ftrace options. | |
63 | ||
c51ba517 MH |
64 | ftrace.[instance.INSTANCE.]tracing_on = 0|1 |
65 | Enable/Disable tracing on this instance when starting boot-time tracing. | |
66 | (you can enable it by the "traceon" event trigger action) | |
67 | ||
47781947 MH |
68 | ftrace.[instance.INSTANCE.]trace_clock = CLOCK |
69 | Set given CLOCK to ftrace's trace_clock. | |
70 | ||
71 | ftrace.[instance.INSTANCE.]buffer_size = SIZE | |
72 | Configure ftrace buffer size to SIZE. You can use "KB" or "MB" | |
73 | for that SIZE. | |
74 | ||
75 | ftrace.[instance.INSTANCE.]alloc_snapshot | |
76 | Allocate snapshot buffer. | |
77 | ||
78 | ftrace.[instance.INSTANCE.]cpumask = CPUMASK | |
79 | Set CPUMASK as trace cpu-mask. | |
80 | ||
81 | ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2[...]] | |
82 | Enable given events on boot. You can use a wild card in EVENT. | |
83 | ||
84 | ftrace.[instance.INSTANCE.]tracer = TRACER | |
85 | Set TRACER to current tracer on boot. (e.g. function) | |
86 | ||
87 | ftrace.[instance.INSTANCE.]ftrace.filters | |
b8381ce7 | 88 | This will take an array of tracing function filter rules. |
47781947 MH |
89 | |
90 | ftrace.[instance.INSTANCE.]ftrace.notraces | |
b8381ce7 | 91 | This will take an array of NON-tracing function filter rules. |
47781947 MH |
92 | |
93 | ||
94 | Ftrace Per-Event Options | |
95 | ------------------------ | |
96 | ||
97 | These options are setting per-event options. | |
98 | ||
99 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable | |
b8381ce7 | 100 | Enable GROUP:EVENT tracing. |
47781947 | 101 | |
ee0a0701 MH |
102 | ftrace.[instance.INSTANCE.]event.GROUP.enable |
103 | Enable all event tracing within GROUP. | |
104 | ||
105 | ftrace.[instance.INSTANCE.]event.enable | |
106 | Enable all event tracing. | |
107 | ||
47781947 MH |
108 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER |
109 | Set FILTER rule to the GROUP:EVENT. | |
110 | ||
111 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2[...]] | |
112 | Set ACTIONs to the GROUP:EVENT. | |
113 | ||
114 | ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2[...]] | |
115 | Defines new kprobe event based on PROBEs. It is able to define | |
116 | multiple probes on one event, but those must have same type of | |
117 | arguments. This option is available only for the event which | |
118 | group name is "kprobes". | |
119 | ||
120 | ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]] | |
121 | Defines new synthetic event with FIELDs. Each field should be | |
122 | "type varname". | |
123 | ||
124 | Note that kprobe and synthetic event definitions can be written under | |
125 | instance node, but those are also visible from other instances. So please | |
126 | take care for event name conflict. | |
127 | ||
55978953 MH |
128 | Ftrace Histogram Options |
129 | ------------------------ | |
130 | ||
131 | Since it is too long to write a histogram action as a string for per-event | |
132 | action option, there are tree-style options under per-event 'hist' subkey | |
133 | for the histogram actions. For the detail of the each parameter, | |
134 | please read the event histogram document [3]_. | |
135 | ||
136 | .. [3] See :ref:`Documentation/trace/histogram.rst <histogram>` | |
137 | ||
138 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]keys = KEY1[, KEY2[...]] | |
139 | Set histogram key parameters. (Mandatory) | |
140 | The 'N' is a digit string for the multiple histogram. You can omit it | |
141 | if there is one histogram on the event. | |
142 | ||
143 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]values = VAL1[, VAL2[...]] | |
144 | Set histogram value parameters. | |
145 | ||
146 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]sort = SORT1[, SORT2[...]] | |
147 | Set histogram sort parameter options. | |
148 | ||
149 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]size = NR_ENTRIES | |
150 | Set histogram size (number of entries). | |
151 | ||
152 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]name = NAME | |
153 | Set histogram name. | |
154 | ||
155 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]var.VARIABLE = EXPR | |
156 | Define a new VARIABLE by EXPR expression. | |
157 | ||
158 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<pause|continue|clear> | |
159 | Set histogram control parameter. You can set one of them. | |
160 | ||
161 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]event = GROUP.EVENT | |
162 | Set histogram 'onmatch' handler matching event parameter. | |
163 | The 'M' is a digit string for the multiple 'onmatch' handler. You can omit it | |
164 | if there is one 'onmatch' handler on this histogram. | |
165 | ||
166 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]trace = EVENT[, ARG1[...]] | |
167 | Set histogram 'trace' action for 'onmatch'. | |
168 | EVENT must be a synthetic event name, and ARG1... are parameters | |
169 | for that event. Mandatory if 'onmatch.event' option is set. | |
170 | ||
171 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmax.[M.]var = VAR | |
172 | Set histogram 'onmax' handler variable parameter. | |
173 | ||
174 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onchange.[M.]var = VAR | |
175 | Set histogram 'onchange' handler variable parameter. | |
176 | ||
177 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]save = ARG1[, ARG2[...]] | |
178 | Set histogram 'save' action parameters for 'onmax' or 'onchange' handler. | |
179 | This option or below 'snapshot' option is mandatory if 'onmax.var' or | |
180 | 'onchange.var' option is set. | |
181 | ||
182 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]snapshot | |
183 | Set histogram 'snapshot' action for 'onmax' or 'onchange' handler. | |
184 | This option or above 'save' option is mandatory if 'onmax.var' or | |
185 | 'onchange.var' option is set. | |
186 | ||
187 | ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.filter = FILTER_EXPR | |
188 | Set histogram filter expression. You don't need 'if' in the FILTER_EXPR. | |
189 | ||
190 | Note that this 'hist' option can conflict with the per-event 'actions' | |
191 | option if the 'actions' option has a histogram action. | |
192 | ||
47781947 | 193 | |
fd264ce9 MH |
194 | When to Start |
195 | ============= | |
196 | ||
197 | All boot-time tracing options starting with ``ftrace`` will be enabled at the | |
198 | end of core_initcall. This means you can trace the events from postcore_initcall. | |
199 | Most of the subsystems and architecture dependent drivers will be initialized | |
200 | after that (arch_initcall or subsys_initcall). Thus, you can trace those with | |
201 | boot-time tracing. | |
202 | If you want to trace events before core_initcall, you can use the options | |
203 | starting with ``kernel``. Some of them will be enabled eariler than the initcall | |
204 | processing (for example,. ``kernel.ftrace=function`` and ``kernel.trace_event`` | |
205 | will start before the initcall.) | |
206 | ||
207 | ||
47781947 MH |
208 | Examples |
209 | ======== | |
210 | ||
211 | For example, to add filter and actions for each event, define kprobe | |
212 | events, and synthetic events with histogram, write a boot config like | |
213 | below:: | |
214 | ||
215 | ftrace.event { | |
216 | task.task_newtask { | |
217 | filter = "pid < 128" | |
218 | enable | |
219 | } | |
220 | kprobes.vfs_read { | |
221 | probes = "vfs_read $arg1 $arg2" | |
222 | filter = "common_pid < 200" | |
223 | enable | |
224 | } | |
225 | synthetic.initcall_latency { | |
226 | fields = "unsigned long func", "u64 lat" | |
55978953 MH |
227 | hist { |
228 | keys = func.sym, lat | |
229 | values = lat | |
230 | sort = lat | |
231 | } | |
47781947 | 232 | } |
55978953 MH |
233 | initcall.initcall_start.hist { |
234 | keys = func | |
235 | var.ts0 = common_timestamp.usecs | |
47781947 | 236 | } |
55978953 MH |
237 | initcall.initcall_finish.hist { |
238 | keys = func | |
239 | var.lat = common_timestamp.usecs - $ts0 | |
240 | onmatch { | |
241 | event = initcall.initcall_start | |
242 | trace = initcall_latency, func, $lat | |
243 | } | |
47781947 MH |
244 | } |
245 | } | |
246 | ||
b8381ce7 | 247 | Also, boot-time tracing supports "instance" node, which allows us to run |
47781947 | 248 | several tracers for different purpose at once. For example, one tracer |
b8381ce7 MH |
249 | is for tracing functions starting with "user\_", and others tracing |
250 | "kernel\_" functions, you can write boot config as below:: | |
47781947 MH |
251 | |
252 | ftrace.instance { | |
253 | foo { | |
254 | tracer = "function" | |
255 | ftrace.filters = "user_*" | |
256 | } | |
257 | bar { | |
258 | tracer = "function" | |
259 | ftrace.filters = "kernel_*" | |
260 | } | |
261 | } | |
262 | ||
263 | The instance node also accepts event nodes so that each instance | |
264 | can customize its event tracing. | |
265 | ||
19c311ae MH |
266 | With the trigger action and kprobes, you can trace function-graph while |
267 | a function is called. For example, this will trace all function calls in | |
268 | the pci_proc_init():: | |
269 | ||
270 | ftrace { | |
271 | tracing_on = 0 | |
272 | tracer = function_graph | |
273 | event.kprobes { | |
274 | start_event { | |
275 | probes = "pci_proc_init" | |
276 | actions = "traceon" | |
277 | } | |
278 | end_event { | |
279 | probes = "pci_proc_init%return" | |
280 | actions = "traceoff" | |
281 | } | |
282 | } | |
283 | } | |
284 | ||
285 | ||
47781947 MH |
286 | This boot-time tracing also supports ftrace kernel parameters via boot |
287 | config. | |
288 | For example, following kernel parameters:: | |
289 | ||
290 | trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*" | |
291 | ||
292 | This can be written in boot config like below:: | |
293 | ||
294 | kernel { | |
295 | trace_options = sym-addr | |
296 | trace_event = "initcall:*" | |
297 | tp_printk | |
298 | trace_buf_size = 1M | |
299 | ftrace = function | |
300 | ftrace_filter = "vfs*" | |
301 | } | |
302 | ||
303 | Note that parameters start with "kernel" prefix instead of "ftrace". |