Commit | Line | Data |
---|---|---|
7d4e3517 MCC |
1 | Dynamic debug |
2 | +++++++++++++ | |
3 | ||
86151fdf JB |
4 | |
5 | Introduction | |
6 | ============ | |
7 | ||
29e36c9f | 8 | This document describes how to use the dynamic debug (dyndbg) feature. |
86151fdf | 9 | |
29e36c9f JC |
10 | Dynamic debug is designed to allow you to dynamically enable/disable |
11 | kernel code to obtain additional kernel information. Currently, if | |
7d4e3517 MCC |
12 | ``CONFIG_DYNAMIC_DEBUG`` is set, then all ``pr_debug()``/``dev_dbg()`` and |
13 | ``print_hex_dump_debug()``/``print_hex_dump_bytes()`` calls can be dynamically | |
7a555613 VK |
14 | enabled per-callsite. |
15 | ||
ceabef7d OZ |
16 | If you do not want to enable dynamic debug globally (i.e. in some embedded |
17 | system), you may set ``CONFIG_DYNAMIC_DEBUG_CORE`` as basic support of dynamic | |
18 | debug and add ``ccflags := -DDYNAMIC_DEBUG_MODULE`` into the Makefile of any | |
19 | modules which you'd like to dynamically debug later. | |
20 | ||
7d4e3517 MCC |
21 | If ``CONFIG_DYNAMIC_DEBUG`` is not set, ``print_hex_dump_debug()`` is just |
22 | shortcut for ``print_hex_dump(KERN_DEBUG)``. | |
7a555613 | 23 | |
7d4e3517 MCC |
24 | For ``print_hex_dump_debug()``/``print_hex_dump_bytes()``, format string is |
25 | its ``prefix_str`` argument, if it is constant string; or ``hexdump`` | |
8c188759 | 26 | in case ``prefix_str`` is built dynamically. |
86151fdf JB |
27 | |
28 | Dynamic debug has even more useful features: | |
29 | ||
29e36c9f JC |
30 | * Simple query language allows turning on and off debugging |
31 | statements by matching any combination of 0 or 1 of: | |
86151fdf JB |
32 | |
33 | - source filename | |
34 | - function name | |
35 | - line number (including ranges of line numbers) | |
36 | - module name | |
37 | - format string | |
38 | ||
7d4e3517 | 39 | * Provides a debugfs control file: ``<debugfs>/dynamic_debug/control`` |
29e36c9f JC |
40 | which can be read to display the complete list of known debug |
41 | statements, to help guide you | |
86151fdf JB |
42 | |
43 | Controlling dynamic debug Behaviour | |
a648ec05 | 44 | =================================== |
86151fdf | 45 | |
7d4e3517 | 46 | The behaviour of ``pr_debug()``/``dev_dbg()`` are controlled via writing to a |
29e36c9f JC |
47 | control file in the 'debugfs' filesystem. Thus, you must first mount |
48 | the debugfs filesystem, in order to make use of this feature. | |
49 | Subsequently, we refer to the control file as: | |
7d4e3517 MCC |
50 | ``<debugfs>/dynamic_debug/control``. For example, if you want to enable |
51 | printing from source file ``svcsock.c``, line 1603 you simply do:: | |
86151fdf | 52 | |
7d4e3517 | 53 | nullarbor:~ # echo 'file svcsock.c line 1603 +p' > |
86151fdf JB |
54 | <debugfs>/dynamic_debug/control |
55 | ||
7d4e3517 | 56 | If you make a mistake with the syntax, the write will fail thus:: |
86151fdf | 57 | |
7d4e3517 | 58 | nullarbor:~ # echo 'file svcsock.c wtf 1 +p' > |
86151fdf | 59 | <debugfs>/dynamic_debug/control |
7d4e3517 | 60 | -bash: echo: write error: Invalid argument |
86151fdf | 61 | |
239a5791 GKH |
62 | Note, for systems without 'debugfs' enabled, the control file can be |
63 | found in ``/proc/dynamic_debug/control``. | |
64 | ||
86151fdf | 65 | Viewing Dynamic Debug Behaviour |
7d4e3517 | 66 | =============================== |
86151fdf | 67 | |
29e36c9f | 68 | You can view the currently configured behaviour of all the debug |
7d4e3517 | 69 | statements via:: |
86151fdf | 70 | |
7d4e3517 MCC |
71 | nullarbor:~ # cat <debugfs>/dynamic_debug/control |
72 | # filename:lineno [module]function flags format | |
73 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:323 [svcxprt_rdma]svc_rdma_cleanup =_ "SVCRDMA Module Removed, deregister RPC RDMA transport\012" | |
74 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:341 [svcxprt_rdma]svc_rdma_init =_ "\011max_inline : %d\012" | |
75 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:340 [svcxprt_rdma]svc_rdma_init =_ "\011sq_depth : %d\012" | |
76 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:338 [svcxprt_rdma]svc_rdma_init =_ "\011max_requests : %d\012" | |
77 | ... | |
86151fdf JB |
78 | |
79 | ||
80 | You can also apply standard Unix text manipulation filters to this | |
7d4e3517 | 81 | data, e.g.:: |
86151fdf | 82 | |
7d4e3517 MCC |
83 | nullarbor:~ # grep -i rdma <debugfs>/dynamic_debug/control | wc -l |
84 | 62 | |
86151fdf | 85 | |
7d4e3517 MCC |
86 | nullarbor:~ # grep -i tcp <debugfs>/dynamic_debug/control | wc -l |
87 | 42 | |
86151fdf | 88 | |
29e36c9f JC |
89 | The third column shows the currently enabled flags for each debug |
90 | statement callsite (see below for definitions of the flags). The | |
7d4e3517 MCC |
91 | default value, with no flags enabled, is ``=_``. So you can view all |
92 | the debug statement callsites with any non-default flags:: | |
86151fdf | 93 | |
7d4e3517 MCC |
94 | nullarbor:~ # awk '$3 != "=_"' <debugfs>/dynamic_debug/control |
95 | # filename:lineno [module]function flags format | |
96 | /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c:1603 [sunrpc]svc_send p "svc_process: st_sendto returned %d\012" | |
86151fdf JB |
97 | |
98 | Command Language Reference | |
99 | ========================== | |
100 | ||
101 | At the lexical level, a command comprises a sequence of words separated | |
7d4e3517 | 102 | by spaces or tabs. So these are all equivalent:: |
86151fdf | 103 | |
31fc93d5 | 104 | nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > |
86151fdf | 105 | <debugfs>/dynamic_debug/control |
31fc93d5 | 106 | nullarbor:~ # echo -n ' file svcsock.c line 1603 +p ' > |
86151fdf | 107 | <debugfs>/dynamic_debug/control |
7d4e3517 | 108 | nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > |
86151fdf JB |
109 | <debugfs>/dynamic_debug/control |
110 | ||
85f7f6c0 | 111 | Command submissions are bounded by a write() system call. |
7d4e3517 | 112 | Multiple commands can be written together, separated by ``;`` or ``\n``:: |
86151fdf | 113 | |
85f7f6c0 JC |
114 | ~# echo "func pnpacpi_get_resources +p; func pnp_assign_mem +p" \ |
115 | > <debugfs>/dynamic_debug/control | |
86151fdf | 116 | |
7d4e3517 | 117 | If your query set is big, you can batch them too:: |
86151fdf | 118 | |
85f7f6c0 | 119 | ~# cat query-batch-file > <debugfs>/dynamic_debug/control |
86151fdf | 120 | |
1afc5fb5 RD |
121 | Another way is to use wildcards. The match rule supports ``*`` (matches |
122 | zero or more characters) and ``?`` (matches exactly one character). For | |
7d4e3517 | 123 | example, you can match all usb drivers:: |
8f073bd0 DC |
124 | |
125 | ~# echo "file drivers/usb/* +p" > <debugfs>/dynamic_debug/control | |
126 | ||
86151fdf | 127 | At the syntactical level, a command comprises a sequence of match |
7d4e3517 | 128 | specifications, followed by a flags change specification:: |
86151fdf | 129 | |
7d4e3517 | 130 | command ::= match-spec* flags-spec |
86151fdf | 131 | |
29e36c9f | 132 | The match-spec's are used to choose a subset of the known pr_debug() |
86151fdf JB |
133 | callsites to which to apply the flags-spec. Think of them as a query |
134 | with implicit ANDs between each pair. Note that an empty list of | |
29e36c9f | 135 | match-specs will select all debug statement callsites. |
86151fdf | 136 | |
29e36c9f JC |
137 | A match specification comprises a keyword, which controls the |
138 | attribute of the callsite to be compared, and a value to compare | |
7d4e3517 MCC |
139 | against. Possible keywords are::: |
140 | ||
141 | match-spec ::= 'func' string | | |
142 | 'file' string | | |
143 | 'module' string | | |
144 | 'format' string | | |
145 | 'line' line-range | |
86151fdf | 146 | |
7d4e3517 MCC |
147 | line-range ::= lineno | |
148 | '-'lineno | | |
149 | lineno'-' | | |
150 | lineno'-'lineno | |
86151fdf | 151 | |
7d4e3517 MCC |
152 | lineno ::= unsigned-int |
153 | ||
154 | .. note:: | |
155 | ||
156 | ``line-range`` cannot contain space, e.g. | |
157 | "1-30" is valid range but "1 - 30" is not. | |
86151fdf | 158 | |
86151fdf JB |
159 | |
160 | The meanings of each keyword are: | |
161 | ||
162 | func | |
163 | The given string is compared against the function name | |
7d4e3517 | 164 | of each callsite. Example:: |
86151fdf | 165 | |
7d4e3517 | 166 | func svc_tcp_accept |
86151fdf JB |
167 | |
168 | file | |
2b678319 JC |
169 | The given string is compared against either the full pathname, the |
170 | src-root relative pathname, or the basename of the source file of | |
7d4e3517 | 171 | each callsite. Examples:: |
86151fdf | 172 | |
7d4e3517 MCC |
173 | file svcsock.c |
174 | file kernel/freezer.c | |
175 | file /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c | |
86151fdf JB |
176 | |
177 | module | |
178 | The given string is compared against the module name | |
179 | of each callsite. The module name is the string as | |
7d4e3517 MCC |
180 | seen in ``lsmod``, i.e. without the directory or the ``.ko`` |
181 | suffix and with ``-`` changed to ``_``. Examples:: | |
86151fdf | 182 | |
7d4e3517 MCC |
183 | module sunrpc |
184 | module nfsd | |
86151fdf JB |
185 | |
186 | format | |
187 | The given string is searched for in the dynamic debug format | |
188 | string. Note that the string does not need to match the | |
189 | entire format, only some part. Whitespace and other | |
190 | special characters can be escaped using C octal character | |
7d4e3517 | 191 | escape ``\ooo`` notation, e.g. the space character is ``\040``. |
9898abb3 | 192 | Alternatively, the string can be enclosed in double quote |
7d4e3517 MCC |
193 | characters (``"``) or single quote characters (``'``). |
194 | Examples:: | |
86151fdf | 195 | |
7d4e3517 MCC |
196 | format svcrdma: // many of the NFS/RDMA server pr_debugs |
197 | format readahead // some pr_debugs in the readahead cache | |
198 | format nfsd:\040SETATTR // one way to match a format with whitespace | |
199 | format "nfsd: SETATTR" // a neater way to match a format with whitespace | |
200 | format 'nfsd: SETATTR' // yet another way to match a format with whitespace | |
86151fdf JB |
201 | |
202 | line | |
203 | The given line number or range of line numbers is compared | |
7d4e3517 | 204 | against the line number of each ``pr_debug()`` callsite. A single |
86151fdf JB |
205 | line number matches the callsite line number exactly. A |
206 | range of line numbers matches any callsite between the first | |
207 | and last line number inclusive. An empty first number means | |
8c188759 RD |
208 | the first line in the file, an empty last line number means the |
209 | last line number in the file. Examples:: | |
86151fdf | 210 | |
7d4e3517 MCC |
211 | line 1603 // exactly line 1603 |
212 | line 1600-1605 // the six lines from line 1600 to line 1605 | |
213 | line -1605 // the 1605 lines from line 1 to line 1605 | |
214 | line 1600- // all lines from line 1600 to the end of the file | |
86151fdf JB |
215 | |
216 | The flags specification comprises a change operation followed | |
217 | by one or more flag characters. The change operation is one | |
7d4e3517 | 218 | of the characters:: |
86151fdf | 219 | |
29e36c9f JC |
220 | - remove the given flags |
221 | + add the given flags | |
222 | = set the flags to the given flags | |
86151fdf | 223 | |
7d4e3517 | 224 | The flags are:: |
86151fdf | 225 | |
29e36c9f JC |
226 | p enables the pr_debug() callsite. |
227 | f Include the function name in the printed message | |
228 | l Include line number in the printed message | |
229 | m Include module name in the printed message | |
230 | t Include thread ID in messages not generated from interrupt context | |
231 | _ No flags are set. (Or'd with others on input) | |
232 | ||
7d4e3517 | 233 | For ``print_hex_dump_debug()`` and ``print_hex_dump_bytes()``, only ``p`` flag |
7a555613 VK |
234 | have meaning, other flags ignored. |
235 | ||
7d4e3517 | 236 | For display, the flags are preceded by ``=`` |
29e36c9f | 237 | (mnemonic: what the flags are currently equal to). |
86151fdf | 238 | |
7d4e3517 MCC |
239 | Note the regexp ``^[-+=][flmpt_]+$`` matches a flags specification. |
240 | To clear all flags at once, use ``=_`` or ``-flmpt``. | |
86151fdf | 241 | |
a648ec05 | 242 | |
29e36c9f | 243 | Debug messages during Boot Process |
a648ec05 TR |
244 | ================================== |
245 | ||
29e36c9f JC |
246 | To activate debug messages for core code and built-in modules during |
247 | the boot process, even before userspace and debugfs exists, use | |
7d4e3517 MCC |
248 | ``dyndbg="QUERY"``, ``module.dyndbg="QUERY"``, or ``ddebug_query="QUERY"`` |
249 | (``ddebug_query`` is obsoleted by ``dyndbg``, and deprecated). QUERY follows | |
29e36c9f JC |
250 | the syntax described above, but must not exceed 1023 characters. Your |
251 | bootloader may impose lower limits. | |
252 | ||
7d4e3517 | 253 | These ``dyndbg`` params are processed just after the ddebug tables are |
29e36c9f JC |
254 | processed, as part of the arch_initcall. Thus you can enable debug |
255 | messages in all code run after this arch_initcall via this boot | |
256 | parameter. | |
a648ec05 | 257 | |
7d4e3517 MCC |
258 | On an x86 system for example ACPI enablement is a subsys_initcall and:: |
259 | ||
29e36c9f | 260 | dyndbg="file ec.c +p" |
7d4e3517 | 261 | |
a648ec05 TR |
262 | will show early Embedded Controller transactions during ACPI setup if |
263 | your machine (typically a laptop) has an Embedded Controller. | |
264 | PCI (or other devices) initialization also is a hot candidate for using | |
265 | this boot parameter for debugging purposes. | |
266 | ||
7d4e3517 | 267 | If ``foo`` module is not built-in, ``foo.dyndbg`` will still be processed at |
29e36c9f | 268 | boot time, without effect, but will be reprocessed when module is |
005ae6df | 269 | loaded later. ``ddebug_query=`` and bare ``dyndbg=`` are only processed at |
29e36c9f JC |
270 | boot. |
271 | ||
272 | ||
273 | Debug Messages at Module Initialization Time | |
274 | ============================================ | |
275 | ||
7d4e3517 MCC |
276 | When ``modprobe foo`` is called, modprobe scans ``/proc/cmdline`` for |
277 | ``foo.params``, strips ``foo.``, and passes them to the kernel along with | |
278 | params given in modprobe args or ``/etc/modprob.d/*.conf`` files, | |
29e36c9f JC |
279 | in the following order: |
280 | ||
7d4e3517 MCC |
281 | 1. parameters given via ``/etc/modprobe.d/*.conf``:: |
282 | ||
283 | options foo dyndbg=+pt | |
284 | options foo dyndbg # defaults to +p | |
285 | ||
286 | 2. ``foo.dyndbg`` as given in boot args, ``foo.`` is stripped and passed:: | |
29e36c9f | 287 | |
7d4e3517 | 288 | foo.dyndbg=" func bar +p; func buz +mp" |
29e36c9f | 289 | |
7d4e3517 | 290 | 3. args to modprobe:: |
29e36c9f | 291 | |
7d4e3517 MCC |
292 | modprobe foo dyndbg==pmf # override previous settings |
293 | ||
294 | These ``dyndbg`` queries are applied in order, with last having final say. | |
295 | This allows boot args to override or modify those from ``/etc/modprobe.d`` | |
29e36c9f JC |
296 | (sensible, since 1 is system wide, 2 is kernel or boot specific), and |
297 | modprobe args to override both. | |
298 | ||
7d4e3517 MCC |
299 | In the ``foo.dyndbg="QUERY"`` form, the query must exclude ``module foo``. |
300 | ``foo`` is extracted from the param-name, and applied to each query in | |
301 | ``QUERY``, and only 1 match-spec of each type is allowed. | |
29e36c9f | 302 | |
7d4e3517 | 303 | The ``dyndbg`` option is a "fake" module parameter, which means: |
29e36c9f JC |
304 | |
305 | - modules do not need to define it explicitly | |
306 | - every module gets it tacitly, whether they use pr_debug or not | |
7d4e3517 MCC |
307 | - it doesn't appear in ``/sys/module/$module/parameters/`` |
308 | To see it, grep the control file, or inspect ``/proc/cmdline.`` | |
29e36c9f | 309 | |
7d4e3517 MCC |
310 | For ``CONFIG_DYNAMIC_DEBUG`` kernels, any settings given at boot-time (or |
311 | enabled by ``-DDEBUG`` flag during compilation) can be disabled later via | |
005ae6df | 312 | the debugfs interface if the debug messages are no longer needed:: |
29e36c9f JC |
313 | |
314 | echo "module module_name -p" > <debugfs>/dynamic_debug/control | |
a648ec05 | 315 | |
86151fdf JB |
316 | Examples |
317 | ======== | |
318 | ||
7d4e3517 MCC |
319 | :: |
320 | ||
321 | // enable the message at line 1603 of file svcsock.c | |
322 | nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > | |
86151fdf JB |
323 | <debugfs>/dynamic_debug/control |
324 | ||
7d4e3517 MCC |
325 | // enable all the messages in file svcsock.c |
326 | nullarbor:~ # echo -n 'file svcsock.c +p' > | |
86151fdf JB |
327 | <debugfs>/dynamic_debug/control |
328 | ||
7d4e3517 MCC |
329 | // enable all the messages in the NFS server module |
330 | nullarbor:~ # echo -n 'module nfsd +p' > | |
86151fdf JB |
331 | <debugfs>/dynamic_debug/control |
332 | ||
7d4e3517 MCC |
333 | // enable all 12 messages in the function svc_process() |
334 | nullarbor:~ # echo -n 'func svc_process +p' > | |
86151fdf JB |
335 | <debugfs>/dynamic_debug/control |
336 | ||
7d4e3517 MCC |
337 | // disable all 12 messages in the function svc_process() |
338 | nullarbor:~ # echo -n 'func svc_process -p' > | |
86151fdf | 339 | <debugfs>/dynamic_debug/control |
9898abb3 | 340 | |
7d4e3517 MCC |
341 | // enable messages for NFS calls READ, READLINK, READDIR and READDIR+. |
342 | nullarbor:~ # echo -n 'format "nfsd: READ" +p' > | |
9898abb3 | 343 | <debugfs>/dynamic_debug/control |
29e36c9f | 344 | |
7d4e3517 MCC |
345 | // enable messages in files of which the paths include string "usb" |
346 | nullarbor:~ # echo -n '*usb* +p' > <debugfs>/dynamic_debug/control | |
8f073bd0 | 347 | |
7d4e3517 MCC |
348 | // enable all messages |
349 | nullarbor:~ # echo -n '+p' > <debugfs>/dynamic_debug/control | |
29e36c9f | 350 | |
7d4e3517 MCC |
351 | // add module, function to all enabled messages |
352 | nullarbor:~ # echo -n '+mf' > <debugfs>/dynamic_debug/control | |
29e36c9f | 353 | |
7d4e3517 MCC |
354 | // boot-args example, with newlines and comments for readability |
355 | Kernel command line: ... | |
356 | // see whats going on in dyndbg=value processing | |
357 | dynamic_debug.verbose=1 | |
358 | // enable pr_debugs in 2 builtins, #cmt is stripped | |
359 | dyndbg="module params +p #cmt ; module sys +p" | |
360 | // enable pr_debugs in 2 functions in a module loaded later | |
361 | pc87360.dyndbg="func pc87360_init_device +p; func pc87360_find +p" |