Commit | Line | Data |
---|---|---|
7fb2e8a4 MCC |
1 | ================================================= |
2 | Using kgdb, kdb and the kernel debugger internals | |
3 | ================================================= | |
4 | ||
5 | :Author: Jason Wessel | |
6 | ||
7 | Introduction | |
8 | ============ | |
9 | ||
10 | The kernel has two different debugger front ends (kdb and kgdb) which | |
11 | interface to the debug core. It is possible to use either of the | |
12 | debugger front ends and dynamically transition between them if you | |
13 | configure the kernel properly at compile and runtime. | |
14 | ||
15 | Kdb is simplistic shell-style interface which you can use on a system | |
16 | console with a keyboard or serial console. You can use it to inspect | |
17 | memory, registers, process lists, dmesg, and even set breakpoints to | |
18 | stop in a certain location. Kdb is not a source level debugger, although | |
19 | you can set breakpoints and execute some basic kernel run control. Kdb | |
20 | is mainly aimed at doing some analysis to aid in development or | |
21 | diagnosing kernel problems. You can access some symbols by name in | |
22 | kernel built-ins or in kernel modules if the code was built with | |
23 | ``CONFIG_KALLSYMS``. | |
24 | ||
25 | Kgdb is intended to be used as a source level debugger for the Linux | |
26 | kernel. It is used along with gdb to debug a Linux kernel. The | |
27 | expectation is that gdb can be used to "break in" to the kernel to | |
28 | inspect memory, variables and look through call stack information | |
29 | similar to the way an application developer would use gdb to debug an | |
30 | application. It is possible to place breakpoints in kernel code and | |
31 | perform some limited execution stepping. | |
32 | ||
33 | Two machines are required for using kgdb. One of these machines is a | |
34 | development machine and the other is the target machine. The kernel to | |
35 | be debugged runs on the target machine. The development machine runs an | |
36 | instance of gdb against the vmlinux file which contains the symbols (not | |
37 | a boot image such as bzImage, zImage, uImage...). In gdb the developer | |
38 | specifies the connection parameters and connects to kgdb. The type of | |
39 | connection a developer makes with gdb depends on the availability of | |
40 | kgdb I/O modules compiled as built-ins or loadable kernel modules in the | |
41 | test machine's kernel. | |
42 | ||
43 | Compiling a kernel | |
44 | ================== | |
45 | ||
46 | - In order to enable compilation of kdb, you must first enable kgdb. | |
47 | ||
48 | - The kgdb test compile options are described in the kgdb test suite | |
49 | chapter. | |
50 | ||
51 | Kernel config options for kgdb | |
52 | ------------------------------ | |
53 | ||
821c6df8 MCC |
54 | To enable ``CONFIG_KGDB`` you should look under |
55 | :menuselection:`Kernel hacking --> Kernel debugging` and select | |
56 | :menuselection:`KGDB: kernel debugger`. | |
7fb2e8a4 MCC |
57 | |
58 | While it is not a hard requirement that you have symbols in your vmlinux | |
59 | file, gdb tends not to be very useful without the symbolic data, so you | |
821c6df8 MCC |
60 | will want to turn on ``CONFIG_DEBUG_INFO`` which is called |
61 | :menuselection:`Compile the kernel with debug info` in the config menu. | |
7fb2e8a4 MCC |
62 | |
63 | It is advised, but not required, that you turn on the | |
821c6df8 MCC |
64 | ``CONFIG_FRAME_POINTER`` kernel option which is called :menuselection:`Compile |
65 | the kernel with frame pointers` in the config menu. This option inserts code | |
c12af30b TY |
66 | into the compiled executable which saves the frame information in registers |
67 | or on the stack at different points which allows a debugger such as gdb to | |
68 | more accurately construct stack back traces while debugging the kernel. | |
7fb2e8a4 MCC |
69 | |
70 | If the architecture that you are using supports the kernel option | |
821c6df8 | 71 | ``CONFIG_STRICT_KERNEL_RWX``, you should consider turning it off. This |
7fb2e8a4 MCC |
72 | option will prevent the use of software breakpoints because it marks |
73 | certain regions of the kernel's memory space as read-only. If kgdb | |
74 | supports it for the architecture you are using, you can use hardware | |
821c6df8 | 75 | breakpoints if you desire to run with the ``CONFIG_STRICT_KERNEL_RWX`` |
7fb2e8a4 MCC |
76 | option turned on, else you need to turn off this option. |
77 | ||
78 | Next you should choose one of more I/O drivers to interconnect debugging | |
79 | host and debugged target. Early boot debugging requires a KGDB I/O | |
80 | driver that supports early debugging and the driver must be built into | |
81 | the kernel directly. Kgdb I/O driver configuration takes place via | |
82 | kernel or module parameters which you can learn more about in the in the | |
821c6df8 | 83 | section that describes the parameter kgdboc. |
7fb2e8a4 | 84 | |
821c6df8 | 85 | Here is an example set of ``.config`` symbols to enable or disable for kgdb:: |
7fb2e8a4 | 86 | |
821c6df8 MCC |
87 | # CONFIG_STRICT_KERNEL_RWX is not set |
88 | CONFIG_FRAME_POINTER=y | |
89 | CONFIG_KGDB=y | |
90 | CONFIG_KGDB_SERIAL_CONSOLE=y | |
7fb2e8a4 MCC |
91 | |
92 | Kernel config options for kdb | |
93 | ----------------------------- | |
94 | ||
95 | Kdb is quite a bit more complex than the simple gdbstub sitting on top | |
96 | of the kernel's debug core. Kdb must implement a shell, and also adds | |
97 | some helper functions in other parts of the kernel, responsible for | |
98 | printing out interesting data such as what you would see if you ran | |
821c6df8 | 99 | ``lsmod``, or ``ps``. In order to build kdb into the kernel you follow the |
7fb2e8a4 MCC |
100 | same steps as you would for kgdb. |
101 | ||
102 | The main config option for kdb is ``CONFIG_KGDB_KDB`` which is called | |
821c6df8 MCC |
103 | :menuselection:`KGDB_KDB: include kdb frontend for kgdb` in the config menu. |
104 | In theory you would have already also selected an I/O driver such as the | |
105 | ``CONFIG_KGDB_SERIAL_CONSOLE`` interface if you plan on using kdb on a | |
7fb2e8a4 MCC |
106 | serial port, when you were configuring kgdb. |
107 | ||
108 | If you want to use a PS/2-style keyboard with kdb, you would select | |
821c6df8 MCC |
109 | ``CONFIG_KDB_KEYBOARD`` which is called :menuselection:`KGDB_KDB: keyboard as |
110 | input device` in the config menu. The ``CONFIG_KDB_KEYBOARD`` option is not | |
111 | used for anything in the gdb interface to kgdb. The ``CONFIG_KDB_KEYBOARD`` | |
7fb2e8a4 MCC |
112 | option only works with kdb. |
113 | ||
821c6df8 | 114 | Here is an example set of ``.config`` symbols to enable/disable kdb:: |
7fb2e8a4 | 115 | |
821c6df8 MCC |
116 | # CONFIG_STRICT_KERNEL_RWX is not set |
117 | CONFIG_FRAME_POINTER=y | |
118 | CONFIG_KGDB=y | |
119 | CONFIG_KGDB_SERIAL_CONSOLE=y | |
120 | CONFIG_KGDB_KDB=y | |
121 | CONFIG_KDB_KEYBOARD=y | |
7fb2e8a4 MCC |
122 | |
123 | Kernel Debugger Boot Arguments | |
124 | ============================== | |
125 | ||
126 | This section describes the various runtime kernel parameters that affect | |
127 | the configuration of the kernel debugger. The following chapter covers | |
128 | using kdb and kgdb as well as providing some examples of the | |
129 | configuration parameters. | |
130 | ||
131 | Kernel parameter: kgdboc | |
132 | ------------------------ | |
133 | ||
134 | The kgdboc driver was originally an abbreviation meant to stand for | |
135 | "kgdb over console". Today it is the primary mechanism to configure how | |
136 | to communicate from gdb to kgdb as well as the devices you want to use | |
137 | to interact with the kdb shell. | |
138 | ||
139 | For kgdb/gdb, kgdboc is designed to work with a single serial port. It | |
140 | is intended to cover the circumstance where you want to use a serial | |
141 | console as your primary console as well as using it to perform kernel | |
142 | debugging. It is also possible to use kgdb on a serial port which is not | |
143 | designated as a system console. Kgdboc may be configured as a kernel | |
144 | built-in or a kernel loadable module. You can only make use of | |
145 | ``kgdbwait`` and early debugging if you build kgdboc into the kernel as | |
146 | a built-in. | |
147 | ||
148 | Optionally you can elect to activate kms (Kernel Mode Setting) | |
149 | integration. When you use kms with kgdboc and you have a video driver | |
150 | that has atomic mode setting hooks, it is possible to enter the debugger | |
151 | on the graphics console. When the kernel execution is resumed, the | |
152 | previous graphics mode will be restored. This integration can serve as a | |
153 | useful tool to aid in diagnosing crashes or doing analysis of memory | |
154 | with kdb while allowing the full graphics console applications to run. | |
155 | ||
156 | kgdboc arguments | |
157 | ~~~~~~~~~~~~~~~~ | |
158 | ||
821c6df8 MCC |
159 | Usage:: |
160 | ||
161 | kgdboc=[kms][[,]kbd][[,]serial_device][,baud] | |
7fb2e8a4 MCC |
162 | |
163 | The order listed above must be observed if you use any of the optional | |
164 | configurations together. | |
165 | ||
166 | Abbreviations: | |
167 | ||
168 | - kms = Kernel Mode Setting | |
169 | ||
170 | - kbd = Keyboard | |
171 | ||
172 | You can configure kgdboc to use the keyboard, and/or a serial device | |
173 | depending on if you are using kdb and/or kgdb, in one of the following | |
174 | scenarios. The order listed above must be observed if you use any of the | |
175 | optional configurations together. Using kms + only gdb is generally not | |
176 | a useful combination. | |
177 | ||
178 | Using loadable module or built-in | |
179 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
180 | ||
181 | 1. As a kernel built-in: | |
182 | ||
821c6df8 MCC |
183 | Use the kernel boot argument:: |
184 | ||
185 | kgdboc=<tty-device>,[baud] | |
7fb2e8a4 MCC |
186 | |
187 | 2. As a kernel loadable module: | |
188 | ||
821c6df8 MCC |
189 | Use the command:: |
190 | ||
191 | modprobe kgdboc kgdboc=<tty-device>,[baud] | |
7fb2e8a4 MCC |
192 | |
193 | Here are two examples of how you might format the kgdboc string. The | |
194 | first is for an x86 target using the first serial port. The second | |
195 | example is for the ARM Versatile AB using the second serial port. | |
196 | ||
197 | 1. ``kgdboc=ttyS0,115200`` | |
198 | ||
199 | 2. ``kgdboc=ttyAMA1,115200`` | |
200 | ||
201 | Configure kgdboc at runtime with sysfs | |
202 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
203 | ||
204 | At run time you can enable or disable kgdboc by echoing a parameters | |
205 | into the sysfs. Here are two examples: | |
206 | ||
821c6df8 | 207 | 1. Enable kgdboc on ttyS0:: |
7fb2e8a4 | 208 | |
821c6df8 | 209 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 | 210 | |
821c6df8 | 211 | 2. Disable kgdboc:: |
7fb2e8a4 | 212 | |
821c6df8 | 213 | echo "" > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 | 214 | |
821c6df8 MCC |
215 | .. note:: |
216 | ||
217 | You do not need to specify the baud if you are configuring the | |
218 | console on tty which is already configured or open. | |
7fb2e8a4 MCC |
219 | |
220 | More examples | |
221 | ^^^^^^^^^^^^^ | |
222 | ||
223 | You can configure kgdboc to use the keyboard, and/or a serial device | |
224 | depending on if you are using kdb and/or kgdb, in one of the following | |
225 | scenarios. | |
226 | ||
821c6df8 MCC |
227 | 1. kdb and kgdb over only a serial port:: |
228 | ||
229 | kgdboc=<serial_device>[,baud] | |
230 | ||
231 | Example:: | |
7fb2e8a4 | 232 | |
821c6df8 | 233 | kgdboc=ttyS0,115200 |
7fb2e8a4 | 234 | |
821c6df8 | 235 | 2. kdb and kgdb with keyboard and a serial port:: |
7fb2e8a4 | 236 | |
821c6df8 | 237 | kgdboc=kbd,<serial_device>[,baud] |
7fb2e8a4 | 238 | |
821c6df8 | 239 | Example:: |
7fb2e8a4 | 240 | |
821c6df8 | 241 | kgdboc=kbd,ttyS0,115200 |
7fb2e8a4 | 242 | |
821c6df8 | 243 | 3. kdb with a keyboard:: |
7fb2e8a4 | 244 | |
821c6df8 | 245 | kgdboc=kbd |
7fb2e8a4 | 246 | |
821c6df8 | 247 | 4. kdb with kernel mode setting:: |
7fb2e8a4 | 248 | |
821c6df8 | 249 | kgdboc=kms,kbd |
7fb2e8a4 | 250 | |
821c6df8 | 251 | 5. kdb with kernel mode setting and kgdb over a serial port:: |
7fb2e8a4 | 252 | |
821c6df8 | 253 | kgdboc=kms,kbd,ttyS0,115200 |
7fb2e8a4 | 254 | |
821c6df8 MCC |
255 | .. note:: |
256 | ||
257 | Kgdboc does not support interrupting the target via the gdb remote | |
258 | protocol. You must manually send a :kbd:`SysRq-G` unless you have a proxy | |
259 | that splits console output to a terminal program. A console proxy has a | |
260 | separate TCP port for the debugger and a separate TCP port for the | |
261 | "human" console. The proxy can take care of sending the :kbd:`SysRq-G` | |
262 | for you. | |
7fb2e8a4 MCC |
263 | |
264 | When using kgdboc with no debugger proxy, you can end up connecting the | |
265 | debugger at one of two entry points. If an exception occurs after you | |
266 | have loaded kgdboc, a message should print on the console stating it is | |
267 | waiting for the debugger. In this case you disconnect your terminal | |
268 | program and then connect the debugger in its place. If you want to | |
269 | interrupt the target system and forcibly enter a debug session you have | |
821c6df8 | 270 | to issue a :kbd:`Sysrq` sequence and then type the letter :kbd:`g`. Then you |
7fb2e8a4 | 271 | disconnect the terminal session and connect gdb. Your options if you |
821c6df8 | 272 | don't like this are to hack gdb to send the :kbd:`SysRq-G` for you as well as |
7fb2e8a4 MCC |
273 | on the initial connect, or to use a debugger proxy that allows an |
274 | unmodified gdb to do the debugging. | |
275 | ||
f71fc3bc DA |
276 | Kernel parameter: ``kgdboc_earlycon`` |
277 | ------------------------------------- | |
278 | ||
279 | If you specify the kernel parameter ``kgdboc_earlycon`` and your serial | |
280 | driver registers a boot console that supports polling (doesn't need | |
281 | interrupts and implements a nonblocking read() function) kgdb will attempt | |
282 | to work using the boot console until it can transition to the regular | |
283 | tty driver specified by the ``kgdboc`` parameter. | |
284 | ||
285 | Normally there is only one boot console (especially that implements the | |
286 | read() function) so just adding ``kgdboc_earlycon`` on its own is | |
287 | sufficient to make this work. If you have more than one boot console you | |
288 | can add the boot console's name to differentiate. Note that names that | |
289 | are registered through the boot console layer and the tty layer are not | |
290 | the same for the same port. | |
291 | ||
292 | For instance, on one board to be explicit you might do:: | |
293 | ||
294 | kgdboc_earlycon=qcom_geni kgdboc=ttyMSM0 | |
295 | ||
296 | If the only boot console on the device was "qcom_geni", you could simplify:: | |
297 | ||
298 | kgdboc_earlycon kgdboc=ttyMSM0 | |
299 | ||
821c6df8 MCC |
300 | Kernel parameter: ``kgdbwait`` |
301 | ------------------------------ | |
7fb2e8a4 MCC |
302 | |
303 | The Kernel command line option ``kgdbwait`` makes kgdb wait for a | |
304 | debugger connection during booting of a kernel. You can only use this | |
305 | option if you compiled a kgdb I/O driver into the kernel and you | |
306 | specified the I/O driver configuration as a kernel command line option. | |
307 | The kgdbwait parameter should always follow the configuration parameter | |
308 | for the kgdb I/O driver in the kernel command line else the I/O driver | |
309 | will not be configured prior to asking the kernel to use it to wait. | |
310 | ||
311 | The kernel will stop and wait as early as the I/O driver and | |
312 | architecture allows when you use this option. If you build the kgdb I/O | |
313 | driver as a loadable kernel module kgdbwait will not do anything. | |
314 | ||
821c6df8 MCC |
315 | Kernel parameter: ``kgdbcon`` |
316 | ----------------------------- | |
7fb2e8a4 | 317 | |
2d88fc62 | 318 | The ``kgdbcon`` feature allows you to see printk() messages inside gdb |
821c6df8 | 319 | while gdb is connected to the kernel. Kdb does not make use of the kgdbcon |
7fb2e8a4 MCC |
320 | feature. |
321 | ||
322 | Kgdb supports using the gdb serial protocol to send console messages to | |
323 | the debugger when the debugger is connected and running. There are two | |
324 | ways to activate this feature. | |
325 | ||
821c6df8 MCC |
326 | 1. Activate with the kernel command line option:: |
327 | ||
328 | kgdbcon | |
7fb2e8a4 | 329 | |
821c6df8 | 330 | 2. Use sysfs before configuring an I/O driver:: |
7fb2e8a4 | 331 | |
821c6df8 | 332 | echo 1 > /sys/module/kgdb/parameters/kgdb_use_con |
7fb2e8a4 | 333 | |
821c6df8 | 334 | .. note:: |
7fb2e8a4 | 335 | |
821c6df8 | 336 | If you do this after you configure the kgdb I/O driver, the |
7fb2e8a4 MCC |
337 | setting will not take effect until the next point the I/O is |
338 | reconfigured. | |
339 | ||
821c6df8 MCC |
340 | .. important:: |
341 | ||
342 | You cannot use kgdboc + kgdbcon on a tty that is an | |
343 | active system console. An example of incorrect usage is:: | |
344 | ||
345 | console=ttyS0,115200 kgdboc=ttyS0 kgdbcon | |
7fb2e8a4 MCC |
346 | |
347 | It is possible to use this option with kgdboc on a tty that is not a | |
348 | system console. | |
349 | ||
821c6df8 MCC |
350 | Run time parameter: ``kgdbreboot`` |
351 | ---------------------------------- | |
7fb2e8a4 MCC |
352 | |
353 | The kgdbreboot feature allows you to change how the debugger deals with | |
354 | the reboot notification. You have 3 choices for the behavior. The | |
355 | default behavior is always set to 0. | |
356 | ||
821c6df8 | 357 | .. tabularcolumns:: |p{0.4cm}|p{11.5cm}|p{5.6cm}| |
7fb2e8a4 | 358 | |
821c6df8 MCC |
359 | .. flat-table:: |
360 | :widths: 1 10 8 | |
7fb2e8a4 | 361 | |
821c6df8 MCC |
362 | * - 1 |
363 | - ``echo -1 > /sys/module/debug_core/parameters/kgdbreboot`` | |
364 | - Ignore the reboot notification entirely. | |
7fb2e8a4 | 365 | |
821c6df8 MCC |
366 | * - 2 |
367 | - ``echo 0 > /sys/module/debug_core/parameters/kgdbreboot`` | |
368 | - Send the detach message to any attached debugger client. | |
7fb2e8a4 | 369 | |
821c6df8 MCC |
370 | * - 3 |
371 | - ``echo 1 > /sys/module/debug_core/parameters/kgdbreboot`` | |
372 | - Enter the debugger on reboot notify. | |
7fb2e8a4 | 373 | |
14994a9b ZZ |
374 | Kernel parameter: ``nokaslr`` |
375 | ----------------------------- | |
376 | ||
377 | If the architecture that you are using enable KASLR by default, | |
378 | you should consider turning it off. KASLR randomizes the | |
379 | virtual address where the kernel image is mapped and confuse | |
380 | gdb which resolve kernel symbol address from symbol table | |
381 | of vmlinux. | |
382 | ||
7fb2e8a4 MCC |
383 | Using kdb |
384 | ========= | |
385 | ||
386 | Quick start for kdb on a serial port | |
387 | ------------------------------------ | |
388 | ||
389 | This is a quick example of how to use kdb. | |
390 | ||
821c6df8 | 391 | 1. Configure kgdboc at boot using kernel parameters:: |
7fb2e8a4 | 392 | |
14994a9b | 393 | console=ttyS0,115200 kgdboc=ttyS0,115200 nokaslr |
7fb2e8a4 MCC |
394 | |
395 | OR | |
396 | ||
397 | Configure kgdboc after the kernel has booted; assuming you are using | |
821c6df8 | 398 | a serial port console:: |
7fb2e8a4 | 399 | |
821c6df8 | 400 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 MCC |
401 | |
402 | 2. Enter the kernel debugger manually or by waiting for an oops or | |
403 | fault. There are several ways you can enter the kernel debugger | |
821c6df8 | 404 | manually; all involve using the :kbd:`SysRq-G`, which means you must have |
e765c747 | 405 | enabled ``CONFIG_MAGIC_SYSRQ=y`` in your kernel config. |
7fb2e8a4 | 406 | |
821c6df8 | 407 | - When logged in as root or with a super user session you can run:: |
7fb2e8a4 | 408 | |
821c6df8 | 409 | echo g > /proc/sysrq-trigger |
7fb2e8a4 MCC |
410 | |
411 | - Example using minicom 2.2 | |
412 | ||
821c6df8 | 413 | Press: :kbd:`CTRL-A` :kbd:`f` :kbd:`g` |
7fb2e8a4 MCC |
414 | |
415 | - When you have telneted to a terminal server that supports sending | |
416 | a remote break | |
417 | ||
821c6df8 | 418 | Press: :kbd:`CTRL-]` |
7fb2e8a4 | 419 | |
821c6df8 | 420 | Type in: ``send break`` |
7fb2e8a4 | 421 | |
821c6df8 | 422 | Press: :kbd:`Enter` :kbd:`g` |
7fb2e8a4 | 423 | |
821c6df8 | 424 | 3. From the kdb prompt you can run the ``help`` command to see a complete |
7fb2e8a4 MCC |
425 | list of the commands that are available. |
426 | ||
427 | Some useful commands in kdb include: | |
428 | ||
821c6df8 MCC |
429 | =========== ================================================================= |
430 | ``lsmod`` Shows where kernel modules are loaded | |
431 | ``ps`` Displays only the active processes | |
432 | ``ps A`` Shows all the processes | |
433 | ``summary`` Shows kernel version info and memory usage | |
2d88fc62 | 434 | ``bt`` Get a backtrace of the current process using dump_stack() |
821c6df8 MCC |
435 | ``dmesg`` View the kernel syslog buffer |
436 | ``go`` Continue the system | |
437 | =========== ================================================================= | |
7fb2e8a4 MCC |
438 | |
439 | 4. When you are done using kdb you need to consider rebooting the system | |
821c6df8 | 440 | or using the ``go`` command to resuming normal kernel execution. If you |
7fb2e8a4 MCC |
441 | have paused the kernel for a lengthy period of time, applications |
442 | that rely on timely networking or anything to do with real wall clock | |
443 | time could be adversely affected, so you should take this into | |
444 | consideration when using the kernel debugger. | |
445 | ||
446 | Quick start for kdb using a keyboard connected console | |
447 | ------------------------------------------------------ | |
448 | ||
449 | This is a quick example of how to use kdb with a keyboard. | |
450 | ||
821c6df8 | 451 | 1. Configure kgdboc at boot using kernel parameters:: |
7fb2e8a4 | 452 | |
821c6df8 | 453 | kgdboc=kbd |
7fb2e8a4 MCC |
454 | |
455 | OR | |
456 | ||
821c6df8 | 457 | Configure kgdboc after the kernel has booted:: |
7fb2e8a4 | 458 | |
821c6df8 | 459 | echo kbd > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 MCC |
460 | |
461 | 2. Enter the kernel debugger manually or by waiting for an oops or | |
462 | fault. There are several ways you can enter the kernel debugger | |
821c6df8 | 463 | manually; all involve using the :kbd:`SysRq-G`, which means you must have |
e765c747 | 464 | enabled ``CONFIG_MAGIC_SYSRQ=y`` in your kernel config. |
7fb2e8a4 | 465 | |
821c6df8 | 466 | - When logged in as root or with a super user session you can run:: |
7fb2e8a4 | 467 | |
821c6df8 | 468 | echo g > /proc/sysrq-trigger |
7fb2e8a4 | 469 | |
821c6df8 | 470 | - Example using a laptop keyboard: |
7fb2e8a4 | 471 | |
821c6df8 | 472 | Press and hold down: :kbd:`Alt` |
7fb2e8a4 | 473 | |
821c6df8 | 474 | Press and hold down: :kbd:`Fn` |
7fb2e8a4 | 475 | |
821c6df8 | 476 | Press and release the key with the label: :kbd:`SysRq` |
7fb2e8a4 | 477 | |
821c6df8 | 478 | Release: :kbd:`Fn` |
7fb2e8a4 | 479 | |
821c6df8 | 480 | Press and release: :kbd:`g` |
7fb2e8a4 | 481 | |
821c6df8 | 482 | Release: :kbd:`Alt` |
7fb2e8a4 MCC |
483 | |
484 | - Example using a PS/2 101-key keyboard | |
485 | ||
821c6df8 | 486 | Press and hold down: :kbd:`Alt` |
7fb2e8a4 | 487 | |
821c6df8 | 488 | Press and release the key with the label: :kbd:`SysRq` |
7fb2e8a4 | 489 | |
821c6df8 | 490 | Press and release: :kbd:`g` |
7fb2e8a4 | 491 | |
821c6df8 | 492 | Release: :kbd:`Alt` |
7fb2e8a4 | 493 | |
821c6df8 | 494 | 3. Now type in a kdb command such as ``help``, ``dmesg``, ``bt`` or ``go`` to |
7fb2e8a4 MCC |
495 | continue kernel execution. |
496 | ||
497 | Using kgdb / gdb | |
498 | ================ | |
499 | ||
500 | In order to use kgdb you must activate it by passing configuration | |
501 | information to one of the kgdb I/O drivers. If you do not pass any | |
502 | configuration information kgdb will not do anything at all. Kgdb will | |
503 | only actively hook up to the kernel trap hooks if a kgdb I/O driver is | |
504 | loaded and configured. If you unconfigure a kgdb I/O driver, kgdb will | |
505 | unregister all the kernel hook points. | |
506 | ||
507 | All kgdb I/O drivers can be reconfigured at run time, if | |
508 | ``CONFIG_SYSFS`` and ``CONFIG_MODULES`` are enabled, by echo'ing a new | |
509 | config string to ``/sys/module/<driver>/parameter/<option>``. The driver | |
510 | can be unconfigured by passing an empty string. You cannot change the | |
511 | configuration while the debugger is attached. Make sure to detach the | |
512 | debugger with the ``detach`` command prior to trying to unconfigure a | |
513 | kgdb I/O driver. | |
514 | ||
515 | Connecting with gdb to a serial port | |
516 | ------------------------------------ | |
517 | ||
518 | 1. Configure kgdboc | |
519 | ||
821c6df8 | 520 | Configure kgdboc at boot using kernel parameters:: |
7fb2e8a4 | 521 | |
821c6df8 | 522 | kgdboc=ttyS0,115200 |
7fb2e8a4 MCC |
523 | |
524 | OR | |
525 | ||
821c6df8 | 526 | Configure kgdboc after the kernel has booted:: |
7fb2e8a4 | 527 | |
821c6df8 | 528 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 MCC |
529 | |
530 | 2. Stop kernel execution (break into the debugger) | |
531 | ||
532 | In order to connect to gdb via kgdboc, the kernel must first be | |
533 | stopped. There are several ways to stop the kernel which include | |
821c6df8 | 534 | using kgdbwait as a boot argument, via a :kbd:`SysRq-G`, or running the |
7fb2e8a4 MCC |
535 | kernel until it takes an exception where it waits for the debugger to |
536 | attach. | |
537 | ||
821c6df8 | 538 | - When logged in as root or with a super user session you can run:: |
7fb2e8a4 | 539 | |
821c6df8 | 540 | echo g > /proc/sysrq-trigger |
7fb2e8a4 MCC |
541 | |
542 | - Example using minicom 2.2 | |
543 | ||
821c6df8 | 544 | Press: :kbd:`CTRL-A` :kbd:`f` :kbd:`g` |
7fb2e8a4 MCC |
545 | |
546 | - When you have telneted to a terminal server that supports sending | |
547 | a remote break | |
548 | ||
821c6df8 | 549 | Press: :kbd:`CTRL-]` |
7fb2e8a4 | 550 | |
821c6df8 | 551 | Type in: ``send break`` |
7fb2e8a4 | 552 | |
821c6df8 | 553 | Press: :kbd:`Enter` :kbd:`g` |
7fb2e8a4 MCC |
554 | |
555 | 3. Connect from gdb | |
556 | ||
821c6df8 | 557 | Example (using a directly connected port):: |
7fb2e8a4 MCC |
558 | |
559 | % gdb ./vmlinux | |
689d8014 | 560 | (gdb) set serial baud 115200 |
7fb2e8a4 MCC |
561 | (gdb) target remote /dev/ttyS0 |
562 | ||
563 | ||
821c6df8 | 564 | Example (kgdb to a terminal server on TCP port 2012):: |
7fb2e8a4 MCC |
565 | |
566 | % gdb ./vmlinux | |
567 | (gdb) target remote 192.168.2.2:2012 | |
568 | ||
569 | ||
570 | Once connected, you can debug a kernel the way you would debug an | |
571 | application program. | |
572 | ||
573 | If you are having problems connecting or something is going seriously | |
574 | wrong while debugging, it will most often be the case that you want | |
575 | to enable gdb to be verbose about its target communications. You do | |
821c6df8 MCC |
576 | this prior to issuing the ``target remote`` command by typing in:: |
577 | ||
578 | set debug remote 1 | |
7fb2e8a4 MCC |
579 | |
580 | Remember if you continue in gdb, and need to "break in" again, you need | |
821c6df8 MCC |
581 | to issue an other :kbd:`SysRq-G`. It is easy to create a simple entry point by |
582 | putting a breakpoint at ``sys_sync`` and then you can run ``sync`` from a | |
7fb2e8a4 MCC |
583 | shell or script to break into the debugger. |
584 | ||
585 | kgdb and kdb interoperability | |
586 | ============================= | |
587 | ||
588 | It is possible to transition between kdb and kgdb dynamically. The debug | |
589 | core will remember which you used the last time and automatically start | |
590 | in the same mode. | |
591 | ||
592 | Switching between kdb and kgdb | |
593 | ------------------------------ | |
594 | ||
595 | Switching from kgdb to kdb | |
596 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
597 | ||
598 | There are two ways to switch from kgdb to kdb: you can use gdb to issue | |
821c6df8 | 599 | a maintenance packet, or you can blindly type the command ``$3#33``. |
7fb2e8a4 MCC |
600 | Whenever the kernel debugger stops in kgdb mode it will print the |
601 | message ``KGDB or $3#33 for KDB``. It is important to note that you have | |
602 | to type the sequence correctly in one pass. You cannot type a backspace | |
603 | or delete because kgdb will interpret that as part of the debug stream. | |
604 | ||
821c6df8 | 605 | 1. Change from kgdb to kdb by blindly typing:: |
7fb2e8a4 | 606 | |
821c6df8 | 607 | $3#33 |
7fb2e8a4 | 608 | |
821c6df8 | 609 | 2. Change from kgdb to kdb with gdb:: |
7fb2e8a4 | 610 | |
821c6df8 | 611 | maintenance packet 3 |
7fb2e8a4 | 612 | |
821c6df8 MCC |
613 | .. note:: |
614 | ||
615 | Now you must kill gdb. Typically you press :kbd:`CTRL-Z` and issue | |
616 | the command:: | |
617 | ||
618 | kill -9 % | |
7fb2e8a4 MCC |
619 | |
620 | Change from kdb to kgdb | |
621 | ~~~~~~~~~~~~~~~~~~~~~~~ | |
622 | ||
623 | There are two ways you can change from kdb to kgdb. You can manually | |
624 | enter kgdb mode by issuing the kgdb command from the kdb shell prompt, | |
625 | or you can connect gdb while the kdb shell prompt is active. The kdb | |
626 | shell looks for the typical first commands that gdb would issue with the | |
627 | gdb remote protocol and if it sees one of those commands it | |
628 | automatically changes into kgdb mode. | |
629 | ||
821c6df8 | 630 | 1. From kdb issue the command:: |
7fb2e8a4 | 631 | |
821c6df8 | 632 | kgdb |
7fb2e8a4 MCC |
633 | |
634 | Now disconnect your terminal program and connect gdb in its place | |
635 | ||
636 | 2. At the kdb prompt, disconnect the terminal program and connect gdb in | |
637 | its place. | |
638 | ||
639 | Running kdb commands from gdb | |
640 | ----------------------------- | |
641 | ||
642 | It is possible to run a limited set of kdb commands from gdb, using the | |
643 | gdb monitor command. You don't want to execute any of the run control or | |
644 | breakpoint operations, because it can disrupt the state of the kernel | |
645 | debugger. You should be using gdb for breakpoints and run control | |
646 | operations if you have gdb connected. The more useful commands to run | |
647 | are things like lsmod, dmesg, ps or possibly some of the memory | |
648 | information commands. To see all the kdb commands you can run | |
649 | ``monitor help``. | |
650 | ||
821c6df8 | 651 | Example:: |
7fb2e8a4 MCC |
652 | |
653 | (gdb) monitor ps | |
654 | 1 idle process (state I) and | |
655 | 27 sleeping system daemon (state M) processes suppressed, | |
656 | use 'ps A' to see all. | |
657 | Task Addr Pid Parent [*] cpu State Thread Command | |
658 | ||
659 | 0xc78291d0 1 0 0 0 S 0xc7829404 init | |
660 | 0xc7954150 942 1 0 0 S 0xc7954384 dropbear | |
661 | 0xc78789c0 944 1 0 0 S 0xc7878bf4 sh | |
662 | (gdb) | |
663 | ||
7fb2e8a4 MCC |
664 | kgdb Test Suite |
665 | =============== | |
666 | ||
667 | When kgdb is enabled in the kernel config you can also elect to enable | |
821c6df8 | 668 | the config parameter ``KGDB_TESTS``. Turning this on will enable a special |
7fb2e8a4 MCC |
669 | kgdb I/O module which is designed to test the kgdb internal functions. |
670 | ||
671 | The kgdb tests are mainly intended for developers to test the kgdb | |
672 | internals as well as a tool for developing a new kgdb architecture | |
673 | specific implementation. These tests are not really for end users of the | |
674 | Linux kernel. The primary source of documentation would be to look in | |
821c6df8 | 675 | the ``drivers/misc/kgdbts.c`` file. |
7fb2e8a4 MCC |
676 | |
677 | The kgdb test suite can also be configured at compile time to run the | |
678 | core set of tests by setting the kernel config parameter | |
821c6df8 | 679 | ``KGDB_TESTS_ON_BOOT``. This particular option is aimed at automated |
7fb2e8a4 MCC |
680 | regression testing and does not require modifying the kernel boot config |
681 | arguments. If this is turned on, the kgdb test suite can be disabled by | |
821c6df8 | 682 | specifying ``kgdbts=`` as a kernel boot argument. |
7fb2e8a4 MCC |
683 | |
684 | Kernel Debugger Internals | |
685 | ========================= | |
686 | ||
687 | Architecture Specifics | |
688 | ---------------------- | |
689 | ||
690 | The kernel debugger is organized into a number of components: | |
691 | ||
692 | 1. The debug core | |
693 | ||
821c6df8 | 694 | The debug core is found in ``kernel/debugger/debug_core.c``. It |
7fb2e8a4 MCC |
695 | contains: |
696 | ||
697 | - A generic OS exception handler which includes sync'ing the | |
698 | processors into a stopped state on an multi-CPU system. | |
699 | ||
700 | - The API to talk to the kgdb I/O drivers | |
701 | ||
702 | - The API to make calls to the arch-specific kgdb implementation | |
703 | ||
704 | - The logic to perform safe memory reads and writes to memory while | |
705 | using the debugger | |
706 | ||
707 | - A full implementation for software breakpoints unless overridden | |
708 | by the arch | |
709 | ||
710 | - The API to invoke either the kdb or kgdb frontend to the debug | |
711 | core. | |
712 | ||
713 | - The structures and callback API for atomic kernel mode setting. | |
714 | ||
821c6df8 | 715 | .. note:: kgdboc is where the kms callbacks are invoked. |
7fb2e8a4 MCC |
716 | |
717 | 2. kgdb arch-specific implementation | |
718 | ||
821c6df8 MCC |
719 | This implementation is generally found in ``arch/*/kernel/kgdb.c``. As |
720 | an example, ``arch/x86/kernel/kgdb.c`` contains the specifics to | |
7fb2e8a4 MCC |
721 | implement HW breakpoint as well as the initialization to dynamically |
722 | register and unregister for the trap handlers on this architecture. | |
723 | The arch-specific portion implements: | |
724 | ||
725 | - contains an arch-specific trap catcher which invokes | |
2d88fc62 | 726 | kgdb_handle_exception() to start kgdb about doing its work |
7fb2e8a4 | 727 | |
365ff56f | 728 | - translation to and from gdb specific packet format to struct pt_regs |
7fb2e8a4 MCC |
729 | |
730 | - Registration and unregistration of architecture specific trap | |
731 | hooks | |
732 | ||
733 | - Any special exception handling and cleanup | |
734 | ||
735 | - NMI exception handling and cleanup | |
736 | ||
737 | - (optional) HW breakpoints | |
738 | ||
739 | 3. gdbstub frontend (aka kgdb) | |
740 | ||
821c6df8 | 741 | The gdbstub is located in ``kernel/debug/gdbstub.c``. It contains: |
7fb2e8a4 MCC |
742 | |
743 | - All the logic to implement the gdb serial protocol | |
744 | ||
745 | 4. kdb frontend | |
746 | ||
747 | The kdb debugger shell is broken down into a number of components. | |
748 | The kdb core is located in kernel/debug/kdb. There are a number of | |
749 | helper functions in some of the other kernel components to make it | |
750 | possible for kdb to examine and report information about the kernel | |
751 | without taking locks that could cause a kernel deadlock. The kdb core | |
752 | contains implements the following functionality. | |
753 | ||
754 | - A simple shell | |
755 | ||
756 | - The kdb core command set | |
757 | ||
758 | - A registration API to register additional kdb shell commands. | |
759 | ||
821c6df8 | 760 | - A good example of a self-contained kdb module is the ``ftdump`` |
7fb2e8a4 | 761 | command for dumping the ftrace buffer. See: |
821c6df8 | 762 | ``kernel/trace/trace_kdb.c`` |
7fb2e8a4 MCC |
763 | |
764 | - For an example of how to dynamically register a new kdb command | |
765 | you can build the kdb_hello.ko kernel module from | |
821c6df8 MCC |
766 | ``samples/kdb/kdb_hello.c``. To build this example you can set |
767 | ``CONFIG_SAMPLES=y`` and ``CONFIG_SAMPLE_KDB=m`` in your kernel | |
768 | config. Later run ``modprobe kdb_hello`` and the next time you | |
769 | enter the kdb shell, you can run the ``hello`` command. | |
7fb2e8a4 | 770 | |
2d88fc62 | 771 | - The implementation for kdb_printf() which emits messages directly |
7fb2e8a4 MCC |
772 | to I/O drivers, bypassing the kernel log. |
773 | ||
774 | - SW / HW breakpoint management for the kdb shell | |
775 | ||
776 | 5. kgdb I/O driver | |
777 | ||
778 | Each kgdb I/O driver has to provide an implementation for the | |
779 | following: | |
780 | ||
781 | - configuration via built-in or module | |
782 | ||
783 | - dynamic configuration and kgdb hook registration calls | |
784 | ||
785 | - read and write character interface | |
786 | ||
787 | - A cleanup handler for unconfiguring from the kgdb core | |
788 | ||
789 | - (optional) Early debug methodology | |
790 | ||
791 | Any given kgdb I/O driver has to operate very closely with the | |
792 | hardware and must do it in such a way that does not enable interrupts | |
793 | or change other parts of the system context without completely | |
794 | restoring them. The kgdb core will repeatedly "poll" a kgdb I/O | |
795 | driver for characters when it needs input. The I/O driver is expected | |
796 | to return immediately if there is no data available. Doing so allows | |
797 | for the future possibility to touch watchdog hardware in such a way | |
798 | as to have a target system not reset when these are enabled. | |
799 | ||
800 | If you are intent on adding kgdb architecture specific support for a new | |
801 | architecture, the architecture should define ``HAVE_ARCH_KGDB`` in the | |
802 | architecture specific Kconfig file. This will enable kgdb for the | |
803 | architecture, and at that point you must create an architecture specific | |
804 | kgdb implementation. | |
805 | ||
806 | There are a few flags which must be set on every architecture in their | |
821c6df8 | 807 | ``asm/kgdb.h`` file. These are: |
7fb2e8a4 | 808 | |
821c6df8 MCC |
809 | - ``NUMREGBYTES``: |
810 | The size in bytes of all of the registers, so that we | |
811 | can ensure they will all fit into a packet. | |
7fb2e8a4 | 812 | |
821c6df8 MCC |
813 | - ``BUFMAX``: |
814 | The size in bytes of the buffer GDB will read into. This must | |
815 | be larger than NUMREGBYTES. | |
7fb2e8a4 | 816 | |
821c6df8 MCC |
817 | - ``CACHE_FLUSH_IS_SAFE``: |
818 | Set to 1 if it is always safe to call | |
819 | flush_cache_range or flush_icache_range. On some architectures, | |
820 | these functions may not be safe to call on SMP since we keep other | |
821 | CPUs in a holding pattern. | |
7fb2e8a4 MCC |
822 | |
823 | There are also the following functions for the common backend, found in | |
821c6df8 | 824 | ``kernel/kgdb.c``, that must be supplied by the architecture-specific |
7fb2e8a4 MCC |
825 | backend unless marked as (optional), in which case a default function |
826 | maybe used if the architecture does not need to provide a specific | |
827 | implementation. | |
828 | ||
829 | .. kernel-doc:: include/linux/kgdb.h | |
830 | :internal: | |
831 | ||
832 | kgdboc internals | |
833 | ---------------- | |
834 | ||
835 | kgdboc and uarts | |
836 | ~~~~~~~~~~~~~~~~ | |
837 | ||
838 | The kgdboc driver is actually a very thin driver that relies on the | |
839 | underlying low level to the hardware driver having "polling hooks" to | |
840 | which the tty driver is attached. In the initial implementation of | |
841 | kgdboc the serial_core was changed to expose a low level UART hook for | |
842 | doing polled mode reading and writing of a single character while in an | |
843 | atomic context. When kgdb makes an I/O request to the debugger, kgdboc | |
844 | invokes a callback in the serial core which in turn uses the callback in | |
845 | the UART driver. | |
846 | ||
847 | When using kgdboc with a UART, the UART driver must implement two | |
365ff56f | 848 | callbacks in the struct uart_ops. |
821c6df8 | 849 | Example from ``drivers/8250.c``:: |
7fb2e8a4 | 850 | |
7fb2e8a4 MCC |
851 | |
852 | #ifdef CONFIG_CONSOLE_POLL | |
853 | .poll_get_char = serial8250_get_poll_char, | |
854 | .poll_put_char = serial8250_put_poll_char, | |
855 | #endif | |
856 | ||
857 | ||
858 | Any implementation specifics around creating a polling driver use the | |
859 | ``#ifdef CONFIG_CONSOLE_POLL``, as shown above. Keep in mind that | |
860 | polling hooks have to be implemented in such a way that they can be | |
861 | called from an atomic context and have to restore the state of the UART | |
862 | chip on return such that the system can return to normal when the | |
863 | debugger detaches. You need to be very careful with any kind of lock you | |
864 | consider, because failing here is most likely going to mean pressing the | |
865 | reset button. | |
866 | ||
867 | kgdboc and keyboards | |
821c6df8 | 868 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
7fb2e8a4 MCC |
869 | |
870 | The kgdboc driver contains logic to configure communications with an | |
871 | attached keyboard. The keyboard infrastructure is only compiled into the | |
821c6df8 | 872 | kernel when ``CONFIG_KDB_KEYBOARD=y`` is set in the kernel configuration. |
7fb2e8a4 | 873 | |
06467a78 | 874 | The core polled keyboard driver for PS/2 type keyboards is in |
821c6df8 | 875 | ``drivers/char/kdb_keyboard.c``. This driver is hooked into the debug core |
7fb2e8a4 | 876 | when kgdboc populates the callback in the array called |
365ff56f | 877 | :c:expr:`kdb_poll_funcs[]`. The kdb_get_kbd_char() is the top-level |
7fb2e8a4 MCC |
878 | function which polls hardware for single character input. |
879 | ||
880 | kgdboc and kms | |
821c6df8 | 881 | ~~~~~~~~~~~~~~~~~~ |
7fb2e8a4 MCC |
882 | |
883 | The kgdboc driver contains logic to request the graphics display to | |
821c6df8 | 884 | switch to a text context when you are using ``kgdboc=kms,kbd``, provided |
7fb2e8a4 MCC |
885 | that you have a video driver which has a frame buffer console and atomic |
886 | kernel mode setting support. | |
887 | ||
888 | Every time the kernel debugger is entered it calls | |
2d88fc62 | 889 | kgdboc_pre_exp_handler() which in turn calls con_debug_enter() |
821c6df8 | 890 | in the virtual console layer. On resuming kernel execution, the kernel |
2d88fc62 PM |
891 | debugger calls kgdboc_post_exp_handler() which in turn calls |
892 | con_debug_leave(). | |
7fb2e8a4 MCC |
893 | |
894 | Any video driver that wants to be compatible with the kernel debugger | |
821c6df8 MCC |
895 | and the atomic kms callbacks must implement the ``mode_set_base_atomic``, |
896 | ``fb_debug_enter`` and ``fb_debug_leave operations``. For the | |
897 | ``fb_debug_enter`` and ``fb_debug_leave`` the option exists to use the | |
7fb2e8a4 MCC |
898 | generic drm fb helper functions or implement something custom for the |
899 | hardware. The following example shows the initialization of the | |
900 | .mode_set_base_atomic operation in | |
821c6df8 | 901 | drivers/gpu/drm/i915/intel_display.c:: |
7fb2e8a4 | 902 | |
7fb2e8a4 MCC |
903 | |
904 | static const struct drm_crtc_helper_funcs intel_helper_funcs = { | |
905 | [...] | |
906 | .mode_set_base_atomic = intel_pipe_set_base_atomic, | |
907 | [...] | |
908 | }; | |
909 | ||
910 | ||
7fb2e8a4 MCC |
911 | Here is an example of how the i915 driver initializes the |
912 | fb_debug_enter and fb_debug_leave functions to use the generic drm | |
821c6df8 | 913 | helpers in ``drivers/gpu/drm/i915/intel_fb.c``:: |
7fb2e8a4 | 914 | |
7fb2e8a4 MCC |
915 | |
916 | static struct fb_ops intelfb_ops = { | |
917 | [...] | |
918 | .fb_debug_enter = drm_fb_helper_debug_enter, | |
919 | .fb_debug_leave = drm_fb_helper_debug_leave, | |
920 | [...] | |
921 | }; | |
922 | ||
923 | ||
7fb2e8a4 MCC |
924 | Credits |
925 | ======= | |
926 | ||
927 | The following people have contributed to this document: | |
928 | ||
821c6df8 | 929 | 1. Amit Kale <amitkale@linsyssoft.com> |
7fb2e8a4 | 930 | |
821c6df8 | 931 | 2. Tom Rini <trini@kernel.crashing.org> |
7fb2e8a4 MCC |
932 | |
933 | In March 2008 this document was completely rewritten by: | |
934 | ||
821c6df8 | 935 | - Jason Wessel <jason.wessel@windriver.com> |
7fb2e8a4 MCC |
936 | |
937 | In Jan 2010 this document was updated to include kdb. | |
938 | ||
821c6df8 | 939 | - Jason Wessel <jason.wessel@windriver.com> |