Commit | Line | Data |
---|---|---|
7fb2e8a4 MCC |
1 | ================================================= |
2 | Using kgdb, kdb and the kernel debugger internals | |
3 | ================================================= | |
4 | ||
5 | :Author: Jason Wessel | |
6 | ||
7 | Introduction | |
8 | ============ | |
9 | ||
10 | The kernel has two different debugger front ends (kdb and kgdb) which | |
11 | interface to the debug core. It is possible to use either of the | |
12 | debugger front ends and dynamically transition between them if you | |
13 | configure the kernel properly at compile and runtime. | |
14 | ||
15 | Kdb is simplistic shell-style interface which you can use on a system | |
16 | console with a keyboard or serial console. You can use it to inspect | |
17 | memory, registers, process lists, dmesg, and even set breakpoints to | |
18 | stop in a certain location. Kdb is not a source level debugger, although | |
19 | you can set breakpoints and execute some basic kernel run control. Kdb | |
20 | is mainly aimed at doing some analysis to aid in development or | |
21 | diagnosing kernel problems. You can access some symbols by name in | |
22 | kernel built-ins or in kernel modules if the code was built with | |
23 | ``CONFIG_KALLSYMS``. | |
24 | ||
25 | Kgdb is intended to be used as a source level debugger for the Linux | |
26 | kernel. It is used along with gdb to debug a Linux kernel. The | |
27 | expectation is that gdb can be used to "break in" to the kernel to | |
28 | inspect memory, variables and look through call stack information | |
29 | similar to the way an application developer would use gdb to debug an | |
30 | application. It is possible to place breakpoints in kernel code and | |
31 | perform some limited execution stepping. | |
32 | ||
33 | Two machines are required for using kgdb. One of these machines is a | |
34 | development machine and the other is the target machine. The kernel to | |
35 | be debugged runs on the target machine. The development machine runs an | |
36 | instance of gdb against the vmlinux file which contains the symbols (not | |
37 | a boot image such as bzImage, zImage, uImage...). In gdb the developer | |
38 | specifies the connection parameters and connects to kgdb. The type of | |
39 | connection a developer makes with gdb depends on the availability of | |
40 | kgdb I/O modules compiled as built-ins or loadable kernel modules in the | |
41 | test machine's kernel. | |
42 | ||
43 | Compiling a kernel | |
44 | ================== | |
45 | ||
46 | - In order to enable compilation of kdb, you must first enable kgdb. | |
47 | ||
48 | - The kgdb test compile options are described in the kgdb test suite | |
49 | chapter. | |
50 | ||
51 | Kernel config options for kgdb | |
52 | ------------------------------ | |
53 | ||
821c6df8 MCC |
54 | To enable ``CONFIG_KGDB`` you should look under |
55 | :menuselection:`Kernel hacking --> Kernel debugging` and select | |
56 | :menuselection:`KGDB: kernel debugger`. | |
7fb2e8a4 MCC |
57 | |
58 | While it is not a hard requirement that you have symbols in your vmlinux | |
59 | file, gdb tends not to be very useful without the symbolic data, so you | |
821c6df8 MCC |
60 | will want to turn on ``CONFIG_DEBUG_INFO`` which is called |
61 | :menuselection:`Compile the kernel with debug info` in the config menu. | |
7fb2e8a4 MCC |
62 | |
63 | It is advised, but not required, that you turn on the | |
821c6df8 MCC |
64 | ``CONFIG_FRAME_POINTER`` kernel option which is called :menuselection:`Compile |
65 | the kernel with frame pointers` in the config menu. This option inserts code | |
7fb2e8a4 MCC |
66 | to into the compiled executable which saves the frame information in |
67 | registers or on the stack at different points which allows a debugger | |
68 | such as gdb to more accurately construct stack back traces while | |
69 | debugging the kernel. | |
70 | ||
71 | If the architecture that you are using supports the kernel option | |
821c6df8 | 72 | ``CONFIG_STRICT_KERNEL_RWX``, you should consider turning it off. This |
7fb2e8a4 MCC |
73 | option will prevent the use of software breakpoints because it marks |
74 | certain regions of the kernel's memory space as read-only. If kgdb | |
75 | supports it for the architecture you are using, you can use hardware | |
821c6df8 | 76 | breakpoints if you desire to run with the ``CONFIG_STRICT_KERNEL_RWX`` |
7fb2e8a4 MCC |
77 | option turned on, else you need to turn off this option. |
78 | ||
79 | Next you should choose one of more I/O drivers to interconnect debugging | |
80 | host and debugged target. Early boot debugging requires a KGDB I/O | |
81 | driver that supports early debugging and the driver must be built into | |
82 | the kernel directly. Kgdb I/O driver configuration takes place via | |
83 | kernel or module parameters which you can learn more about in the in the | |
821c6df8 | 84 | section that describes the parameter kgdboc. |
7fb2e8a4 | 85 | |
821c6df8 | 86 | Here is an example set of ``.config`` symbols to enable or disable for kgdb:: |
7fb2e8a4 | 87 | |
821c6df8 MCC |
88 | # CONFIG_STRICT_KERNEL_RWX is not set |
89 | CONFIG_FRAME_POINTER=y | |
90 | CONFIG_KGDB=y | |
91 | CONFIG_KGDB_SERIAL_CONSOLE=y | |
7fb2e8a4 MCC |
92 | |
93 | Kernel config options for kdb | |
94 | ----------------------------- | |
95 | ||
96 | Kdb is quite a bit more complex than the simple gdbstub sitting on top | |
97 | of the kernel's debug core. Kdb must implement a shell, and also adds | |
98 | some helper functions in other parts of the kernel, responsible for | |
99 | printing out interesting data such as what you would see if you ran | |
821c6df8 | 100 | ``lsmod``, or ``ps``. In order to build kdb into the kernel you follow the |
7fb2e8a4 MCC |
101 | same steps as you would for kgdb. |
102 | ||
103 | The main config option for kdb is ``CONFIG_KGDB_KDB`` which is called | |
821c6df8 MCC |
104 | :menuselection:`KGDB_KDB: include kdb frontend for kgdb` in the config menu. |
105 | In theory you would have already also selected an I/O driver such as the | |
106 | ``CONFIG_KGDB_SERIAL_CONSOLE`` interface if you plan on using kdb on a | |
7fb2e8a4 MCC |
107 | serial port, when you were configuring kgdb. |
108 | ||
109 | If you want to use a PS/2-style keyboard with kdb, you would select | |
821c6df8 MCC |
110 | ``CONFIG_KDB_KEYBOARD`` which is called :menuselection:`KGDB_KDB: keyboard as |
111 | input device` in the config menu. The ``CONFIG_KDB_KEYBOARD`` option is not | |
112 | used for anything in the gdb interface to kgdb. The ``CONFIG_KDB_KEYBOARD`` | |
7fb2e8a4 MCC |
113 | option only works with kdb. |
114 | ||
821c6df8 | 115 | Here is an example set of ``.config`` symbols to enable/disable kdb:: |
7fb2e8a4 | 116 | |
821c6df8 MCC |
117 | # CONFIG_STRICT_KERNEL_RWX is not set |
118 | CONFIG_FRAME_POINTER=y | |
119 | CONFIG_KGDB=y | |
120 | CONFIG_KGDB_SERIAL_CONSOLE=y | |
121 | CONFIG_KGDB_KDB=y | |
122 | CONFIG_KDB_KEYBOARD=y | |
7fb2e8a4 MCC |
123 | |
124 | Kernel Debugger Boot Arguments | |
125 | ============================== | |
126 | ||
127 | This section describes the various runtime kernel parameters that affect | |
128 | the configuration of the kernel debugger. The following chapter covers | |
129 | using kdb and kgdb as well as providing some examples of the | |
130 | configuration parameters. | |
131 | ||
132 | Kernel parameter: kgdboc | |
133 | ------------------------ | |
134 | ||
135 | The kgdboc driver was originally an abbreviation meant to stand for | |
136 | "kgdb over console". Today it is the primary mechanism to configure how | |
137 | to communicate from gdb to kgdb as well as the devices you want to use | |
138 | to interact with the kdb shell. | |
139 | ||
140 | For kgdb/gdb, kgdboc is designed to work with a single serial port. It | |
141 | is intended to cover the circumstance where you want to use a serial | |
142 | console as your primary console as well as using it to perform kernel | |
143 | debugging. It is also possible to use kgdb on a serial port which is not | |
144 | designated as a system console. Kgdboc may be configured as a kernel | |
145 | built-in or a kernel loadable module. You can only make use of | |
146 | ``kgdbwait`` and early debugging if you build kgdboc into the kernel as | |
147 | a built-in. | |
148 | ||
149 | Optionally you can elect to activate kms (Kernel Mode Setting) | |
150 | integration. When you use kms with kgdboc and you have a video driver | |
151 | that has atomic mode setting hooks, it is possible to enter the debugger | |
152 | on the graphics console. When the kernel execution is resumed, the | |
153 | previous graphics mode will be restored. This integration can serve as a | |
154 | useful tool to aid in diagnosing crashes or doing analysis of memory | |
155 | with kdb while allowing the full graphics console applications to run. | |
156 | ||
157 | kgdboc arguments | |
158 | ~~~~~~~~~~~~~~~~ | |
159 | ||
821c6df8 MCC |
160 | Usage:: |
161 | ||
162 | kgdboc=[kms][[,]kbd][[,]serial_device][,baud] | |
7fb2e8a4 MCC |
163 | |
164 | The order listed above must be observed if you use any of the optional | |
165 | configurations together. | |
166 | ||
167 | Abbreviations: | |
168 | ||
169 | - kms = Kernel Mode Setting | |
170 | ||
171 | - kbd = Keyboard | |
172 | ||
173 | You can configure kgdboc to use the keyboard, and/or a serial device | |
174 | depending on if you are using kdb and/or kgdb, in one of the following | |
175 | scenarios. The order listed above must be observed if you use any of the | |
176 | optional configurations together. Using kms + only gdb is generally not | |
177 | a useful combination. | |
178 | ||
179 | Using loadable module or built-in | |
180 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
181 | ||
182 | 1. As a kernel built-in: | |
183 | ||
821c6df8 MCC |
184 | Use the kernel boot argument:: |
185 | ||
186 | kgdboc=<tty-device>,[baud] | |
7fb2e8a4 MCC |
187 | |
188 | 2. As a kernel loadable module: | |
189 | ||
821c6df8 MCC |
190 | Use the command:: |
191 | ||
192 | modprobe kgdboc kgdboc=<tty-device>,[baud] | |
7fb2e8a4 MCC |
193 | |
194 | Here are two examples of how you might format the kgdboc string. The | |
195 | first is for an x86 target using the first serial port. The second | |
196 | example is for the ARM Versatile AB using the second serial port. | |
197 | ||
198 | 1. ``kgdboc=ttyS0,115200`` | |
199 | ||
200 | 2. ``kgdboc=ttyAMA1,115200`` | |
201 | ||
202 | Configure kgdboc at runtime with sysfs | |
203 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
204 | ||
205 | At run time you can enable or disable kgdboc by echoing a parameters | |
206 | into the sysfs. Here are two examples: | |
207 | ||
821c6df8 | 208 | 1. Enable kgdboc on ttyS0:: |
7fb2e8a4 | 209 | |
821c6df8 | 210 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 | 211 | |
821c6df8 | 212 | 2. Disable kgdboc:: |
7fb2e8a4 | 213 | |
821c6df8 | 214 | echo "" > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 | 215 | |
821c6df8 MCC |
216 | .. note:: |
217 | ||
218 | You do not need to specify the baud if you are configuring the | |
219 | console on tty which is already configured or open. | |
7fb2e8a4 MCC |
220 | |
221 | More examples | |
222 | ^^^^^^^^^^^^^ | |
223 | ||
224 | You can configure kgdboc to use the keyboard, and/or a serial device | |
225 | depending on if you are using kdb and/or kgdb, in one of the following | |
226 | scenarios. | |
227 | ||
821c6df8 MCC |
228 | 1. kdb and kgdb over only a serial port:: |
229 | ||
230 | kgdboc=<serial_device>[,baud] | |
231 | ||
232 | Example:: | |
7fb2e8a4 | 233 | |
821c6df8 | 234 | kgdboc=ttyS0,115200 |
7fb2e8a4 | 235 | |
821c6df8 | 236 | 2. kdb and kgdb with keyboard and a serial port:: |
7fb2e8a4 | 237 | |
821c6df8 | 238 | kgdboc=kbd,<serial_device>[,baud] |
7fb2e8a4 | 239 | |
821c6df8 | 240 | Example:: |
7fb2e8a4 | 241 | |
821c6df8 | 242 | kgdboc=kbd,ttyS0,115200 |
7fb2e8a4 | 243 | |
821c6df8 | 244 | 3. kdb with a keyboard:: |
7fb2e8a4 | 245 | |
821c6df8 | 246 | kgdboc=kbd |
7fb2e8a4 | 247 | |
821c6df8 | 248 | 4. kdb with kernel mode setting:: |
7fb2e8a4 | 249 | |
821c6df8 | 250 | kgdboc=kms,kbd |
7fb2e8a4 | 251 | |
821c6df8 | 252 | 5. kdb with kernel mode setting and kgdb over a serial port:: |
7fb2e8a4 | 253 | |
821c6df8 | 254 | kgdboc=kms,kbd,ttyS0,115200 |
7fb2e8a4 | 255 | |
821c6df8 MCC |
256 | .. note:: |
257 | ||
258 | Kgdboc does not support interrupting the target via the gdb remote | |
259 | protocol. You must manually send a :kbd:`SysRq-G` unless you have a proxy | |
260 | that splits console output to a terminal program. A console proxy has a | |
261 | separate TCP port for the debugger and a separate TCP port for the | |
262 | "human" console. The proxy can take care of sending the :kbd:`SysRq-G` | |
263 | for you. | |
7fb2e8a4 MCC |
264 | |
265 | When using kgdboc with no debugger proxy, you can end up connecting the | |
266 | debugger at one of two entry points. If an exception occurs after you | |
267 | have loaded kgdboc, a message should print on the console stating it is | |
268 | waiting for the debugger. In this case you disconnect your terminal | |
269 | program and then connect the debugger in its place. If you want to | |
270 | interrupt the target system and forcibly enter a debug session you have | |
821c6df8 | 271 | to issue a :kbd:`Sysrq` sequence and then type the letter :kbd:`g`. Then you |
7fb2e8a4 | 272 | disconnect the terminal session and connect gdb. Your options if you |
821c6df8 | 273 | don't like this are to hack gdb to send the :kbd:`SysRq-G` for you as well as |
7fb2e8a4 MCC |
274 | on the initial connect, or to use a debugger proxy that allows an |
275 | unmodified gdb to do the debugging. | |
276 | ||
821c6df8 MCC |
277 | Kernel parameter: ``kgdbwait`` |
278 | ------------------------------ | |
7fb2e8a4 MCC |
279 | |
280 | The Kernel command line option ``kgdbwait`` makes kgdb wait for a | |
281 | debugger connection during booting of a kernel. You can only use this | |
282 | option if you compiled a kgdb I/O driver into the kernel and you | |
283 | specified the I/O driver configuration as a kernel command line option. | |
284 | The kgdbwait parameter should always follow the configuration parameter | |
285 | for the kgdb I/O driver in the kernel command line else the I/O driver | |
286 | will not be configured prior to asking the kernel to use it to wait. | |
287 | ||
288 | The kernel will stop and wait as early as the I/O driver and | |
289 | architecture allows when you use this option. If you build the kgdb I/O | |
290 | driver as a loadable kernel module kgdbwait will not do anything. | |
291 | ||
821c6df8 MCC |
292 | Kernel parameter: ``kgdbcon`` |
293 | ----------------------------- | |
7fb2e8a4 | 294 | |
821c6df8 MCC |
295 | The ``kgdbcon`` feature allows you to see :c:func:`printk` messages inside gdb |
296 | while gdb is connected to the kernel. Kdb does not make use of the kgdbcon | |
7fb2e8a4 MCC |
297 | feature. |
298 | ||
299 | Kgdb supports using the gdb serial protocol to send console messages to | |
300 | the debugger when the debugger is connected and running. There are two | |
301 | ways to activate this feature. | |
302 | ||
821c6df8 MCC |
303 | 1. Activate with the kernel command line option:: |
304 | ||
305 | kgdbcon | |
7fb2e8a4 | 306 | |
821c6df8 | 307 | 2. Use sysfs before configuring an I/O driver:: |
7fb2e8a4 | 308 | |
821c6df8 | 309 | echo 1 > /sys/module/kgdb/parameters/kgdb_use_con |
7fb2e8a4 | 310 | |
821c6df8 | 311 | .. note:: |
7fb2e8a4 | 312 | |
821c6df8 | 313 | If you do this after you configure the kgdb I/O driver, the |
7fb2e8a4 MCC |
314 | setting will not take effect until the next point the I/O is |
315 | reconfigured. | |
316 | ||
821c6df8 MCC |
317 | .. important:: |
318 | ||
319 | You cannot use kgdboc + kgdbcon on a tty that is an | |
320 | active system console. An example of incorrect usage is:: | |
321 | ||
322 | console=ttyS0,115200 kgdboc=ttyS0 kgdbcon | |
7fb2e8a4 MCC |
323 | |
324 | It is possible to use this option with kgdboc on a tty that is not a | |
325 | system console. | |
326 | ||
821c6df8 MCC |
327 | Run time parameter: ``kgdbreboot`` |
328 | ---------------------------------- | |
7fb2e8a4 MCC |
329 | |
330 | The kgdbreboot feature allows you to change how the debugger deals with | |
331 | the reboot notification. You have 3 choices for the behavior. The | |
332 | default behavior is always set to 0. | |
333 | ||
821c6df8 | 334 | .. tabularcolumns:: |p{0.4cm}|p{11.5cm}|p{5.6cm}| |
7fb2e8a4 | 335 | |
821c6df8 MCC |
336 | .. flat-table:: |
337 | :widths: 1 10 8 | |
7fb2e8a4 | 338 | |
821c6df8 MCC |
339 | * - 1 |
340 | - ``echo -1 > /sys/module/debug_core/parameters/kgdbreboot`` | |
341 | - Ignore the reboot notification entirely. | |
7fb2e8a4 | 342 | |
821c6df8 MCC |
343 | * - 2 |
344 | - ``echo 0 > /sys/module/debug_core/parameters/kgdbreboot`` | |
345 | - Send the detach message to any attached debugger client. | |
7fb2e8a4 | 346 | |
821c6df8 MCC |
347 | * - 3 |
348 | - ``echo 1 > /sys/module/debug_core/parameters/kgdbreboot`` | |
349 | - Enter the debugger on reboot notify. | |
7fb2e8a4 | 350 | |
14994a9b ZZ |
351 | Kernel parameter: ``nokaslr`` |
352 | ----------------------------- | |
353 | ||
354 | If the architecture that you are using enable KASLR by default, | |
355 | you should consider turning it off. KASLR randomizes the | |
356 | virtual address where the kernel image is mapped and confuse | |
357 | gdb which resolve kernel symbol address from symbol table | |
358 | of vmlinux. | |
359 | ||
7fb2e8a4 MCC |
360 | Using kdb |
361 | ========= | |
362 | ||
363 | Quick start for kdb on a serial port | |
364 | ------------------------------------ | |
365 | ||
366 | This is a quick example of how to use kdb. | |
367 | ||
821c6df8 | 368 | 1. Configure kgdboc at boot using kernel parameters:: |
7fb2e8a4 | 369 | |
14994a9b | 370 | console=ttyS0,115200 kgdboc=ttyS0,115200 nokaslr |
7fb2e8a4 MCC |
371 | |
372 | OR | |
373 | ||
374 | Configure kgdboc after the kernel has booted; assuming you are using | |
821c6df8 | 375 | a serial port console:: |
7fb2e8a4 | 376 | |
821c6df8 | 377 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 MCC |
378 | |
379 | 2. Enter the kernel debugger manually or by waiting for an oops or | |
380 | fault. There are several ways you can enter the kernel debugger | |
821c6df8 MCC |
381 | manually; all involve using the :kbd:`SysRq-G`, which means you must have |
382 | enabled ``CONFIG_MAGIC_SysRq=y`` in your kernel config. | |
7fb2e8a4 | 383 | |
821c6df8 | 384 | - When logged in as root or with a super user session you can run:: |
7fb2e8a4 | 385 | |
821c6df8 | 386 | echo g > /proc/sysrq-trigger |
7fb2e8a4 MCC |
387 | |
388 | - Example using minicom 2.2 | |
389 | ||
821c6df8 | 390 | Press: :kbd:`CTRL-A` :kbd:`f` :kbd:`g` |
7fb2e8a4 MCC |
391 | |
392 | - When you have telneted to a terminal server that supports sending | |
393 | a remote break | |
394 | ||
821c6df8 | 395 | Press: :kbd:`CTRL-]` |
7fb2e8a4 | 396 | |
821c6df8 | 397 | Type in: ``send break`` |
7fb2e8a4 | 398 | |
821c6df8 | 399 | Press: :kbd:`Enter` :kbd:`g` |
7fb2e8a4 | 400 | |
821c6df8 | 401 | 3. From the kdb prompt you can run the ``help`` command to see a complete |
7fb2e8a4 MCC |
402 | list of the commands that are available. |
403 | ||
404 | Some useful commands in kdb include: | |
405 | ||
821c6df8 MCC |
406 | =========== ================================================================= |
407 | ``lsmod`` Shows where kernel modules are loaded | |
408 | ``ps`` Displays only the active processes | |
409 | ``ps A`` Shows all the processes | |
410 | ``summary`` Shows kernel version info and memory usage | |
411 | ``bt`` Get a backtrace of the current process using :c:func:`dump_stack` | |
412 | ``dmesg`` View the kernel syslog buffer | |
413 | ``go`` Continue the system | |
414 | =========== ================================================================= | |
7fb2e8a4 MCC |
415 | |
416 | 4. When you are done using kdb you need to consider rebooting the system | |
821c6df8 | 417 | or using the ``go`` command to resuming normal kernel execution. If you |
7fb2e8a4 MCC |
418 | have paused the kernel for a lengthy period of time, applications |
419 | that rely on timely networking or anything to do with real wall clock | |
420 | time could be adversely affected, so you should take this into | |
421 | consideration when using the kernel debugger. | |
422 | ||
423 | Quick start for kdb using a keyboard connected console | |
424 | ------------------------------------------------------ | |
425 | ||
426 | This is a quick example of how to use kdb with a keyboard. | |
427 | ||
821c6df8 | 428 | 1. Configure kgdboc at boot using kernel parameters:: |
7fb2e8a4 | 429 | |
821c6df8 | 430 | kgdboc=kbd |
7fb2e8a4 MCC |
431 | |
432 | OR | |
433 | ||
821c6df8 | 434 | Configure kgdboc after the kernel has booted:: |
7fb2e8a4 | 435 | |
821c6df8 | 436 | echo kbd > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 MCC |
437 | |
438 | 2. Enter the kernel debugger manually or by waiting for an oops or | |
439 | fault. There are several ways you can enter the kernel debugger | |
821c6df8 MCC |
440 | manually; all involve using the :kbd:`SysRq-G`, which means you must have |
441 | enabled ``CONFIG_MAGIC_SysRq=y`` in your kernel config. | |
7fb2e8a4 | 442 | |
821c6df8 | 443 | - When logged in as root or with a super user session you can run:: |
7fb2e8a4 | 444 | |
821c6df8 | 445 | echo g > /proc/sysrq-trigger |
7fb2e8a4 | 446 | |
821c6df8 | 447 | - Example using a laptop keyboard: |
7fb2e8a4 | 448 | |
821c6df8 | 449 | Press and hold down: :kbd:`Alt` |
7fb2e8a4 | 450 | |
821c6df8 | 451 | Press and hold down: :kbd:`Fn` |
7fb2e8a4 | 452 | |
821c6df8 | 453 | Press and release the key with the label: :kbd:`SysRq` |
7fb2e8a4 | 454 | |
821c6df8 | 455 | Release: :kbd:`Fn` |
7fb2e8a4 | 456 | |
821c6df8 | 457 | Press and release: :kbd:`g` |
7fb2e8a4 | 458 | |
821c6df8 | 459 | Release: :kbd:`Alt` |
7fb2e8a4 MCC |
460 | |
461 | - Example using a PS/2 101-key keyboard | |
462 | ||
821c6df8 | 463 | Press and hold down: :kbd:`Alt` |
7fb2e8a4 | 464 | |
821c6df8 | 465 | Press and release the key with the label: :kbd:`SysRq` |
7fb2e8a4 | 466 | |
821c6df8 | 467 | Press and release: :kbd:`g` |
7fb2e8a4 | 468 | |
821c6df8 | 469 | Release: :kbd:`Alt` |
7fb2e8a4 | 470 | |
821c6df8 | 471 | 3. Now type in a kdb command such as ``help``, ``dmesg``, ``bt`` or ``go`` to |
7fb2e8a4 MCC |
472 | continue kernel execution. |
473 | ||
474 | Using kgdb / gdb | |
475 | ================ | |
476 | ||
477 | In order to use kgdb you must activate it by passing configuration | |
478 | information to one of the kgdb I/O drivers. If you do not pass any | |
479 | configuration information kgdb will not do anything at all. Kgdb will | |
480 | only actively hook up to the kernel trap hooks if a kgdb I/O driver is | |
481 | loaded and configured. If you unconfigure a kgdb I/O driver, kgdb will | |
482 | unregister all the kernel hook points. | |
483 | ||
484 | All kgdb I/O drivers can be reconfigured at run time, if | |
485 | ``CONFIG_SYSFS`` and ``CONFIG_MODULES`` are enabled, by echo'ing a new | |
486 | config string to ``/sys/module/<driver>/parameter/<option>``. The driver | |
487 | can be unconfigured by passing an empty string. You cannot change the | |
488 | configuration while the debugger is attached. Make sure to detach the | |
489 | debugger with the ``detach`` command prior to trying to unconfigure a | |
490 | kgdb I/O driver. | |
491 | ||
492 | Connecting with gdb to a serial port | |
493 | ------------------------------------ | |
494 | ||
495 | 1. Configure kgdboc | |
496 | ||
821c6df8 | 497 | Configure kgdboc at boot using kernel parameters:: |
7fb2e8a4 | 498 | |
821c6df8 | 499 | kgdboc=ttyS0,115200 |
7fb2e8a4 MCC |
500 | |
501 | OR | |
502 | ||
821c6df8 | 503 | Configure kgdboc after the kernel has booted:: |
7fb2e8a4 | 504 | |
821c6df8 | 505 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc |
7fb2e8a4 MCC |
506 | |
507 | 2. Stop kernel execution (break into the debugger) | |
508 | ||
509 | In order to connect to gdb via kgdboc, the kernel must first be | |
510 | stopped. There are several ways to stop the kernel which include | |
821c6df8 | 511 | using kgdbwait as a boot argument, via a :kbd:`SysRq-G`, or running the |
7fb2e8a4 MCC |
512 | kernel until it takes an exception where it waits for the debugger to |
513 | attach. | |
514 | ||
821c6df8 | 515 | - When logged in as root or with a super user session you can run:: |
7fb2e8a4 | 516 | |
821c6df8 | 517 | echo g > /proc/sysrq-trigger |
7fb2e8a4 MCC |
518 | |
519 | - Example using minicom 2.2 | |
520 | ||
821c6df8 | 521 | Press: :kbd:`CTRL-A` :kbd:`f` :kbd:`g` |
7fb2e8a4 MCC |
522 | |
523 | - When you have telneted to a terminal server that supports sending | |
524 | a remote break | |
525 | ||
821c6df8 | 526 | Press: :kbd:`CTRL-]` |
7fb2e8a4 | 527 | |
821c6df8 | 528 | Type in: ``send break`` |
7fb2e8a4 | 529 | |
821c6df8 | 530 | Press: :kbd:`Enter` :kbd:`g` |
7fb2e8a4 MCC |
531 | |
532 | 3. Connect from gdb | |
533 | ||
821c6df8 | 534 | Example (using a directly connected port):: |
7fb2e8a4 MCC |
535 | |
536 | % gdb ./vmlinux | |
537 | (gdb) set remotebaud 115200 | |
538 | (gdb) target remote /dev/ttyS0 | |
539 | ||
540 | ||
821c6df8 | 541 | Example (kgdb to a terminal server on TCP port 2012):: |
7fb2e8a4 MCC |
542 | |
543 | % gdb ./vmlinux | |
544 | (gdb) target remote 192.168.2.2:2012 | |
545 | ||
546 | ||
547 | Once connected, you can debug a kernel the way you would debug an | |
548 | application program. | |
549 | ||
550 | If you are having problems connecting or something is going seriously | |
551 | wrong while debugging, it will most often be the case that you want | |
552 | to enable gdb to be verbose about its target communications. You do | |
821c6df8 MCC |
553 | this prior to issuing the ``target remote`` command by typing in:: |
554 | ||
555 | set debug remote 1 | |
7fb2e8a4 MCC |
556 | |
557 | Remember if you continue in gdb, and need to "break in" again, you need | |
821c6df8 MCC |
558 | to issue an other :kbd:`SysRq-G`. It is easy to create a simple entry point by |
559 | putting a breakpoint at ``sys_sync`` and then you can run ``sync`` from a | |
7fb2e8a4 MCC |
560 | shell or script to break into the debugger. |
561 | ||
562 | kgdb and kdb interoperability | |
563 | ============================= | |
564 | ||
565 | It is possible to transition between kdb and kgdb dynamically. The debug | |
566 | core will remember which you used the last time and automatically start | |
567 | in the same mode. | |
568 | ||
569 | Switching between kdb and kgdb | |
570 | ------------------------------ | |
571 | ||
572 | Switching from kgdb to kdb | |
573 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
574 | ||
575 | There are two ways to switch from kgdb to kdb: you can use gdb to issue | |
821c6df8 | 576 | a maintenance packet, or you can blindly type the command ``$3#33``. |
7fb2e8a4 MCC |
577 | Whenever the kernel debugger stops in kgdb mode it will print the |
578 | message ``KGDB or $3#33 for KDB``. It is important to note that you have | |
579 | to type the sequence correctly in one pass. You cannot type a backspace | |
580 | or delete because kgdb will interpret that as part of the debug stream. | |
581 | ||
821c6df8 | 582 | 1. Change from kgdb to kdb by blindly typing:: |
7fb2e8a4 | 583 | |
821c6df8 | 584 | $3#33 |
7fb2e8a4 | 585 | |
821c6df8 | 586 | 2. Change from kgdb to kdb with gdb:: |
7fb2e8a4 | 587 | |
821c6df8 | 588 | maintenance packet 3 |
7fb2e8a4 | 589 | |
821c6df8 MCC |
590 | .. note:: |
591 | ||
592 | Now you must kill gdb. Typically you press :kbd:`CTRL-Z` and issue | |
593 | the command:: | |
594 | ||
595 | kill -9 % | |
7fb2e8a4 MCC |
596 | |
597 | Change from kdb to kgdb | |
598 | ~~~~~~~~~~~~~~~~~~~~~~~ | |
599 | ||
600 | There are two ways you can change from kdb to kgdb. You can manually | |
601 | enter kgdb mode by issuing the kgdb command from the kdb shell prompt, | |
602 | or you can connect gdb while the kdb shell prompt is active. The kdb | |
603 | shell looks for the typical first commands that gdb would issue with the | |
604 | gdb remote protocol and if it sees one of those commands it | |
605 | automatically changes into kgdb mode. | |
606 | ||
821c6df8 | 607 | 1. From kdb issue the command:: |
7fb2e8a4 | 608 | |
821c6df8 | 609 | kgdb |
7fb2e8a4 MCC |
610 | |
611 | Now disconnect your terminal program and connect gdb in its place | |
612 | ||
613 | 2. At the kdb prompt, disconnect the terminal program and connect gdb in | |
614 | its place. | |
615 | ||
616 | Running kdb commands from gdb | |
617 | ----------------------------- | |
618 | ||
619 | It is possible to run a limited set of kdb commands from gdb, using the | |
620 | gdb monitor command. You don't want to execute any of the run control or | |
621 | breakpoint operations, because it can disrupt the state of the kernel | |
622 | debugger. You should be using gdb for breakpoints and run control | |
623 | operations if you have gdb connected. The more useful commands to run | |
624 | are things like lsmod, dmesg, ps or possibly some of the memory | |
625 | information commands. To see all the kdb commands you can run | |
626 | ``monitor help``. | |
627 | ||
821c6df8 | 628 | Example:: |
7fb2e8a4 MCC |
629 | |
630 | (gdb) monitor ps | |
631 | 1 idle process (state I) and | |
632 | 27 sleeping system daemon (state M) processes suppressed, | |
633 | use 'ps A' to see all. | |
634 | Task Addr Pid Parent [*] cpu State Thread Command | |
635 | ||
636 | 0xc78291d0 1 0 0 0 S 0xc7829404 init | |
637 | 0xc7954150 942 1 0 0 S 0xc7954384 dropbear | |
638 | 0xc78789c0 944 1 0 0 S 0xc7878bf4 sh | |
639 | (gdb) | |
640 | ||
7fb2e8a4 MCC |
641 | kgdb Test Suite |
642 | =============== | |
643 | ||
644 | When kgdb is enabled in the kernel config you can also elect to enable | |
821c6df8 | 645 | the config parameter ``KGDB_TESTS``. Turning this on will enable a special |
7fb2e8a4 MCC |
646 | kgdb I/O module which is designed to test the kgdb internal functions. |
647 | ||
648 | The kgdb tests are mainly intended for developers to test the kgdb | |
649 | internals as well as a tool for developing a new kgdb architecture | |
650 | specific implementation. These tests are not really for end users of the | |
651 | Linux kernel. The primary source of documentation would be to look in | |
821c6df8 | 652 | the ``drivers/misc/kgdbts.c`` file. |
7fb2e8a4 MCC |
653 | |
654 | The kgdb test suite can also be configured at compile time to run the | |
655 | core set of tests by setting the kernel config parameter | |
821c6df8 | 656 | ``KGDB_TESTS_ON_BOOT``. This particular option is aimed at automated |
7fb2e8a4 MCC |
657 | regression testing and does not require modifying the kernel boot config |
658 | arguments. If this is turned on, the kgdb test suite can be disabled by | |
821c6df8 | 659 | specifying ``kgdbts=`` as a kernel boot argument. |
7fb2e8a4 MCC |
660 | |
661 | Kernel Debugger Internals | |
662 | ========================= | |
663 | ||
664 | Architecture Specifics | |
665 | ---------------------- | |
666 | ||
667 | The kernel debugger is organized into a number of components: | |
668 | ||
669 | 1. The debug core | |
670 | ||
821c6df8 | 671 | The debug core is found in ``kernel/debugger/debug_core.c``. It |
7fb2e8a4 MCC |
672 | contains: |
673 | ||
674 | - A generic OS exception handler which includes sync'ing the | |
675 | processors into a stopped state on an multi-CPU system. | |
676 | ||
677 | - The API to talk to the kgdb I/O drivers | |
678 | ||
679 | - The API to make calls to the arch-specific kgdb implementation | |
680 | ||
681 | - The logic to perform safe memory reads and writes to memory while | |
682 | using the debugger | |
683 | ||
684 | - A full implementation for software breakpoints unless overridden | |
685 | by the arch | |
686 | ||
687 | - The API to invoke either the kdb or kgdb frontend to the debug | |
688 | core. | |
689 | ||
690 | - The structures and callback API for atomic kernel mode setting. | |
691 | ||
821c6df8 | 692 | .. note:: kgdboc is where the kms callbacks are invoked. |
7fb2e8a4 MCC |
693 | |
694 | 2. kgdb arch-specific implementation | |
695 | ||
821c6df8 MCC |
696 | This implementation is generally found in ``arch/*/kernel/kgdb.c``. As |
697 | an example, ``arch/x86/kernel/kgdb.c`` contains the specifics to | |
7fb2e8a4 MCC |
698 | implement HW breakpoint as well as the initialization to dynamically |
699 | register and unregister for the trap handlers on this architecture. | |
700 | The arch-specific portion implements: | |
701 | ||
702 | - contains an arch-specific trap catcher which invokes | |
821c6df8 | 703 | :c:func:`kgdb_handle_exception` to start kgdb about doing its work |
7fb2e8a4 | 704 | |
821c6df8 | 705 | - translation to and from gdb specific packet format to :c:type:`pt_regs` |
7fb2e8a4 MCC |
706 | |
707 | - Registration and unregistration of architecture specific trap | |
708 | hooks | |
709 | ||
710 | - Any special exception handling and cleanup | |
711 | ||
712 | - NMI exception handling and cleanup | |
713 | ||
714 | - (optional) HW breakpoints | |
715 | ||
716 | 3. gdbstub frontend (aka kgdb) | |
717 | ||
821c6df8 | 718 | The gdbstub is located in ``kernel/debug/gdbstub.c``. It contains: |
7fb2e8a4 MCC |
719 | |
720 | - All the logic to implement the gdb serial protocol | |
721 | ||
722 | 4. kdb frontend | |
723 | ||
724 | The kdb debugger shell is broken down into a number of components. | |
725 | The kdb core is located in kernel/debug/kdb. There are a number of | |
726 | helper functions in some of the other kernel components to make it | |
727 | possible for kdb to examine and report information about the kernel | |
728 | without taking locks that could cause a kernel deadlock. The kdb core | |
729 | contains implements the following functionality. | |
730 | ||
731 | - A simple shell | |
732 | ||
733 | - The kdb core command set | |
734 | ||
735 | - A registration API to register additional kdb shell commands. | |
736 | ||
821c6df8 | 737 | - A good example of a self-contained kdb module is the ``ftdump`` |
7fb2e8a4 | 738 | command for dumping the ftrace buffer. See: |
821c6df8 | 739 | ``kernel/trace/trace_kdb.c`` |
7fb2e8a4 MCC |
740 | |
741 | - For an example of how to dynamically register a new kdb command | |
742 | you can build the kdb_hello.ko kernel module from | |
821c6df8 MCC |
743 | ``samples/kdb/kdb_hello.c``. To build this example you can set |
744 | ``CONFIG_SAMPLES=y`` and ``CONFIG_SAMPLE_KDB=m`` in your kernel | |
745 | config. Later run ``modprobe kdb_hello`` and the next time you | |
746 | enter the kdb shell, you can run the ``hello`` command. | |
7fb2e8a4 | 747 | |
821c6df8 | 748 | - The implementation for :c:func:`kdb_printf` which emits messages directly |
7fb2e8a4 MCC |
749 | to I/O drivers, bypassing the kernel log. |
750 | ||
751 | - SW / HW breakpoint management for the kdb shell | |
752 | ||
753 | 5. kgdb I/O driver | |
754 | ||
755 | Each kgdb I/O driver has to provide an implementation for the | |
756 | following: | |
757 | ||
758 | - configuration via built-in or module | |
759 | ||
760 | - dynamic configuration and kgdb hook registration calls | |
761 | ||
762 | - read and write character interface | |
763 | ||
764 | - A cleanup handler for unconfiguring from the kgdb core | |
765 | ||
766 | - (optional) Early debug methodology | |
767 | ||
768 | Any given kgdb I/O driver has to operate very closely with the | |
769 | hardware and must do it in such a way that does not enable interrupts | |
770 | or change other parts of the system context without completely | |
771 | restoring them. The kgdb core will repeatedly "poll" a kgdb I/O | |
772 | driver for characters when it needs input. The I/O driver is expected | |
773 | to return immediately if there is no data available. Doing so allows | |
774 | for the future possibility to touch watchdog hardware in such a way | |
775 | as to have a target system not reset when these are enabled. | |
776 | ||
777 | If you are intent on adding kgdb architecture specific support for a new | |
778 | architecture, the architecture should define ``HAVE_ARCH_KGDB`` in the | |
779 | architecture specific Kconfig file. This will enable kgdb for the | |
780 | architecture, and at that point you must create an architecture specific | |
781 | kgdb implementation. | |
782 | ||
783 | There are a few flags which must be set on every architecture in their | |
821c6df8 | 784 | ``asm/kgdb.h`` file. These are: |
7fb2e8a4 | 785 | |
821c6df8 MCC |
786 | - ``NUMREGBYTES``: |
787 | The size in bytes of all of the registers, so that we | |
788 | can ensure they will all fit into a packet. | |
7fb2e8a4 | 789 | |
821c6df8 MCC |
790 | - ``BUFMAX``: |
791 | The size in bytes of the buffer GDB will read into. This must | |
792 | be larger than NUMREGBYTES. | |
7fb2e8a4 | 793 | |
821c6df8 MCC |
794 | - ``CACHE_FLUSH_IS_SAFE``: |
795 | Set to 1 if it is always safe to call | |
796 | flush_cache_range or flush_icache_range. On some architectures, | |
797 | these functions may not be safe to call on SMP since we keep other | |
798 | CPUs in a holding pattern. | |
7fb2e8a4 MCC |
799 | |
800 | There are also the following functions for the common backend, found in | |
821c6df8 | 801 | ``kernel/kgdb.c``, that must be supplied by the architecture-specific |
7fb2e8a4 MCC |
802 | backend unless marked as (optional), in which case a default function |
803 | maybe used if the architecture does not need to provide a specific | |
804 | implementation. | |
805 | ||
806 | .. kernel-doc:: include/linux/kgdb.h | |
807 | :internal: | |
808 | ||
809 | kgdboc internals | |
810 | ---------------- | |
811 | ||
812 | kgdboc and uarts | |
813 | ~~~~~~~~~~~~~~~~ | |
814 | ||
815 | The kgdboc driver is actually a very thin driver that relies on the | |
816 | underlying low level to the hardware driver having "polling hooks" to | |
817 | which the tty driver is attached. In the initial implementation of | |
818 | kgdboc the serial_core was changed to expose a low level UART hook for | |
819 | doing polled mode reading and writing of a single character while in an | |
820 | atomic context. When kgdb makes an I/O request to the debugger, kgdboc | |
821 | invokes a callback in the serial core which in turn uses the callback in | |
822 | the UART driver. | |
823 | ||
824 | When using kgdboc with a UART, the UART driver must implement two | |
821c6df8 MCC |
825 | callbacks in the :c:type:`struct uart_ops <uart_ops>`. |
826 | Example from ``drivers/8250.c``:: | |
7fb2e8a4 | 827 | |
7fb2e8a4 MCC |
828 | |
829 | #ifdef CONFIG_CONSOLE_POLL | |
830 | .poll_get_char = serial8250_get_poll_char, | |
831 | .poll_put_char = serial8250_put_poll_char, | |
832 | #endif | |
833 | ||
834 | ||
835 | Any implementation specifics around creating a polling driver use the | |
836 | ``#ifdef CONFIG_CONSOLE_POLL``, as shown above. Keep in mind that | |
837 | polling hooks have to be implemented in such a way that they can be | |
838 | called from an atomic context and have to restore the state of the UART | |
839 | chip on return such that the system can return to normal when the | |
840 | debugger detaches. You need to be very careful with any kind of lock you | |
841 | consider, because failing here is most likely going to mean pressing the | |
842 | reset button. | |
843 | ||
844 | kgdboc and keyboards | |
821c6df8 | 845 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
7fb2e8a4 MCC |
846 | |
847 | The kgdboc driver contains logic to configure communications with an | |
848 | attached keyboard. The keyboard infrastructure is only compiled into the | |
821c6df8 | 849 | kernel when ``CONFIG_KDB_KEYBOARD=y`` is set in the kernel configuration. |
7fb2e8a4 MCC |
850 | |
851 | The core polled keyboard driver driver for PS/2 type keyboards is in | |
821c6df8 | 852 | ``drivers/char/kdb_keyboard.c``. This driver is hooked into the debug core |
7fb2e8a4 | 853 | when kgdboc populates the callback in the array called |
821c6df8 | 854 | :c:type:`kdb_poll_funcs[]`. The :c:func:`kdb_get_kbd_char` is the top-level |
7fb2e8a4 MCC |
855 | function which polls hardware for single character input. |
856 | ||
857 | kgdboc and kms | |
821c6df8 | 858 | ~~~~~~~~~~~~~~~~~~ |
7fb2e8a4 MCC |
859 | |
860 | The kgdboc driver contains logic to request the graphics display to | |
821c6df8 | 861 | switch to a text context when you are using ``kgdboc=kms,kbd``, provided |
7fb2e8a4 MCC |
862 | that you have a video driver which has a frame buffer console and atomic |
863 | kernel mode setting support. | |
864 | ||
865 | Every time the kernel debugger is entered it calls | |
821c6df8 MCC |
866 | :c:func:`kgdboc_pre_exp_handler` which in turn calls :c:func:`con_debug_enter` |
867 | in the virtual console layer. On resuming kernel execution, the kernel | |
868 | debugger calls :c:func:`kgdboc_post_exp_handler` which in turn calls | |
869 | :c:func:`con_debug_leave`. | |
7fb2e8a4 MCC |
870 | |
871 | Any video driver that wants to be compatible with the kernel debugger | |
821c6df8 MCC |
872 | and the atomic kms callbacks must implement the ``mode_set_base_atomic``, |
873 | ``fb_debug_enter`` and ``fb_debug_leave operations``. For the | |
874 | ``fb_debug_enter`` and ``fb_debug_leave`` the option exists to use the | |
7fb2e8a4 MCC |
875 | generic drm fb helper functions or implement something custom for the |
876 | hardware. The following example shows the initialization of the | |
877 | .mode_set_base_atomic operation in | |
821c6df8 | 878 | drivers/gpu/drm/i915/intel_display.c:: |
7fb2e8a4 | 879 | |
7fb2e8a4 MCC |
880 | |
881 | static const struct drm_crtc_helper_funcs intel_helper_funcs = { | |
882 | [...] | |
883 | .mode_set_base_atomic = intel_pipe_set_base_atomic, | |
884 | [...] | |
885 | }; | |
886 | ||
887 | ||
7fb2e8a4 MCC |
888 | Here is an example of how the i915 driver initializes the |
889 | fb_debug_enter and fb_debug_leave functions to use the generic drm | |
821c6df8 | 890 | helpers in ``drivers/gpu/drm/i915/intel_fb.c``:: |
7fb2e8a4 | 891 | |
7fb2e8a4 MCC |
892 | |
893 | static struct fb_ops intelfb_ops = { | |
894 | [...] | |
895 | .fb_debug_enter = drm_fb_helper_debug_enter, | |
896 | .fb_debug_leave = drm_fb_helper_debug_leave, | |
897 | [...] | |
898 | }; | |
899 | ||
900 | ||
7fb2e8a4 MCC |
901 | Credits |
902 | ======= | |
903 | ||
904 | The following people have contributed to this document: | |
905 | ||
821c6df8 | 906 | 1. Amit Kale <amitkale@linsyssoft.com> |
7fb2e8a4 | 907 | |
821c6df8 | 908 | 2. Tom Rini <trini@kernel.crashing.org> |
7fb2e8a4 MCC |
909 | |
910 | In March 2008 this document was completely rewritten by: | |
911 | ||
821c6df8 | 912 | - Jason Wessel <jason.wessel@windriver.com> |
7fb2e8a4 MCC |
913 | |
914 | In Jan 2010 this document was updated to include kdb. | |
915 | ||
821c6df8 | 916 | - Jason Wessel <jason.wessel@windriver.com> |