| 1 | .. SPDX-License-Identifier: GPL-2.0 |
| 2 | |
| 3 | ========== |
| 4 | Netconsole |
| 5 | ========== |
| 6 | |
| 7 | |
| 8 | started by Ingo Molnar <mingo@redhat.com>, 2001.09.17 |
| 9 | |
| 10 | 2.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003 |
| 11 | |
| 12 | IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013 |
| 13 | |
| 14 | Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015 |
| 15 | |
| 16 | Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023 |
| 17 | |
| 18 | Please send bug reports to Matt Mackall <mpm@selenic.com> |
| 19 | Satyam Sharma <satyam.sharma@gmail.com>, and Cong Wang <xiyou.wangcong@gmail.com> |
| 20 | |
| 21 | Introduction: |
| 22 | ============= |
| 23 | |
| 24 | This module logs kernel printk messages over UDP allowing debugging of |
| 25 | problem where disk logging fails and serial consoles are impractical. |
| 26 | |
| 27 | It can be used either built-in or as a module. As a built-in, |
| 28 | netconsole initializes immediately after NIC cards and will bring up |
| 29 | the specified interface as soon as possible. While this doesn't allow |
| 30 | capture of early kernel panics, it does capture most of the boot |
| 31 | process. |
| 32 | |
| 33 | Sender and receiver configuration: |
| 34 | ================================== |
| 35 | |
| 36 | It takes a string configuration parameter "netconsole" in the |
| 37 | following format:: |
| 38 | |
| 39 | netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr] |
| 40 | |
| 41 | where |
| 42 | + if present, enable extended console support |
| 43 | r if present, prepend kernel version (release) to the message |
| 44 | src-port source for UDP packets (defaults to 6665) |
| 45 | src-ip source IP to use (interface address) |
| 46 | dev network interface (eth0) |
| 47 | tgt-port port for logging agent (6666) |
| 48 | tgt-ip IP address for logging agent |
| 49 | tgt-macaddr ethernet MAC address for logging agent (broadcast) |
| 50 | |
| 51 | Examples:: |
| 52 | |
| 53 | linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc |
| 54 | |
| 55 | or:: |
| 56 | |
| 57 | insmod netconsole netconsole=@/,@10.0.0.2/ |
| 58 | |
| 59 | or using IPv6:: |
| 60 | |
| 61 | insmod netconsole netconsole=@/,@fd00:1:2:3::1/ |
| 62 | |
| 63 | It also supports logging to multiple remote agents by specifying |
| 64 | parameters for the multiple agents separated by semicolons and the |
| 65 | complete string enclosed in "quotes", thusly:: |
| 66 | |
| 67 | modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/" |
| 68 | |
| 69 | Built-in netconsole starts immediately after the TCP stack is |
| 70 | initialized and attempts to bring up the supplied dev at the supplied |
| 71 | address. |
| 72 | |
| 73 | The remote host has several options to receive the kernel messages, |
| 74 | for example: |
| 75 | |
| 76 | 1) syslogd |
| 77 | |
| 78 | 2) netcat |
| 79 | |
| 80 | On distributions using a BSD-based netcat version (e.g. Fedora, |
| 81 | openSUSE and Ubuntu) the listening port must be specified without |
| 82 | the -p switch:: |
| 83 | |
| 84 | nc -u -l -p <port>' / 'nc -u -l <port> |
| 85 | |
| 86 | or:: |
| 87 | |
| 88 | netcat -u -l -p <port>' / 'netcat -u -l <port> |
| 89 | |
| 90 | 3) socat |
| 91 | |
| 92 | :: |
| 93 | |
| 94 | socat udp-recv:<port> - |
| 95 | |
| 96 | Dynamic reconfiguration: |
| 97 | ======================== |
| 98 | |
| 99 | Dynamic reconfigurability is a useful addition to netconsole that enables |
| 100 | remote logging targets to be dynamically added, removed, or have their |
| 101 | parameters reconfigured at runtime from a configfs-based userspace interface. |
| 102 | |
| 103 | To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the |
| 104 | netconsole module (or kernel, if netconsole is built-in). |
| 105 | |
| 106 | Some examples follow (where configfs is mounted at the /sys/kernel/config |
| 107 | mountpoint). |
| 108 | |
| 109 | To add a remote logging target (target names can be arbitrary):: |
| 110 | |
| 111 | cd /sys/kernel/config/netconsole/ |
| 112 | mkdir target1 |
| 113 | |
| 114 | Note that newly created targets have default parameter values (as mentioned |
| 115 | above) and are disabled by default -- they must first be enabled by writing |
| 116 | "1" to the "enabled" attribute (usually after setting parameters accordingly) |
| 117 | as described below. |
| 118 | |
| 119 | To remove a target:: |
| 120 | |
| 121 | rmdir /sys/kernel/config/netconsole/othertarget/ |
| 122 | |
| 123 | The interface exposes these parameters of a netconsole target to userspace: |
| 124 | |
| 125 | ============== ================================= ============ |
| 126 | enabled Is this target currently enabled? (read-write) |
| 127 | extended Extended mode enabled (read-write) |
| 128 | release Prepend kernel release to message (read-write) |
| 129 | dev_name Local network interface name (read-write) |
| 130 | local_port Source UDP port to use (read-write) |
| 131 | remote_port Remote agent's UDP port (read-write) |
| 132 | local_ip Source IP address to use (read-write) |
| 133 | remote_ip Remote agent's IP address (read-write) |
| 134 | local_mac Local interface's MAC address (read-only) |
| 135 | remote_mac Remote agent's MAC address (read-write) |
| 136 | ============== ================================= ============ |
| 137 | |
| 138 | The "enabled" attribute is also used to control whether the parameters of |
| 139 | a target can be updated or not -- you can modify the parameters of only |
| 140 | disabled targets (i.e. if "enabled" is 0). |
| 141 | |
| 142 | To update a target's parameters:: |
| 143 | |
| 144 | cat enabled # check if enabled is 1 |
| 145 | echo 0 > enabled # disable the target (if required) |
| 146 | echo eth2 > dev_name # set local interface |
| 147 | echo 10.0.0.4 > remote_ip # update some parameter |
| 148 | echo cb:a9:87:65:43:21 > remote_mac # update more parameters |
| 149 | echo 1 > enabled # enable target again |
| 150 | |
| 151 | You can also update the local interface dynamically. This is especially |
| 152 | useful if you want to use interfaces that have newly come up (and may not |
| 153 | have existed when netconsole was loaded / initialized). |
| 154 | |
| 155 | Netconsole targets defined at boot time (or module load time) with the |
| 156 | `netconsole=` param are assigned the name `cmdline<index>`. For example, the |
| 157 | first target in the parameter is named `cmdline0`. You can control and modify |
| 158 | these targets by creating configfs directories with the matching name. |
| 159 | |
| 160 | Let's suppose you have two netconsole targets defined at boot time:: |
| 161 | |
| 162 | netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc |
| 163 | |
| 164 | You can modify these targets in runtime by creating the following targets:: |
| 165 | |
| 166 | mkdir cmdline0 |
| 167 | cat cmdline0/remote_ip |
| 168 | 10.0.0.2 |
| 169 | |
| 170 | mkdir cmdline1 |
| 171 | cat cmdline1/remote_ip |
| 172 | 10.0.0.3 |
| 173 | |
| 174 | Extended console: |
| 175 | ================= |
| 176 | |
| 177 | If '+' is prefixed to the configuration line or "extended" config file |
| 178 | is set to 1, extended console support is enabled. An example boot |
| 179 | param follows:: |
| 180 | |
| 181 | linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc |
| 182 | |
| 183 | Log messages are transmitted with extended metadata header in the |
| 184 | following format which is the same as /dev/kmsg:: |
| 185 | |
| 186 | <level>,<sequnum>,<timestamp>,<contflag>;<message text> |
| 187 | |
| 188 | If 'r' (release) feature is enabled, the kernel release version is |
| 189 | prepended to the start of the message. Example:: |
| 190 | |
| 191 | 6.4.0,6,444,501151268,-;netconsole: network logging started |
| 192 | |
| 193 | Non printable characters in <message text> are escaped using "\xff" |
| 194 | notation. If the message contains optional dictionary, verbatim |
| 195 | newline is used as the delimiter. |
| 196 | |
| 197 | If a message doesn't fit in certain number of bytes (currently 1000), |
| 198 | the message is split into multiple fragments by netconsole. These |
| 199 | fragments are transmitted with "ncfrag" header field added:: |
| 200 | |
| 201 | ncfrag=<byte-offset>/<total-bytes> |
| 202 | |
| 203 | For example, assuming a lot smaller chunk size, a message "the first |
| 204 | chunk, the 2nd chunk." may be split as follows:: |
| 205 | |
| 206 | 6,416,1758426,-,ncfrag=0/31;the first chunk, |
| 207 | 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. |
| 208 | |
| 209 | Miscellaneous notes: |
| 210 | ==================== |
| 211 | |
| 212 | .. Warning:: |
| 213 | |
| 214 | the default target ethernet setting uses the broadcast |
| 215 | ethernet address to send packets, which can cause increased load on |
| 216 | other systems on the same ethernet segment. |
| 217 | |
| 218 | .. Tip:: |
| 219 | |
| 220 | some LAN switches may be configured to suppress ethernet broadcasts |
| 221 | so it is advised to explicitly specify the remote agents' MAC addresses |
| 222 | from the config parameters passed to netconsole. |
| 223 | |
| 224 | .. Tip:: |
| 225 | |
| 226 | to find out the MAC address of, say, 10.0.0.2, you may try using:: |
| 227 | |
| 228 | ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2 |
| 229 | |
| 230 | .. Tip:: |
| 231 | |
| 232 | in case the remote logging agent is on a separate LAN subnet than |
| 233 | the sender, it is suggested to try specifying the MAC address of the |
| 234 | default gateway (you may use /sbin/route -n to find it out) as the |
| 235 | remote MAC address instead. |
| 236 | |
| 237 | .. note:: |
| 238 | |
| 239 | the network device (eth1 in the above case) can run any kind |
| 240 | of other network traffic, netconsole is not intrusive. Netconsole |
| 241 | might cause slight delays in other traffic if the volume of kernel |
| 242 | messages is high, but should have no other impact. |
| 243 | |
| 244 | .. note:: |
| 245 | |
| 246 | if you find that the remote logging agent is not receiving or |
| 247 | printing all messages from the sender, it is likely that you have set |
| 248 | the "console_loglevel" parameter (on the sender) to only send high |
| 249 | priority messages to the console. You can change this at runtime using:: |
| 250 | |
| 251 | dmesg -n 8 |
| 252 | |
| 253 | or by specifying "debug" on the kernel command line at boot, to send |
| 254 | all kernel messages to the console. A specific value for this parameter |
| 255 | can also be set using the "loglevel" kernel boot option. See the |
| 256 | dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst |
| 257 | for details. |
| 258 | |
| 259 | Netconsole was designed to be as instantaneous as possible, to |
| 260 | enable the logging of even the most critical kernel bugs. It works |
| 261 | from IRQ contexts as well, and does not enable interrupts while |
| 262 | sending packets. Due to these unique needs, configuration cannot |
| 263 | be more automatic, and some fundamental limitations will remain: |
| 264 | only IP networks, UDP packets and ethernet devices are supported. |