b08794a9 |
1 | =========== |
2 | SNMP counter |
3 | =========== |
4 | |
5 | This document explains the meaning of SNMP counters. |
6 | |
7 | General IPv4 counters |
8 | ==================== |
9 | All layer 4 packets and ICMP packets will change these counters, but |
10 | these counters won't be changed by layer 2 packets (such as STP) or |
11 | ARP packets. |
12 | |
13 | * IpInReceives |
14 | Defined in `RFC1213 ipInReceives`_ |
15 | |
16 | .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26 |
17 | |
18 | The number of packets received by the IP layer. It gets increasing at the |
19 | beginning of ip_rcv function, always be updated together with |
8e2ea53a |
20 | IpExtInOctets. It will be increased even if the packet is dropped |
21 | later (e.g. due to the IP header is invalid or the checksum is wrong |
22 | and so on). It indicates the number of aggregated segments after |
b08794a9 |
23 | GRO/LRO. |
24 | |
25 | * IpInDelivers |
26 | Defined in `RFC1213 ipInDelivers`_ |
27 | |
28 | .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28 |
29 | |
30 | The number of packets delivers to the upper layer protocols. E.g. TCP, UDP, |
31 | ICMP and so on. If no one listens on a raw socket, only kernel |
32 | supported protocols will be delivered, if someone listens on the raw |
33 | socket, all valid IP packets will be delivered. |
34 | |
35 | * IpOutRequests |
36 | Defined in `RFC1213 ipOutRequests`_ |
37 | |
38 | .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28 |
39 | |
40 | The number of packets sent via IP layer, for both single cast and |
41 | multicast packets, and would always be updated together with |
42 | IpExtOutOctets. |
43 | |
44 | * IpExtInOctets and IpExtOutOctets |
80cc4950 |
45 | They are Linux kernel extensions, no RFC definitions. Please note, |
b08794a9 |
46 | RFC1213 indeed defines ifInOctets and ifOutOctets, but they |
47 | are different things. The ifInOctets and ifOutOctets include the MAC |
48 | layer header size but IpExtInOctets and IpExtOutOctets don't, they |
49 | only include the IP layer header and the IP layer data. |
50 | |
51 | * IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts |
52 | They indicate the number of four kinds of ECN IP packets, please refer |
53 | `Explicit Congestion Notification`_ for more details. |
54 | |
55 | .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6 |
56 | |
57 | These 4 counters calculate how many packets received per ECN |
58 | status. They count the real frame number regardless the LRO/GRO. So |
59 | for the same packet, you might find that IpInReceives count 1, but |
60 | IpExtInNoECTPkts counts 2 or more. |
61 | |
8e2ea53a |
62 | * IpInHdrErrors |
63 | Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is |
64 | dropped due to the IP header error. It might happen in both IP input |
65 | and IP forward paths. |
66 | |
67 | .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27 |
68 | |
69 | * IpInAddrErrors |
70 | Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two |
71 | scenarios: (1) The IP address is invalid. (2) The destination IP |
72 | address is not a local address and IP forwarding is not enabled |
73 | |
74 | .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27 |
75 | |
76 | * IpExtInNoRoutes |
77 | This counter means the packet is dropped when the IP stack receives a |
78 | packet and can't find a route for it from the route table. It might |
79 | happen when IP forwarding is enabled and the destination IP address is |
80 | not a local address and there is no route for the destination IP |
81 | address. |
82 | |
83 | * IpInUnknownProtos |
84 | Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the |
85 | layer 4 protocol is unsupported by kernel. If an application is using |
86 | raw socket, kernel will always deliver the packet to the raw socket |
87 | and this counter won't be increased. |
88 | |
89 | .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27 |
90 | |
91 | * IpExtInTruncatedPkts |
92 | For IPv4 packet, it means the actual data size is smaller than the |
93 | "Total Length" field in the IPv4 header. |
94 | |
95 | * IpInDiscards |
96 | Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped |
97 | in the IP receiving path and due to kernel internal reasons (e.g. no |
98 | enough memory). |
99 | |
100 | .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28 |
101 | |
102 | * IpOutDiscards |
103 | Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is |
104 | dropped in the IP sending path and due to kernel internal reasons. |
105 | |
106 | .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28 |
107 | |
108 | * IpOutNoRoutes |
109 | Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is |
110 | dropped in the IP sending path and no route is found for it. |
111 | |
112 | .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29 |
113 | |
b08794a9 |
114 | ICMP counters |
115 | ============ |
116 | * IcmpInMsgs and IcmpOutMsgs |
117 | Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_ |
118 | |
119 | .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41 |
120 | .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43 |
121 | |
122 | As mentioned in the RFC1213, these two counters include errors, they |
123 | would be increased even if the ICMP packet has an invalid type. The |
124 | ICMP output path will check the header of a raw socket, so the |
125 | IcmpOutMsgs would still be updated if the IP header is constructed by |
126 | a userspace program. |
127 | |
128 | * ICMP named types |
129 | | These counters include most of common ICMP types, they are: |
130 | | IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_ |
131 | | IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_ |
132 | | IcmpInParmProbs: `RFC1213 icmpInParmProbs`_ |
133 | | IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_ |
134 | | IcmpInRedirects: `RFC1213 icmpInRedirects`_ |
135 | | IcmpInEchos: `RFC1213 icmpInEchos`_ |
136 | | IcmpInEchoReps: `RFC1213 icmpInEchoReps`_ |
137 | | IcmpInTimestamps: `RFC1213 icmpInTimestamps`_ |
138 | | IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_ |
139 | | IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_ |
140 | | IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_ |
141 | | IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_ |
142 | | IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_ |
143 | | IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_ |
144 | | IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_ |
145 | | IcmpOutRedirects: `RFC1213 icmpOutRedirects`_ |
146 | | IcmpOutEchos: `RFC1213 icmpOutEchos`_ |
147 | | IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_ |
148 | | IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_ |
149 | | IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_ |
150 | | IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_ |
151 | | IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_ |
152 | |
153 | .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41 |
154 | .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41 |
155 | .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42 |
156 | .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42 |
157 | .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42 |
158 | .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42 |
159 | .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42 |
160 | .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42 |
161 | .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43 |
162 | .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43 |
163 | .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43 |
164 | |
165 | .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44 |
166 | .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44 |
167 | .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44 |
168 | .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44 |
169 | .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44 |
170 | .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45 |
171 | .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45 |
172 | .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45 |
173 | .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45 |
174 | .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45 |
175 | .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46 |
176 | |
177 | Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP |
178 | Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are |
179 | straightforward. The 'In' counter means kernel receives such a packet |
180 | and the 'Out' counter means kernel sends such a packet. |
181 | |
182 | * ICMP numeric types |
183 | They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the |
184 | ICMP type number. These counters track all kinds of ICMP packets. The |
185 | ICMP type number definition could be found in the `ICMP parameters`_ |
186 | document. |
187 | |
188 | .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml |
189 | |
190 | For example, if the Linux kernel sends an ICMP Echo packet, the |
191 | IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply |
192 | packet, IcmpMsgInType0 would increase 1. |
193 | |
194 | * IcmpInCsumErrors |
195 | This counter indicates the checksum of the ICMP packet is |
196 | wrong. Kernel verifies the checksum after updating the IcmpInMsgs and |
197 | before updating IcmpMsgInType[N]. If a packet has bad checksum, the |
198 | IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated. |
199 | |
200 | * IcmpInErrors and IcmpOutErrors |
201 | Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_ |
202 | |
203 | .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41 |
204 | .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43 |
205 | |
206 | When an error occurs in the ICMP packet handler path, these two |
207 | counters would be updated. The receiving packet path use IcmpInErrors |
208 | and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors |
209 | is increased, IcmpInErrors would always be increased too. |
210 | |
211 | relationship of the ICMP counters |
212 | ------------------------------- |
213 | The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they |
214 | are updated at the same time. The sum of IcmpMsgInType[N] plus |
215 | IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel |
216 | receives an ICMP packet, kernel follows below logic: |
217 | |
218 | 1. increase IcmpInMsgs |
219 | 2. if has any error, update IcmpInErrors and finish the process |
220 | 3. update IcmpMsgOutType[N] |
221 | 4. handle the packet depending on the type, if has any error, update |
222 | IcmpInErrors and finish the process |
223 | |
224 | So if all errors occur in step (2), IcmpInMsgs should be equal to the |
225 | sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in |
226 | step (4), IcmpInMsgs should be equal to the sum of |
227 | IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4), |
228 | IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus |
229 | IcmpInErrors. |
230 | |
80cc4950 |
231 | General TCP counters |
232 | ================== |
233 | * TcpInSegs |
234 | Defined in `RFC1213 tcpInSegs`_ |
235 | |
236 | .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48 |
237 | |
238 | The number of packets received by the TCP layer. As mentioned in |
239 | RFC1213, it includes the packets received in error, such as checksum |
240 | error, invalid TCP header and so on. Only one error won't be included: |
241 | if the layer 2 destination address is not the NIC's layer 2 |
242 | address. It might happen if the packet is a multicast or broadcast |
243 | packet, or the NIC is in promiscuous mode. In these situations, the |
244 | packets would be delivered to the TCP layer, but the TCP layer will discard |
245 | these packets before increasing TcpInSegs. The TcpInSegs counter |
246 | isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs |
247 | counter would only increase 1. |
248 | |
249 | * TcpOutSegs |
250 | Defined in `RFC1213 tcpOutSegs`_ |
251 | |
252 | .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48 |
253 | |
254 | The number of packets sent by the TCP layer. As mentioned in RFC1213, |
255 | it excludes the retransmitted packets. But it includes the SYN, ACK |
256 | and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of |
257 | GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will |
258 | increase 2. |
259 | |
260 | * TcpActiveOpens |
261 | Defined in `RFC1213 tcpActiveOpens`_ |
262 | |
263 | .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47 |
264 | |
265 | It means the TCP layer sends a SYN, and come into the SYN-SENT |
266 | state. Every time TcpActiveOpens increases 1, TcpOutSegs should always |
267 | increase 1. |
268 | |
269 | * TcpPassiveOpens |
270 | Defined in `RFC1213 tcpPassiveOpens`_ |
271 | |
272 | .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47 |
273 | |
274 | It means the TCP layer receives a SYN, replies a SYN+ACK, come into |
275 | the SYN-RCVD state. |
276 | |
712ee16c |
277 | * TcpExtTCPRcvCoalesce |
278 | When packets are received by the TCP layer and are not be read by the |
279 | application, the TCP layer will try to merge them. This counter |
280 | indicate how many packets are merged in such situation. If GRO is |
281 | enabled, lots of packets would be merged by GRO, these packets |
282 | wouldn't be counted to TcpExtTCPRcvCoalesce. |
283 | |
284 | * TcpExtTCPAutoCorking |
285 | When sending packets, the TCP layer will try to merge small packets to |
286 | a bigger one. This counter increase 1 for every packet merged in such |
287 | situation. Please refer to the LWN article for more details: |
288 | https://lwn.net/Articles/576263/ |
289 | |
290 | * TcpExtTCPOrigDataSent |
291 | This counter is explained by `kernel commit f19c29e3e391`_, I pasted the |
292 | explaination below:: |
293 | |
294 | TCPOrigDataSent: number of outgoing packets with original data (excluding |
295 | retransmission but including data-in-SYN). This counter is different from |
296 | TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is |
297 | more useful to track the TCP retransmission rate. |
298 | |
299 | * TCPSynRetrans |
300 | This counter is explained by `kernel commit f19c29e3e391`_, I pasted the |
301 | explaination below:: |
302 | |
303 | TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down |
304 | retransmissions into SYN, fast-retransmits, timeout retransmits, etc. |
305 | |
306 | * TCPFastOpenActiveFail |
307 | This counter is explained by `kernel commit f19c29e3e391`_, I pasted the |
308 | explaination below:: |
309 | |
310 | TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because |
311 | the remote does not accept it or the attempts timed out. |
312 | |
313 | .. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd |
314 | |
315 | * TcpExtListenOverflows and TcpExtListenDrops |
316 | When kernel receives a SYN from a client, and if the TCP accept queue |
317 | is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows. |
318 | At the same time kernel will also add 1 to TcpExtListenDrops. When a |
319 | TCP socket is in LISTEN state, and kernel need to drop a packet, |
320 | kernel would always add 1 to TcpExtListenDrops. So increase |
321 | TcpExtListenOverflows would let TcpExtListenDrops increasing at the |
322 | same time, but TcpExtListenDrops would also increase without |
323 | TcpExtListenOverflows increasing, e.g. a memory allocation fail would |
324 | also let TcpExtListenDrops increase. |
325 | |
326 | Note: The above explanation is based on kernel 4.10 or above version, on |
327 | an old kernel, the TCP stack has different behavior when TCP accept |
328 | queue is full. On the old kernel, TCP stack won't drop the SYN, it |
329 | would complete the 3-way handshake. As the accept queue is full, TCP |
330 | stack will keep the socket in the TCP half-open queue. As it is in the |
331 | half open queue, TCP stack will send SYN+ACK on an exponential backoff |
332 | timer, after client replies ACK, TCP stack checks whether the accept |
333 | queue is still full, if it is not full, moves the socket to the accept |
334 | queue, if it is full, keeps the socket in the half-open queue, at next |
335 | time client replies ACK, this socket will get another chance to move |
336 | to the accept queue. |
337 | |
338 | |
a6c7c7aa |
339 | * TcpEstabResets |
340 | Defined in `RFC1213 tcpEstabResets`_. |
341 | |
342 | .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48 |
343 | |
344 | * TcpAttemptFails |
345 | Defined in `RFC1213 tcpAttemptFails`_. |
346 | |
347 | .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48 |
348 | |
349 | * TcpOutRsts |
350 | Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates |
351 | the 'segments sent containing the RST flag', but in linux kernel, this |
352 | couner indicates the segments kerenl tried to send. The sending |
353 | process might be failed due to some errors (e.g. memory alloc failed). |
354 | |
355 | .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52 |
356 | |
357 | |
358 | TCP Fast Path |
80cc4950 |
359 | ============ |
360 | When kernel receives a TCP packet, it has two paths to handler the |
361 | packet, one is fast path, another is slow path. The comment in kernel |
362 | code provides a good explanation of them, I pasted them below:: |
363 | |
364 | It is split into a fast path and a slow path. The fast path is |
365 | disabled when: |
366 | |
367 | - A zero window was announced from us |
368 | - zero window probing |
369 | is only handled properly on the slow path. |
370 | - Out of order segments arrived. |
371 | - Urgent data is expected. |
372 | - There is no buffer space left |
373 | - Unexpected TCP flags/window values/header lengths are received |
374 | (detected by checking the TCP header against pred_flags) |
375 | - Data is sent in both directions. The fast path only supports pure senders |
376 | or pure receivers (this means either the sequence number or the ack |
377 | value must stay constant) |
378 | - Unexpected TCP option. |
379 | |
380 | Kernel will try to use fast path unless any of the above conditions |
381 | are satisfied. If the packets are out of order, kernel will handle |
382 | them in slow path, which means the performance might be not very |
383 | good. Kernel would also come into slow path if the "Delayed ack" is |
384 | used, because when using "Delayed ack", the data is sent in both |
385 | directions. When the TCP window scale option is not used, kernel will |
386 | try to enable fast path immediately when the connection comes into the |
387 | established state, but if the TCP window scale option is used, kernel |
388 | will disable the fast path at first, and try to enable it after kernel |
389 | receives packets. |
390 | |
391 | * TcpExtTCPPureAcks and TcpExtTCPHPAcks |
392 | If a packet set ACK flag and has no data, it is a pure ACK packet, if |
393 | kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1, |
394 | if kernel handles it in the slow path, TcpExtTCPPureAcks will |
395 | increase 1. |
396 | |
397 | * TcpExtTCPHPHits |
398 | If a TCP packet has data (which means it is not a pure ACK packet), |
399 | and this packet is handled in the fast path, TcpExtTCPHPHits will |
400 | increase 1. |
401 | |
402 | |
403 | TCP abort |
404 | ======== |
80cc4950 |
405 | * TcpExtTCPAbortOnData |
406 | It means TCP layer has data in flight, but need to close the |
407 | connection. So TCP layer sends a RST to the other side, indicate the |
408 | connection is not closed very graceful. An easy way to increase this |
409 | counter is using the SO_LINGER option. Please refer to the SO_LINGER |
410 | section of the `socket man page`_: |
411 | |
412 | .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html |
413 | |
414 | By default, when an application closes a connection, the close function |
415 | will return immediately and kernel will try to send the in-flight data |
416 | async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger |
417 | to a positive number, the close function won't return immediately, but |
418 | wait for the in-flight data are acked by the other side, the max wait |
419 | time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0, |
420 | when the application closes a connection, kernel will send a RST |
421 | immediately and increase the TcpExtTCPAbortOnData counter. |
422 | |
423 | * TcpExtTCPAbortOnClose |
424 | This counter means the application has unread data in the TCP layer when |
425 | the application wants to close the TCP connection. In such a situation, |
426 | kernel will send a RST to the other side of the TCP connection. |
427 | |
428 | * TcpExtTCPAbortOnMemory |
429 | When an application closes a TCP connection, kernel still need to track |
430 | the connection, let it complete the TCP disconnect process. E.g. an |
431 | app calls the close method of a socket, kernel sends fin to the other |
432 | side of the connection, then the app has no relationship with the |
433 | socket any more, but kernel need to keep the socket, this socket |
434 | becomes an orphan socket, kernel waits for the reply of the other side, |
435 | and would come to the TIME_WAIT state finally. When kernel has no |
436 | enough memory to keep the orphan socket, kernel would send an RST to |
437 | the other side, and delete the socket, in such situation, kernel will |
438 | increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger |
439 | TcpExtTCPAbortOnMemory: |
440 | |
441 | 1. the memory used by the TCP protocol is higher than the third value of |
442 | the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_: |
443 | |
444 | .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html |
445 | |
446 | 2. the orphan socket count is higher than net.ipv4.tcp_max_orphans |
447 | |
448 | |
449 | * TcpExtTCPAbortOnTimeout |
450 | This counter will increase when any of the TCP timers expire. In such |
451 | situation, kernel won't send RST, just give up the connection. |
452 | |
453 | * TcpExtTCPAbortOnLinger |
454 | When a TCP connection comes into FIN_WAIT_2 state, instead of waiting |
455 | for the fin packet from the other side, kernel could send a RST and |
456 | delete the socket immediately. This is not the default behavior of |
457 | Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option, |
458 | you could let kernel follow this behavior. |
459 | |
460 | * TcpExtTCPAbortFailed |
461 | The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is |
462 | satisfied. If an internal error occurs during this process, |
463 | TcpExtTCPAbortFailed will be increased. |
464 | |
465 | .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50 |
466 | |
712ee16c |
467 | TCP Hybrid Slow Start |
468 | ==================== |
469 | The Hybrid Slow Start algorithm is an enhancement of the traditional |
470 | TCP congestion window Slow Start algorithm. It uses two pieces of |
471 | information to detect whether the max bandwidth of the TCP path is |
472 | approached. The two pieces of information are ACK train length and |
473 | increase in packet delay. For detail information, please refer the |
474 | `Hybrid Slow Start paper`_. Either ACK train length or packet delay |
475 | hits a specific threshold, the congestion control algorithm will come |
476 | into the Congestion Avoidance state. Until v4.20, two congestion |
477 | control algorithms are using Hybrid Slow Start, they are cubic (the |
478 | default congestion control algorithm) and cdg. Four snmp counters |
479 | relate with the Hybrid Slow Start algorithm. |
480 | |
481 | .. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf |
482 | |
483 | * TcpExtTCPHystartTrainDetect |
484 | How many times the ACK train length threshold is detected |
485 | |
486 | * TcpExtTCPHystartTrainCwnd |
487 | The sum of CWND detected by ACK train length. Dividing this value by |
488 | TcpExtTCPHystartTrainDetect is the average CWND which detected by the |
489 | ACK train length. |
490 | |
491 | * TcpExtTCPHystartDelayDetect |
492 | How many times the packet delay threshold is detected. |
493 | |
494 | * TcpExtTCPHystartDelayCwnd |
495 | The sum of CWND detected by packet delay. Dividing this value by |
496 | TcpExtTCPHystartDelayDetect is the average CWND which detected by the |
497 | packet delay. |
498 | |
8e2ea53a |
499 | TCP retransmission and congestion control |
500 | ====================================== |
501 | The TCP protocol has two retransmission mechanisms: SACK and fast |
502 | recovery. They are exclusive with each other. When SACK is enabled, |
503 | the kernel TCP stack would use SACK, or kernel would use fast |
504 | recovery. The SACK is a TCP option, which is defined in `RFC2018`_, |
505 | the fast recovery is defined in `RFC6582`_, which is also called |
506 | 'Reno'. |
507 | |
508 | The TCP congestion control is a big and complex topic. To understand |
509 | the related snmp counter, we need to know the states of the congestion |
510 | control state machine. There are 5 states: Open, Disorder, CWR, |
511 | Recovery and Loss. For details about these states, please refer page 5 |
512 | and page 6 of this document: |
513 | https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf |
514 | |
515 | .. _RFC2018: https://tools.ietf.org/html/rfc2018 |
516 | .. _RFC6582: https://tools.ietf.org/html/rfc6582 |
517 | |
518 | * TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery |
519 | When the congestion control comes into Recovery state, if sack is |
520 | used, TcpExtTCPSackRecovery increases 1, if sack is not used, |
521 | TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP |
522 | stack begins to retransmit the lost packets. |
523 | |
524 | * TcpExtTCPSACKReneging |
525 | A packet was acknowledged by SACK, but the receiver has dropped this |
526 | packet, so the sender needs to retransmit this packet. In this |
527 | situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver |
528 | could drop a packet which has been acknowledged by SACK, although it is |
529 | unusual, it is allowed by the TCP protocol. The sender doesn't really |
530 | know what happened on the receiver side. The sender just waits until |
531 | the RTO expires for this packet, then the sender assumes this packet |
532 | has been dropped by the receiver. |
533 | |
534 | * TcpExtTCPRenoReorder |
535 | The reorder packet is detected by fast recovery. It would only be used |
536 | if SACK is disabled. The fast recovery algorithm detects recorder by |
537 | the duplicate ACK number. E.g., if retransmission is triggered, and |
538 | the original retransmitted packet is not lost, it is just out of |
539 | order, the receiver would acknowledge multiple times, one for the |
540 | retransmitted packet, another for the arriving of the original out of |
541 | order packet. Thus the sender would find more ACks than its |
542 | expectation, and the sender knows out of order occurs. |
543 | |
544 | * TcpExtTCPTSReorder |
545 | The reorder packet is detected when a hole is filled. E.g., assume the |
546 | sender sends packet 1,2,3,4,5, and the receiving order is |
547 | 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will |
548 | fill the hole), two conditions will let TcpExtTCPTSReorder increase |
549 | 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet |
550 | 3 is retransmitted but the timestamp of the packet 3's ACK is earlier |
551 | than the retransmission timestamp. |
552 | |
553 | * TcpExtTCPSACKReorder |
554 | The reorder packet detected by SACK. The SACK has two methods to |
555 | detect reorder: (1) DSACK is received by the sender. It means the |
556 | sender sends the same packet more than one times. And the only reason |
557 | is the sender believes an out of order packet is lost so it sends the |
558 | packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and |
559 | the sender has received SACKs for packet 2 and 5, now the sender |
560 | receives SACK for packet 4 and the sender doesn't retransmit the |
561 | packet yet, the sender would know packet 4 is out of order. The TCP |
562 | stack of kernel will increase TcpExtTCPSACKReorder for both of the |
563 | above scenarios. |
564 | |
8e2ea53a |
565 | DSACK |
566 | ===== |
567 | The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report |
568 | duplicate packets to the sender. There are two kinds of |
569 | duplications: (1) a packet which has been acknowledged is |
570 | duplicate. (2) an out of order packet is duplicate. The TCP stack |
571 | counts these two kinds of duplications on both receiver side and |
572 | sender side. |
573 | |
574 | .. _RFC2883 : https://tools.ietf.org/html/rfc2883 |
575 | |
576 | * TcpExtTCPDSACKOldSent |
577 | The TCP stack receives a duplicate packet which has been acked, so it |
578 | sends a DSACK to the sender. |
579 | |
580 | * TcpExtTCPDSACKOfoSent |
581 | The TCP stack receives an out of order duplicate packet, so it sends a |
582 | DSACK to the sender. |
583 | |
584 | * TcpExtTCPDSACKRecv |
a6c7c7aa |
585 | The TCP stack receives a DSACK, which indicates an acknowledged |
8e2ea53a |
586 | duplicate packet is received. |
587 | |
588 | * TcpExtTCPDSACKOfoRecv |
589 | The TCP stack receives a DSACK, which indicate an out of order |
2b965472 |
590 | duplicate packet is received. |
591 | |
a6c7c7aa |
592 | invalid SACK and DSACK |
593 | ==================== |
594 | When a SACK (or DSACK) block is invalid, a corresponding counter would |
595 | be updated. The validation method is base on the start/end sequence |
596 | number of the SACK block. For more details, please refer the comment |
597 | of the function tcp_is_sackblock_valid in the kernel source code. A |
598 | SACK option could have up to 4 blocks, they are checked |
599 | individually. E.g., if 3 blocks of a SACk is invalid, the |
600 | corresponding counter would be updated 3 times. The comment of the |
601 | `Add counters for discarded SACK blocks`_ patch has additional |
602 | explaination: |
603 | |
604 | .. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32 |
605 | |
606 | * TcpExtTCPSACKDiscard |
607 | This counter indicates how many SACK blocks are invalid. If the invalid |
608 | SACK block is caused by ACK recording, the TCP stack will only ignore |
609 | it and won't update this counter. |
610 | |
611 | * TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo |
612 | When a DSACK block is invalid, one of these two counters would be |
613 | updated. Which counter will be updated depends on the undo_marker flag |
614 | of the TCP socket. If the undo_marker is not set, the TCP stack isn't |
615 | likely to re-transmit any packets, and we still receive an invalid |
616 | DSACK block, the reason might be that the packet is duplicated in the |
617 | middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo |
618 | will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld |
619 | will be updated. As implied in its name, it might be an old packet. |
620 | |
621 | SACK shift |
622 | ========= |
623 | The linux networking stack stores data in sk_buff struct (skb for |
624 | short). If a SACK block acrosses multiple skb, the TCP stack will try |
625 | to re-arrange data in these skb. E.g. if a SACK block acknowledges seq |
626 | 10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and |
627 | 15 in skb2 would be moved to skb1. This operation is 'shift'. If a |
628 | SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has |
629 | seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be |
630 | discard, this operation is 'merge'. |
631 | |
632 | * TcpExtTCPSackShifted |
633 | A skb is shifted |
634 | |
635 | * TcpExtTCPSackMerged |
636 | A skb is merged |
637 | |
638 | * TcpExtTCPSackShiftFallback |
639 | A skb should be shifted or merged, but the TCP stack doesn't do it for |
640 | some reasons. |
641 | |
2b965472 |
642 | TCP out of order |
643 | =============== |
644 | * TcpExtTCPOFOQueue |
645 | The TCP layer receives an out of order packet and has enough memory |
646 | to queue it. |
647 | |
648 | * TcpExtTCPOFODrop |
649 | The TCP layer receives an out of order packet but doesn't have enough |
650 | memory, so drops it. Such packets won't be counted into |
651 | TcpExtTCPOFOQueue. |
652 | |
653 | * TcpExtTCPOFOMerge |
654 | The received out of order packet has an overlay with the previous |
655 | packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge |
656 | packets will also be counted into TcpExtTCPOFOQueue. |
657 | |
658 | TCP PAWS |
659 | ======= |
660 | PAWS (Protection Against Wrapped Sequence numbers) is an algorithm |
661 | which is used to drop old packets. It depends on the TCP |
662 | timestamps. For detail information, please refer the `timestamp wiki`_ |
663 | and the `RFC of PAWS`_. |
664 | |
665 | .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17 |
666 | .. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps |
667 | |
668 | * TcpExtPAWSActive |
669 | Packets are dropped by PAWS in Syn-Sent status. |
670 | |
671 | * TcpExtPAWSEstab |
672 | Packets are dropped by PAWS in any status other than Syn-Sent. |
673 | |
674 | TCP ACK skip |
675 | =========== |
676 | In some scenarios, kernel would avoid sending duplicate ACKs too |
677 | frequently. Please find more details in the tcp_invalid_ratelimit |
678 | section of the `sysctl document`_. When kernel decides to skip an ACK |
679 | due to tcp_invalid_ratelimit, kernel would update one of below |
680 | counters to indicate the ACK is skipped in which scenario. The ACK |
681 | would only be skipped if the received packet is either a SYN packet or |
682 | it has no data. |
683 | |
684 | .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt |
685 | |
686 | * TcpExtTCPACKSkippedSynRecv |
687 | The ACK is skipped in Syn-Recv status. The Syn-Recv status means the |
688 | TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is |
689 | waiting for an ACK. Generally, the TCP stack doesn't need to send ACK |
690 | in the Syn-Recv status. But in several scenarios, the TCP stack need |
691 | to send an ACK. E.g., the TCP stack receives the same SYN packet |
692 | repeately, the received packet does not pass the PAWS check, or the |
693 | received packet sequence number is out of window. In these scenarios, |
694 | the TCP stack needs to send ACK. If the ACk sending frequency is higher than |
695 | tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and |
696 | increase TcpExtTCPACKSkippedSynRecv. |
697 | |
698 | |
699 | * TcpExtTCPACKSkippedPAWS |
700 | The ACK is skipped due to PAWS (Protect Against Wrapped Sequence |
701 | numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2 |
702 | or Time-Wait statuses, the skipped ACK would be counted to |
703 | TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or |
704 | TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK |
705 | would be counted to TcpExtTCPACKSkippedPAWS. |
706 | |
707 | * TcpExtTCPACKSkippedSeq |
708 | The sequence number is out of window and the timestamp passes the PAWS |
709 | check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait. |
710 | |
711 | * TcpExtTCPACKSkippedFinWait2 |
712 | The ACK is skipped in Fin-Wait-2 status, the reason would be either |
713 | PAWS check fails or the received sequence number is out of window. |
714 | |
715 | * TcpExtTCPACKSkippedTimeWait |
716 | Tha ACK is skipped in Time-Wait status, the reason would be either |
717 | PAWS check failed or the received sequence number is out of window. |
718 | |
719 | * TcpExtTCPACKSkippedChallenge |
720 | The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines |
721 | 3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_, |
722 | `RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these |
723 | three scenarios, In some TCP status, the linux TCP stack would also |
724 | send challenge ACKs if the ACK number is before the first |
725 | unacknowledged number (more strict than `RFC 5961 section 5.2`_). |
726 | |
727 | .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7 |
728 | .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9 |
729 | .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11 |
730 | |
a6c7c7aa |
731 | TCP receive window |
732 | ================= |
733 | * TcpExtTCPWantZeroWindowAdv |
734 | Depending on current memory usage, the TCP stack tries to set receive |
735 | window to zero. But the receive window might still be a no-zero |
736 | value. For example, if the previous window size is 10, and the TCP |
737 | stack receives 3 bytes, the current window size would be 7 even if the |
738 | window size calculated by the memory usage is zero. |
739 | |
740 | * TcpExtTCPToZeroWindowAdv |
741 | The TCP receive window is set to zero from a no-zero value. |
742 | |
743 | * TcpExtTCPFromZeroWindowAdv |
744 | The TCP receive window is set to no-zero value from zero. |
745 | |
746 | |
747 | Delayed ACK |
748 | ========== |
749 | The TCP Delayed ACK is a technique which is used for reducing the |
750 | packet count in the network. For more details, please refer the |
751 | `Delayed ACK wiki`_ |
752 | |
753 | .. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment |
754 | |
755 | * TcpExtDelayedACKs |
756 | A delayed ACK timer expires. The TCP stack will send a pure ACK packet |
757 | and exit the delayed ACK mode. |
758 | |
759 | * TcpExtDelayedACKLocked |
760 | A delayed ACK timer expires, but the TCP stack can't send an ACK |
761 | immediately due to the socket is locked by a userspace program. The |
762 | TCP stack will send a pure ACK later (after the userspace program |
763 | unlock the socket). When the TCP stack sends the pure ACK later, the |
764 | TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK |
765 | mode. |
766 | |
767 | * TcpExtDelayedACKLost |
768 | It will be updated when the TCP stack receives a packet which has been |
769 | ACKed. A Delayed ACK loss might cause this issue, but it would also be |
770 | triggered by other reasons, such as a packet is duplicated in the |
771 | network. |
772 | |
773 | Tail Loss Probe (TLP) |
774 | =================== |
775 | TLP is an algorithm which is used to detect TCP packet loss. For more |
776 | details, please refer the `TLP paper`_. |
777 | |
778 | .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01 |
779 | |
780 | * TcpExtTCPLossProbes |
781 | A TLP probe packet is sent. |
782 | |
783 | * TcpExtTCPLossProbeRecovery |
784 | A packet loss is detected and recovered by TLP. |
8e2ea53a |
785 | |
b08794a9 |
786 | examples |
787 | ======= |
788 | |
789 | ping test |
790 | -------- |
791 | Run the ping command against the public dns server 8.8.8.8:: |
792 | |
793 | nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1 |
794 | PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. |
795 | 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms |
796 | |
797 | --- 8.8.8.8 ping statistics --- |
798 | 1 packets transmitted, 1 received, 0% packet loss, time 0ms |
799 | rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms |
800 | |
801 | The nstayt result:: |
802 | |
803 | nstatuser@nstat-a:~$ nstat |
804 | #kernel |
805 | IpInReceives 1 0.0 |
806 | IpInDelivers 1 0.0 |
807 | IpOutRequests 1 0.0 |
808 | IcmpInMsgs 1 0.0 |
809 | IcmpInEchoReps 1 0.0 |
810 | IcmpOutMsgs 1 0.0 |
811 | IcmpOutEchos 1 0.0 |
812 | IcmpMsgInType0 1 0.0 |
813 | IcmpMsgOutType8 1 0.0 |
814 | IpExtInOctets 84 0.0 |
815 | IpExtOutOctets 84 0.0 |
816 | IpExtInNoECTPkts 1 0.0 |
817 | |
818 | The Linux server sent an ICMP Echo packet, so IpOutRequests, |
819 | IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The |
820 | server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs, |
821 | IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply |
822 | was passed to the ICMP layer via IP layer, so IpInDelivers was |
823 | increased 1. The default ping data size is 48, so an ICMP Echo packet |
824 | and its corresponding Echo Reply packet are constructed by: |
825 | |
826 | * 14 bytes MAC header |
827 | * 20 bytes IP header |
828 | * 16 bytes ICMP header |
829 | * 48 bytes data (default value of the ping command) |
830 | |
831 | So the IpExtInOctets and IpExtOutOctets are 20+16+48=84. |
80cc4950 |
832 | |
833 | tcp 3-way handshake |
834 | ------------------ |
835 | On server side, we run:: |
836 | |
837 | nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000 |
838 | Listening on [0.0.0.0] (family 0, port 9000) |
839 | |
840 | On client side, we run:: |
841 | |
842 | nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000 |
843 | Connection to 192.168.122.251 9000 port [tcp/*] succeeded! |
844 | |
845 | The server listened on tcp 9000 port, the client connected to it, they |
846 | completed the 3-way handshake. |
847 | |
848 | On server side, we can find below nstat output:: |
849 | |
850 | nstatuser@nstat-b:~$ nstat | grep -i tcp |
851 | TcpPassiveOpens 1 0.0 |
852 | TcpInSegs 2 0.0 |
853 | TcpOutSegs 1 0.0 |
854 | TcpExtTCPPureAcks 1 0.0 |
855 | |
856 | On client side, we can find below nstat output:: |
857 | |
858 | nstatuser@nstat-a:~$ nstat | grep -i tcp |
859 | TcpActiveOpens 1 0.0 |
860 | TcpInSegs 1 0.0 |
861 | TcpOutSegs 2 0.0 |
862 | |
863 | When the server received the first SYN, it replied a SYN+ACK, and came into |
864 | SYN-RCVD state, so TcpPassiveOpens increased 1. The server received |
865 | SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2 |
866 | packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK |
867 | of the 3-way handshake is a pure ACK without data, so |
868 | TcpExtTCPPureAcks increased 1. |
869 | |
870 | When the client sent SYN, the client came into the SYN-SENT state, so |
871 | TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent |
872 | ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased |
873 | 1, TcpOutSegs increased 2. |
874 | |
875 | TCP normal traffic |
876 | ----------------- |
877 | Run nc on server:: |
878 | |
879 | nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 |
880 | Listening on [0.0.0.0] (family 0, port 9000) |
881 | |
882 | Run nc on client:: |
883 | |
884 | nstatuser@nstat-a:~$ nc -v nstat-b 9000 |
885 | Connection to nstat-b 9000 port [tcp/*] succeeded! |
886 | |
887 | Input a string in the nc client ('hello' in our example):: |
888 | |
889 | nstatuser@nstat-a:~$ nc -v nstat-b 9000 |
890 | Connection to nstat-b 9000 port [tcp/*] succeeded! |
891 | hello |
892 | |
893 | The client side nstat output:: |
894 | |
895 | nstatuser@nstat-a:~$ nstat |
896 | #kernel |
897 | IpInReceives 1 0.0 |
898 | IpInDelivers 1 0.0 |
899 | IpOutRequests 1 0.0 |
900 | TcpInSegs 1 0.0 |
901 | TcpOutSegs 1 0.0 |
902 | TcpExtTCPPureAcks 1 0.0 |
903 | TcpExtTCPOrigDataSent 1 0.0 |
904 | IpExtInOctets 52 0.0 |
905 | IpExtOutOctets 58 0.0 |
906 | IpExtInNoECTPkts 1 0.0 |
907 | |
908 | The server side nstat output:: |
909 | |
910 | nstatuser@nstat-b:~$ nstat |
911 | #kernel |
912 | IpInReceives 1 0.0 |
913 | IpInDelivers 1 0.0 |
914 | IpOutRequests 1 0.0 |
915 | TcpInSegs 1 0.0 |
916 | TcpOutSegs 1 0.0 |
917 | IpExtInOctets 58 0.0 |
918 | IpExtOutOctets 52 0.0 |
919 | IpExtInNoECTPkts 1 0.0 |
920 | |
921 | Input a string in nc client side again ('world' in our exmaple):: |
922 | |
923 | nstatuser@nstat-a:~$ nc -v nstat-b 9000 |
924 | Connection to nstat-b 9000 port [tcp/*] succeeded! |
925 | hello |
926 | world |
927 | |
928 | Client side nstat output:: |
929 | |
930 | nstatuser@nstat-a:~$ nstat |
931 | #kernel |
932 | IpInReceives 1 0.0 |
933 | IpInDelivers 1 0.0 |
934 | IpOutRequests 1 0.0 |
935 | TcpInSegs 1 0.0 |
936 | TcpOutSegs 1 0.0 |
937 | TcpExtTCPHPAcks 1 0.0 |
938 | TcpExtTCPOrigDataSent 1 0.0 |
939 | IpExtInOctets 52 0.0 |
940 | IpExtOutOctets 58 0.0 |
941 | IpExtInNoECTPkts 1 0.0 |
942 | |
943 | |
944 | Server side nstat output:: |
945 | |
946 | nstatuser@nstat-b:~$ nstat |
947 | #kernel |
948 | IpInReceives 1 0.0 |
949 | IpInDelivers 1 0.0 |
950 | IpOutRequests 1 0.0 |
951 | TcpInSegs 1 0.0 |
952 | TcpOutSegs 1 0.0 |
953 | TcpExtTCPHPHits 1 0.0 |
954 | IpExtInOctets 58 0.0 |
955 | IpExtOutOctets 52 0.0 |
956 | IpExtInNoECTPkts 1 0.0 |
957 | |
958 | Compare the first client-side nstat and the second client-side nstat, |
959 | we could find one difference: the first one had a 'TcpExtTCPPureAcks', |
960 | but the second one had a 'TcpExtTCPHPAcks'. The first server-side |
961 | nstat and the second server-side nstat had a difference too: the |
962 | second server-side nstat had a TcpExtTCPHPHits, but the first |
963 | server-side nstat didn't have it. The network traffic patterns were |
964 | exactly the same: the client sent a packet to the server, the server |
965 | replied an ACK. But kernel handled them in different ways. When the |
966 | TCP window scale option is not used, kernel will try to enable fast |
967 | path immediately when the connection comes into the established state, |
968 | but if the TCP window scale option is used, kernel will disable the |
969 | fast path at first, and try to enable it after kerenl receives |
970 | packets. We could use the 'ss' command to verify whether the window |
971 | scale option is used. e.g. run below command on either server or |
972 | client:: |
973 | |
974 | nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 ) |
975 | Netid Recv-Q Send-Q Local Address:Port Peer Address:Port |
976 | tcp 0 0 192.168.122.250:40654 192.168.122.251:9000 |
977 | ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98 |
978 | |
979 | The 'wscale:7,7' means both server and client set the window scale |
980 | option to 7. Now we could explain the nstat output in our test: |
981 | |
982 | In the first nstat output of client side, the client sent a packet, server |
983 | reply an ACK, when kernel handled this ACK, the fast path was not |
984 | enabled, so the ACK was counted into 'TcpExtTCPPureAcks'. |
985 | |
986 | In the second nstat output of client side, the client sent a packet again, |
987 | and received another ACK from the server, in this time, the fast path is |
988 | enabled, and the ACK was qualified for fast path, so it was handled by |
989 | the fast path, so this ACK was counted into TcpExtTCPHPAcks. |
990 | |
991 | In the first nstat output of server side, fast path was not enabled, |
992 | so there was no 'TcpExtTCPHPHits'. |
993 | |
994 | In the second nstat output of server side, the fast path was enabled, |
995 | and the packet received from client qualified for fast path, so it |
996 | was counted into 'TcpExtTCPHPHits'. |
997 | |
998 | TcpExtTCPAbortOnClose |
999 | -------------------- |
1000 | On the server side, we run below python script:: |
1001 | |
1002 | import socket |
1003 | import time |
1004 | |
1005 | port = 9000 |
1006 | |
1007 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
1008 | s.bind(('0.0.0.0', port)) |
1009 | s.listen(1) |
1010 | sock, addr = s.accept() |
1011 | while True: |
1012 | time.sleep(9999999) |
1013 | |
1014 | This python script listen on 9000 port, but doesn't read anything from |
1015 | the connection. |
1016 | |
1017 | On the client side, we send the string "hello" by nc:: |
1018 | |
1019 | nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000 |
1020 | |
1021 | Then, we come back to the server side, the server has received the "hello" |
1022 | packet, and the TCP layer has acked this packet, but the application didn't |
1023 | read it yet. We type Ctrl-C to terminate the server script. Then we |
1024 | could find TcpExtTCPAbortOnClose increased 1 on the server side:: |
1025 | |
1026 | nstatuser@nstat-b:~$ nstat | grep -i abort |
1027 | TcpExtTCPAbortOnClose 1 0.0 |
1028 | |
1029 | If we run tcpdump on the server side, we could find the server sent a |
1030 | RST after we type Ctrl-C. |
1031 | |
1032 | TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout |
1033 | ----------------------------------------------- |
1034 | Below is an example which let the orphan socket count be higher than |
1035 | net.ipv4.tcp_max_orphans. |
1036 | Change tcp_max_orphans to a smaller value on client:: |
1037 | |
1038 | sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans" |
1039 | |
1040 | Client code (create 64 connection to server):: |
1041 | |
1042 | nstatuser@nstat-a:~$ cat client_orphan.py |
1043 | import socket |
1044 | import time |
1045 | |
1046 | server = 'nstat-b' # server address |
1047 | port = 9000 |
1048 | |
1049 | count = 64 |
1050 | |
1051 | connection_list = [] |
1052 | |
1053 | for i in range(64): |
1054 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
1055 | s.connect((server, port)) |
1056 | connection_list.append(s) |
1057 | print("connection_count: %d" % len(connection_list)) |
1058 | |
1059 | while True: |
1060 | time.sleep(99999) |
1061 | |
1062 | Server code (accept 64 connection from client):: |
1063 | |
1064 | nstatuser@nstat-b:~$ cat server_orphan.py |
1065 | import socket |
1066 | import time |
1067 | |
1068 | port = 9000 |
1069 | count = 64 |
1070 | |
1071 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
1072 | s.bind(('0.0.0.0', port)) |
1073 | s.listen(count) |
1074 | connection_list = [] |
1075 | while True: |
1076 | sock, addr = s.accept() |
1077 | connection_list.append((sock, addr)) |
1078 | print("connection_count: %d" % len(connection_list)) |
1079 | |
1080 | Run the python scripts on server and client. |
1081 | |
1082 | On server:: |
1083 | |
1084 | python3 server_orphan.py |
1085 | |
1086 | On client:: |
1087 | |
1088 | python3 client_orphan.py |
1089 | |
1090 | Run iptables on server:: |
1091 | |
1092 | sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP |
1093 | |
1094 | Type Ctrl-C on client, stop client_orphan.py. |
1095 | |
1096 | Check TcpExtTCPAbortOnMemory on client:: |
1097 | |
1098 | nstatuser@nstat-a:~$ nstat | grep -i abort |
1099 | TcpExtTCPAbortOnMemory 54 0.0 |
1100 | |
1101 | Check orphane socket count on client:: |
1102 | |
1103 | nstatuser@nstat-a:~$ ss -s |
1104 | Total: 131 (kernel 0) |
1105 | TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0 |
1106 | |
1107 | Transport Total IP IPv6 |
1108 | * 0 - - |
1109 | RAW 1 0 1 |
1110 | UDP 1 1 0 |
1111 | TCP 14 13 1 |
1112 | INET 16 14 2 |
1113 | FRAG 0 0 0 |
1114 | |
1115 | The explanation of the test: after run server_orphan.py and |
1116 | client_orphan.py, we set up 64 connections between server and |
1117 | client. Run the iptables command, the server will drop all packets from |
1118 | the client, type Ctrl-C on client_orphan.py, the system of the client |
1119 | would try to close these connections, and before they are closed |
1120 | gracefully, these connections became orphan sockets. As the iptables |
1121 | of the server blocked packets from the client, the server won't receive fin |
1122 | from the client, so all connection on clients would be stuck on FIN_WAIT_1 |
1123 | stage, so they will keep as orphan sockets until timeout. We have echo |
1124 | 10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would |
1125 | only keep 10 orphan sockets, for all other orphan sockets, the client |
1126 | system sent RST for them and delete them. We have 64 connections, so |
1127 | the 'ss -s' command shows the system has 10 orphan sockets, and the |
1128 | value of TcpExtTCPAbortOnMemory was 54. |
1129 | |
1130 | An additional explanation about orphan socket count: You could find the |
1131 | exactly orphan socket count by the 'ss -s' command, but when kernel |
1132 | decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel |
1133 | doesn't always check the exactly orphan socket count. For increasing |
1134 | performance, kernel checks an approximate count firstly, if the |
1135 | approximate count is more than tcp_max_orphans, kernel checks the |
1136 | exact count again. So if the approximate count is less than |
1137 | tcp_max_orphans, but exactly count is more than tcp_max_orphans, you |
1138 | would find TcpExtTCPAbortOnMemory is not increased at all. If |
1139 | tcp_max_orphans is large enough, it won't occur, but if you decrease |
1140 | tcp_max_orphans to a small value like our test, you might find this |
1141 | issue. So in our test, the client set up 64 connections although the |
1142 | tcp_max_orphans is 10. If the client only set up 11 connections, we |
1143 | can't find the change of TcpExtTCPAbortOnMemory. |
1144 | |
1145 | Continue the previous test, we wait for several minutes. Because of the |
1146 | iptables on the server blocked the traffic, the server wouldn't receive |
1147 | fin, and all the client's orphan sockets would timeout on the |
1148 | FIN_WAIT_1 state finally. So we wait for a few minutes, we could find |
1149 | 10 timeout on the client:: |
1150 | |
1151 | nstatuser@nstat-a:~$ nstat | grep -i abort |
1152 | TcpExtTCPAbortOnTimeout 10 0.0 |
1153 | |
1154 | TcpExtTCPAbortOnLinger |
1155 | --------------------- |
1156 | The server side code:: |
1157 | |
1158 | nstatuser@nstat-b:~$ cat server_linger.py |
1159 | import socket |
1160 | import time |
1161 | |
1162 | port = 9000 |
1163 | |
1164 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
1165 | s.bind(('0.0.0.0', port)) |
1166 | s.listen(1) |
1167 | sock, addr = s.accept() |
1168 | while True: |
1169 | time.sleep(9999999) |
1170 | |
1171 | The client side code:: |
1172 | |
1173 | nstatuser@nstat-a:~$ cat client_linger.py |
1174 | import socket |
1175 | import struct |
1176 | |
1177 | server = 'nstat-b' # server address |
1178 | port = 9000 |
1179 | |
1180 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
1181 | s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10)) |
1182 | s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1)) |
1183 | s.connect((server, port)) |
1184 | s.close() |
1185 | |
1186 | Run server_linger.py on server:: |
1187 | |
1188 | nstatuser@nstat-b:~$ python3 server_linger.py |
1189 | |
1190 | Run client_linger.py on client:: |
1191 | |
1192 | nstatuser@nstat-a:~$ python3 client_linger.py |
1193 | |
1194 | After run client_linger.py, check the output of nstat:: |
1195 | |
1196 | nstatuser@nstat-a:~$ nstat | grep -i abort |
1197 | TcpExtTCPAbortOnLinger 1 0.0 |
712ee16c |
1198 | |
1199 | TcpExtTCPRcvCoalesce |
1200 | ------------------- |
1201 | On the server, we run a program which listen on TCP port 9000, but |
1202 | doesn't read any data:: |
1203 | |
1204 | import socket |
1205 | import time |
1206 | port = 9000 |
1207 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
1208 | s.bind(('0.0.0.0', port)) |
1209 | s.listen(1) |
1210 | sock, addr = s.accept() |
1211 | while True: |
1212 | time.sleep(9999999) |
1213 | |
1214 | Save the above code as server_coalesce.py, and run:: |
1215 | |
1216 | python3 server_coalesce.py |
1217 | |
1218 | On the client, save below code as client_coalesce.py:: |
1219 | |
1220 | import socket |
1221 | server = 'nstat-b' |
1222 | port = 9000 |
1223 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
1224 | s.connect((server, port)) |
1225 | |
1226 | Run:: |
1227 | |
1228 | nstatuser@nstat-a:~$ python3 -i client_coalesce.py |
1229 | |
1230 | We use '-i' to come into the interactive mode, then a packet:: |
1231 | |
1232 | >>> s.send(b'foo') |
1233 | 3 |
1234 | |
1235 | Send a packet again:: |
1236 | |
1237 | >>> s.send(b'bar') |
1238 | 3 |
1239 | |
1240 | On the server, run nstat:: |
1241 | |
1242 | ubuntu@nstat-b:~$ nstat |
1243 | #kernel |
1244 | IpInReceives 2 0.0 |
1245 | IpInDelivers 2 0.0 |
1246 | IpOutRequests 2 0.0 |
1247 | TcpInSegs 2 0.0 |
1248 | TcpOutSegs 2 0.0 |
1249 | TcpExtTCPRcvCoalesce 1 0.0 |
1250 | IpExtInOctets 110 0.0 |
1251 | IpExtOutOctets 104 0.0 |
1252 | IpExtInNoECTPkts 2 0.0 |
1253 | |
1254 | The client sent two packets, server didn't read any data. When |
1255 | the second packet arrived at server, the first packet was still in |
1256 | the receiving queue. So the TCP layer merged the two packets, and we |
1257 | could find the TcpExtTCPRcvCoalesce increased 1. |
1258 | |
1259 | TcpExtListenOverflows and TcpExtListenDrops |
1260 | ---------------------------------------- |
1261 | On server, run the nc command, listen on port 9000:: |
1262 | |
1263 | nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 |
1264 | Listening on [0.0.0.0] (family 0, port 9000) |
1265 | |
1266 | On client, run 3 nc commands in different terminals:: |
1267 | |
1268 | nstatuser@nstat-a:~$ nc -v nstat-b 9000 |
1269 | Connection to nstat-b 9000 port [tcp/*] succeeded! |
1270 | |
1271 | The nc command only accepts 1 connection, and the accept queue length |
1272 | is 1. On current linux implementation, set queue length to n means the |
1273 | actual queue length is n+1. Now we create 3 connections, 1 is accepted |
1274 | by nc, 2 in accepted queue, so the accept queue is full. |
1275 | |
1276 | Before running the 4th nc, we clean the nstat history on the server:: |
1277 | |
1278 | nstatuser@nstat-b:~$ nstat -n |
1279 | |
1280 | Run the 4th nc on the client:: |
1281 | |
1282 | nstatuser@nstat-a:~$ nc -v nstat-b 9000 |
1283 | |
1284 | If the nc server is running on kernel 4.10 or higher version, you |
1285 | won't see the "Connection to ... succeeded!" string, because kernel |
1286 | will drop the SYN if the accept queue is full. If the nc client is running |
1287 | on an old kernel, you would see that the connection is succeeded, |
1288 | because kernel would complete the 3 way handshake and keep the socket |
1289 | on half open queue. I did the test on kernel 4.15. Below is the nstat |
1290 | on the server:: |
1291 | |
1292 | nstatuser@nstat-b:~$ nstat |
1293 | #kernel |
1294 | IpInReceives 4 0.0 |
1295 | IpInDelivers 4 0.0 |
1296 | TcpInSegs 4 0.0 |
1297 | TcpExtListenOverflows 4 0.0 |
1298 | TcpExtListenDrops 4 0.0 |
1299 | IpExtInOctets 240 0.0 |
1300 | IpExtInNoECTPkts 4 0.0 |
1301 | |
1302 | Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time |
1303 | between the 4th nc and the nstat was longer, the value of |
1304 | TcpExtListenOverflows and TcpExtListenDrops would be larger, because |
1305 | the SYN of the 4th nc was dropped, the client was retrying. |
8e2ea53a |
1306 | |
1307 | IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes |
1308 | ---------------------------------------------- |
1309 | server A IP address: 192.168.122.250 |
1310 | server B IP address: 192.168.122.251 |
1311 | Prepare on server A, add a route to server B:: |
1312 | |
1313 | $ sudo ip route add 8.8.8.8/32 via 192.168.122.251 |
1314 | |
1315 | Prepare on server B, disable send_redirects for all interfaces:: |
1316 | |
1317 | $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0 |
1318 | $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0 |
1319 | $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0 |
1320 | $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0 |
1321 | |
1322 | We want to let sever A send a packet to 8.8.8.8, and route the packet |
1323 | to server B. When server B receives such packet, it might send a ICMP |
1324 | Redirect message to server A, set send_redirects to 0 will disable |
1325 | this behavior. |
1326 | |
1327 | First, generate InAddrErrors. On server B, we disable IP forwarding:: |
1328 | |
1329 | $ sudo sysctl -w net.ipv4.conf.all.forwarding=0 |
1330 | |
1331 | On server A, we send packets to 8.8.8.8:: |
1332 | |
1333 | $ nc -v 8.8.8.8 53 |
1334 | |
1335 | On server B, we check the output of nstat:: |
1336 | |
1337 | $ nstat |
1338 | #kernel |
1339 | IpInReceives 3 0.0 |
1340 | IpInAddrErrors 3 0.0 |
1341 | IpExtInOctets 180 0.0 |
1342 | IpExtInNoECTPkts 3 0.0 |
1343 | |
1344 | As we have let server A route 8.8.8.8 to server B, and we disabled IP |
1345 | forwarding on server B, Server A sent packets to server B, then server B |
1346 | dropped packets and increased IpInAddrErrors. As the nc command would |
1347 | re-send the SYN packet if it didn't receive a SYN+ACK, we could find |
1348 | multiple IpInAddrErrors. |
1349 | |
1350 | Second, generate IpExtInNoRoutes. On server B, we enable IP |
1351 | forwarding:: |
1352 | |
1353 | $ sudo sysctl -w net.ipv4.conf.all.forwarding=1 |
1354 | |
1355 | Check the route table of server B and remove the default route:: |
1356 | |
1357 | $ ip route show |
1358 | default via 192.168.122.1 dev ens3 proto static |
1359 | 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251 |
1360 | $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static |
1361 | |
1362 | On server A, we contact 8.8.8.8 again:: |
1363 | |
1364 | $ nc -v 8.8.8.8 53 |
1365 | nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable |
1366 | |
1367 | On server B, run nstat:: |
1368 | |
1369 | $ nstat |
1370 | #kernel |
1371 | IpInReceives 1 0.0 |
1372 | IpOutRequests 1 0.0 |
1373 | IcmpOutMsgs 1 0.0 |
1374 | IcmpOutDestUnreachs 1 0.0 |
1375 | IcmpMsgOutType3 1 0.0 |
1376 | IpExtInNoRoutes 1 0.0 |
1377 | IpExtInOctets 60 0.0 |
1378 | IpExtOutOctets 88 0.0 |
1379 | IpExtInNoECTPkts 1 0.0 |
1380 | |
1381 | We enabled IP forwarding on server B, when server B received a packet |
1382 | which destination IP address is 8.8.8.8, server B will try to forward |
1383 | this packet. We have deleted the default route, there was no route for |
1384 | 8.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP |
1385 | Destination Unreachable" message to server A. |
1386 | |
1387 | Third, generate IpOutNoRoutes. Run ping command on server B:: |
1388 | |
1389 | $ ping -c 1 8.8.8.8 |
1390 | connect: Network is unreachable |
1391 | |
1392 | Run nstat on server B:: |
1393 | |
1394 | $ nstat |
1395 | #kernel |
1396 | IpOutNoRoutes 1 0.0 |
1397 | |
1398 | We have deleted the default route on server B. Server B couldn't find |
1399 | a route for the 8.8.8.8 IP address, so server B increased |
1400 | IpOutNoRoutes. |
2b965472 |
1401 | |
1402 | TcpExtTCPACKSkippedSynRecv |
1403 | ------------------------ |
1404 | In this test, we send 3 same SYN packets from client to server. The |
1405 | first SYN will let server create a socket, set it to Syn-Recv status, |
1406 | and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK |
1407 | again, and record the reply time (the duplicate ACK reply time). The |
1408 | third SYN will let server check the previous duplicate ACK reply time, |
1409 | and decide to skip the duplicate ACK, then increase the |
1410 | TcpExtTCPACKSkippedSynRecv counter. |
1411 | |
1412 | Run tcpdump to capture a SYN packet:: |
1413 | |
1414 | nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000 |
1415 | tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes |
1416 | |
1417 | Open another terminal, run nc command:: |
1418 | |
1419 | nstatuser@nstat-a:~$ nc nstat-b 9000 |
1420 | |
1421 | As the nstat-b didn't listen on port 9000, it should reply a RST, and |
1422 | the nc command exited immediately. It was enough for the tcpdump |
1423 | command to capture a SYN packet. A linux server might use hardware |
1424 | offload for the TCP checksum, so the checksum in the /tmp/syn.pcap |
1425 | might be not correct. We call tcprewrite to fix it:: |
1426 | |
1427 | nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum |
1428 | |
1429 | On nstat-b, we run nc to listen on port 9000:: |
1430 | |
1431 | nstatuser@nstat-b:~$ nc -lkv 9000 |
1432 | Listening on [0.0.0.0] (family 0, port 9000) |
1433 | |
1434 | On nstat-a, we blocked the packet from port 9000, or nstat-a would send |
1435 | RST to nstat-b:: |
1436 | |
1437 | nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP |
1438 | |
1439 | Send 3 SYN repeatly to nstat-b:: |
1440 | |
1441 | nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done |
1442 | |
1443 | Check snmp cunter on nstat-b:: |
1444 | |
1445 | nstatuser@nstat-b:~$ nstat | grep -i skip |
1446 | TcpExtTCPACKSkippedSynRecv 1 0.0 |
1447 | |
1448 | As we expected, TcpExtTCPACKSkippedSynRecv is 1. |
1449 | |
1450 | TcpExtTCPACKSkippedPAWS |
1451 | ---------------------- |
1452 | To trigger PAWS, we could send an old SYN. |
1453 | |
1454 | On nstat-b, let nc listen on port 9000:: |
1455 | |
1456 | nstatuser@nstat-b:~$ nc -lkv 9000 |
1457 | Listening on [0.0.0.0] (family 0, port 9000) |
1458 | |
1459 | On nstat-a, run tcpdump to capture a SYN:: |
1460 | |
1461 | nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000 |
1462 | tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes |
1463 | |
1464 | On nstat-a, run nc as a client to connect nstat-b:: |
1465 | |
1466 | nstatuser@nstat-a:~$ nc -v nstat-b 9000 |
1467 | Connection to nstat-b 9000 port [tcp/*] succeeded! |
1468 | |
1469 | Now the tcpdump has captured the SYN and exit. We should fix the |
1470 | checksum:: |
1471 | |
1472 | nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum |
1473 | |
1474 | Send the SYN packet twice:: |
1475 | |
1476 | nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done |
1477 | |
1478 | On nstat-b, check the snmp counter:: |
1479 | |
1480 | nstatuser@nstat-b:~$ nstat | grep -i skip |
1481 | TcpExtTCPACKSkippedPAWS 1 0.0 |
1482 | |
1483 | We sent two SYN via tcpreplay, both of them would let PAWS check |
1484 | failed, the nstat-b replied an ACK for the first SYN, skipped the ACK |
1485 | for the second SYN, and updated TcpExtTCPACKSkippedPAWS. |
1486 | |
1487 | TcpExtTCPACKSkippedSeq |
1488 | -------------------- |
1489 | To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid |
1490 | timestamp (to pass PAWS check) but the sequence number is out of |
1491 | window. The linux TCP stack would avoid to skip if the packet has |
1492 | data, so we need a pure ACK packet. To generate such a packet, we |
1493 | could create two sockets: one on port 9000, another on port 9001. Then |
1494 | we capture an ACK on port 9001, change the source/destination port |
1495 | numbers to match the port 9000 socket. Then we could trigger |
1496 | TcpExtTCPACKSkippedSeq via this packet. |
1497 | |
1498 | On nstat-b, open two terminals, run two nc commands to listen on both |
1499 | port 9000 and port 9001:: |
1500 | |
1501 | nstatuser@nstat-b:~$ nc -lkv 9000 |
1502 | Listening on [0.0.0.0] (family 0, port 9000) |
1503 | |
1504 | nstatuser@nstat-b:~$ nc -lkv 9001 |
1505 | Listening on [0.0.0.0] (family 0, port 9001) |
1506 | |
1507 | On nstat-a, run two nc clients:: |
1508 | |
1509 | nstatuser@nstat-a:~$ nc -v nstat-b 9000 |
1510 | Connection to nstat-b 9000 port [tcp/*] succeeded! |
1511 | |
1512 | nstatuser@nstat-a:~$ nc -v nstat-b 9001 |
1513 | Connection to nstat-b 9001 port [tcp/*] succeeded! |
1514 | |
1515 | On nstat-a, run tcpdump to capture an ACK:: |
1516 | |
1517 | nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001 |
1518 | tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes |
1519 | |
1520 | On nstat-b, send a packet via the port 9001 socket. E.g. we sent a |
1521 | string 'foo' in our example:: |
1522 | |
1523 | nstatuser@nstat-b:~$ nc -lkv 9001 |
1524 | Listening on [0.0.0.0] (family 0, port 9001) |
1525 | Connection from nstat-a 42132 received! |
1526 | foo |
1527 | |
1528 | On nstat-a, the tcpdump should have caputred the ACK. We should check |
1529 | the source port numbers of the two nc clients:: |
1530 | |
1531 | nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee |
1532 | State Recv-Q Send-Q Local Address:Port Peer Address:Port |
1533 | ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000 |
1534 | ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001 |
1535 | |
1536 | Run tcprewrite, change port 9001 to port 9000, chagne port 42132 to |
1537 | port 50208:: |
1538 | |
1539 | nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum |
1540 | |
1541 | Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b:: |
1542 | |
1543 | nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done |
1544 | |
1545 | Check TcpExtTCPACKSkippedSeq on nstat-b:: |
1546 | |
1547 | nstatuser@nstat-b:~$ nstat | grep -i skip |
1548 | TcpExtTCPACKSkippedSeq 1 0.0 |