1============ 2SNMP counter 3============ 4 5This document explains the meaning of SNMP counters. 6 7General IPv4 counters 8===================== 9All layer 4 packets and ICMP packets will change these counters, but 10these counters won't be changed by layer 2 packets (such as STP) or 11ARP packets. 12 13* IpInReceives 14 15Defined in `RFC1213 ipInReceives`_ 16 17.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26 18 19The number of packets received by the IP layer. It gets increasing at the 20beginning of ip_rcv function, always be updated together with 21IpExtInOctets. It will be increased even if the packet is dropped 22later (e.g. due to the IP header is invalid or the checksum is wrong 23and so on). It indicates the number of aggregated segments after 24GRO/LRO. 25 26* IpInDelivers 27 28Defined in `RFC1213 ipInDelivers`_ 29 30.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28 31 32The number of packets delivers to the upper layer protocols. E.g. TCP, UDP, 33ICMP and so on. If no one listens on a raw socket, only kernel 34supported protocols will be delivered, if someone listens on the raw 35socket, all valid IP packets will be delivered. 36 37* IpOutRequests 38 39Defined in `RFC1213 ipOutRequests`_ 40 41.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28 42 43The number of packets sent via IP layer, for both single cast and 44multicast packets, and would always be updated together with 45IpExtOutOctets. 46 47* IpExtInOctets and IpExtOutOctets 48 49They are Linux kernel extensions, no RFC definitions. Please note, 50RFC1213 indeed defines ifInOctets and ifOutOctets, but they 51are different things. The ifInOctets and ifOutOctets include the MAC 52layer header size but IpExtInOctets and IpExtOutOctets don't, they 53only include the IP layer header and the IP layer data. 54 55* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts 56 57They indicate the number of four kinds of ECN IP packets, please refer 58`Explicit Congestion Notification`_ for more details. 59 60.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6 61 62These 4 counters calculate how many packets received per ECN 63status. They count the real frame number regardless the LRO/GRO. So 64for the same packet, you might find that IpInReceives count 1, but 65IpExtInNoECTPkts counts 2 or more. 66 67* IpInHdrErrors 68 69Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is 70dropped due to the IP header error. It might happen in both IP input 71and IP forward paths. 72 73.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27 74 75* IpInAddrErrors 76 77Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two 78scenarios: (1) The IP address is invalid. (2) The destination IP 79address is not a local address and IP forwarding is not enabled 80 81.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27 82 83* IpExtInNoRoutes 84 85This counter means the packet is dropped when the IP stack receives a 86packet and can't find a route for it from the route table. It might 87happen when IP forwarding is enabled and the destination IP address is 88not a local address and there is no route for the destination IP 89address. 90 91* IpInUnknownProtos 92 93Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the 94layer 4 protocol is unsupported by kernel. If an application is using 95raw socket, kernel will always deliver the packet to the raw socket 96and this counter won't be increased. 97 98.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27 99 100* IpExtInTruncatedPkts 101 102For IPv4 packet, it means the actual data size is smaller than the 103"Total Length" field in the IPv4 header. 104 105* IpInDiscards 106 107Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped 108in the IP receiving path and due to kernel internal reasons (e.g. no 109enough memory). 110 111.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28 112 113* IpOutDiscards 114 115Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is 116dropped in the IP sending path and due to kernel internal reasons. 117 118.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28 119 120* IpOutNoRoutes 121 122Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is 123dropped in the IP sending path and no route is found for it. 124 125.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29 126 127ICMP counters 128============= 129* IcmpInMsgs and IcmpOutMsgs 130 131Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_ 132 133.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41 134.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43 135 136As mentioned in the RFC1213, these two counters include errors, they 137would be increased even if the ICMP packet has an invalid type. The 138ICMP output path will check the header of a raw socket, so the 139IcmpOutMsgs would still be updated if the IP header is constructed by 140a userspace program. 141 142* ICMP named types 143 144| These counters include most of common ICMP types, they are: 145| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_ 146| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_ 147| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_ 148| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_ 149| IcmpInRedirects: `RFC1213 icmpInRedirects`_ 150| IcmpInEchos: `RFC1213 icmpInEchos`_ 151| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_ 152| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_ 153| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_ 154| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_ 155| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_ 156| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_ 157| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_ 158| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_ 159| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_ 160| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_ 161| IcmpOutEchos: `RFC1213 icmpOutEchos`_ 162| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_ 163| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_ 164| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_ 165| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_ 166| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_ 167 168.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41 169.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41 170.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42 171.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42 172.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42 173.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42 174.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42 175.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42 176.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43 177.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43 178.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43 179 180.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44 181.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44 182.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44 183.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44 184.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44 185.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45 186.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45 187.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45 188.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45 189.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45 190.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46 191 192Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP 193Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are 194straightforward. The 'In' counter means kernel receives such a packet 195and the 'Out' counter means kernel sends such a packet. 196 197* ICMP numeric types 198 199They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the 200ICMP type number. These counters track all kinds of ICMP packets. The 201ICMP type number definition could be found in the `ICMP parameters`_ 202document. 203 204.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml 205 206For example, if the Linux kernel sends an ICMP Echo packet, the 207IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply 208packet, IcmpMsgInType0 would increase 1. 209 210* IcmpInCsumErrors 211 212This counter indicates the checksum of the ICMP packet is 213wrong. Kernel verifies the checksum after updating the IcmpInMsgs and 214before updating IcmpMsgInType[N]. If a packet has bad checksum, the 215IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated. 216 217* IcmpInErrors and IcmpOutErrors 218 219Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_ 220 221.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41 222.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43 223 224When an error occurs in the ICMP packet handler path, these two 225counters would be updated. The receiving packet path use IcmpInErrors 226and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors 227is increased, IcmpInErrors would always be increased too. 228 229relationship of the ICMP counters 230--------------------------------- 231The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they 232are updated at the same time. The sum of IcmpMsgInType[N] plus 233IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel 234receives an ICMP packet, kernel follows below logic: 235 2361. increase IcmpInMsgs 2372. if has any error, update IcmpInErrors and finish the process 2383. update IcmpMsgOutType[N] 2394. handle the packet depending on the type, if has any error, update 240 IcmpInErrors and finish the process 241 242So if all errors occur in step (2), IcmpInMsgs should be equal to the 243sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in 244step (4), IcmpInMsgs should be equal to the sum of 245IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4), 246IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus 247IcmpInErrors. 248 249General TCP counters 250==================== 251* TcpInSegs 252 253Defined in `RFC1213 tcpInSegs`_ 254 255.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48 256 257The number of packets received by the TCP layer. As mentioned in 258RFC1213, it includes the packets received in error, such as checksum 259error, invalid TCP header and so on. Only one error won't be included: 260if the layer 2 destination address is not the NIC's layer 2 261address. It might happen if the packet is a multicast or broadcast 262packet, or the NIC is in promiscuous mode. In these situations, the 263packets would be delivered to the TCP layer, but the TCP layer will discard 264these packets before increasing TcpInSegs. The TcpInSegs counter 265isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs 266counter would only increase 1. 267 268* TcpOutSegs 269 270Defined in `RFC1213 tcpOutSegs`_ 271 272.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48 273 274The number of packets sent by the TCP layer. As mentioned in RFC1213, 275it excludes the retransmitted packets. But it includes the SYN, ACK 276and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of 277GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will 278increase 2. 279 280* TcpActiveOpens 281 282Defined in `RFC1213 tcpActiveOpens`_ 283 284.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47 285 286It means the TCP layer sends a SYN, and come into the SYN-SENT 287state. Every time TcpActiveOpens increases 1, TcpOutSegs should always 288increase 1. 289 290* TcpPassiveOpens 291 292Defined in `RFC1213 tcpPassiveOpens`_ 293 294.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47 295 296It means the TCP layer receives a SYN, replies a SYN+ACK, come into 297the SYN-RCVD state. 298 299* TcpExtTCPRcvCoalesce 300 301When packets are received by the TCP layer and are not be read by the 302application, the TCP layer will try to merge them. This counter 303indicate how many packets are merged in such situation. If GRO is 304enabled, lots of packets would be merged by GRO, these packets 305wouldn't be counted to TcpExtTCPRcvCoalesce. 306 307* TcpExtTCPAutoCorking 308 309When sending packets, the TCP layer will try to merge small packets to 310a bigger one. This counter increase 1 for every packet merged in such 311situation. Please refer to the LWN article for more details: 312https://lwn.net/Articles/576263/ 313 314* TcpExtTCPOrigDataSent 315 316This counter is explained by kernel commit f19c29e3e391, I pasted the 317explanation below:: 318 319 TCPOrigDataSent: number of outgoing packets with original data (excluding 320 retransmission but including data-in-SYN). This counter is different from 321 TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is 322 more useful to track the TCP retransmission rate. 323 324* TCPSynRetrans 325 326This counter is explained by kernel commit f19c29e3e391, I pasted the 327explanation below:: 328 329 TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down 330 retransmissions into SYN, fast-retransmits, timeout retransmits, etc. 331 332* TCPFastOpenActiveFail 333 334This counter is explained by kernel commit f19c29e3e391, I pasted the 335explanation below:: 336 337 TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because 338 the remote does not accept it or the attempts timed out. 339 340* TcpExtListenOverflows and TcpExtListenDrops 341 342When kernel receives a SYN from a client, and if the TCP accept queue 343is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows. 344At the same time kernel will also add 1 to TcpExtListenDrops. When a 345TCP socket is in LISTEN state, and kernel need to drop a packet, 346kernel would always add 1 to TcpExtListenDrops. So increase 347TcpExtListenOverflows would let TcpExtListenDrops increasing at the 348same time, but TcpExtListenDrops would also increase without 349TcpExtListenOverflows increasing, e.g. a memory allocation fail would 350also let TcpExtListenDrops increase. 351 352Note: The above explanation is based on kernel 4.10 or above version, on 353an old kernel, the TCP stack has different behavior when TCP accept 354queue is full. On the old kernel, TCP stack won't drop the SYN, it 355would complete the 3-way handshake. As the accept queue is full, TCP 356stack will keep the socket in the TCP half-open queue. As it is in the 357half open queue, TCP stack will send SYN+ACK on an exponential backoff 358timer, after client replies ACK, TCP stack checks whether the accept 359queue is still full, if it is not full, moves the socket to the accept 360queue, if it is full, keeps the socket in the half-open queue, at next 361time client replies ACK, this socket will get another chance to move 362to the accept queue. 363 364 365TCP Fast Open 366============= 367* TcpEstabResets 368 369Defined in `RFC1213 tcpEstabResets`_. 370 371.. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48 372 373* TcpAttemptFails 374 375Defined in `RFC1213 tcpAttemptFails`_. 376 377.. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48 378 379* TcpOutRsts 380 381Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates 382the 'segments sent containing the RST flag', but in linux kernel, this 383counter indicates the segments kernel tried to send. The sending 384process might be failed due to some errors (e.g. memory alloc failed). 385 386.. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52 387 388* TcpExtTCPSpuriousRtxHostQueues 389 390When the TCP stack wants to retransmit a packet, and finds that packet 391is not lost in the network, but the packet is not sent yet, the TCP 392stack would give up the retransmission and update this counter. It 393might happen if a packet stays too long time in a qdisc or driver 394queue. 395 396* TcpEstabResets 397 398The socket receives a RST packet in Establish or CloseWait state. 399 400* TcpExtTCPKeepAlive 401 402This counter indicates many keepalive packets were sent. The keepalive 403won't be enabled by default. A userspace program could enable it by 404setting the SO_KEEPALIVE socket option. 405 406* TcpExtTCPSpuriousRTOs 407 408The spurious retransmission timeout detected by the `F-RTO`_ 409algorithm. 410 411.. _F-RTO: https://tools.ietf.org/html/rfc5682 412 413TCP Fast Path 414============= 415When kernel receives a TCP packet, it has two paths to handler the 416packet, one is fast path, another is slow path. The comment in kernel 417code provides a good explanation of them, I pasted them below:: 418 419 It is split into a fast path and a slow path. The fast path is 420 disabled when: 421 422 - A zero window was announced from us 423 - zero window probing 424 is only handled properly on the slow path. 425 - Out of order segments arrived. 426 - Urgent data is expected. 427 - There is no buffer space left 428 - Unexpected TCP flags/window values/header lengths are received 429 (detected by checking the TCP header against pred_flags) 430 - Data is sent in both directions. The fast path only supports pure senders 431 or pure receivers (this means either the sequence number or the ack 432 value must stay constant) 433 - Unexpected TCP option. 434 435Kernel will try to use fast path unless any of the above conditions 436are satisfied. If the packets are out of order, kernel will handle 437them in slow path, which means the performance might be not very 438good. Kernel would also come into slow path if the "Delayed ack" is 439used, because when using "Delayed ack", the data is sent in both 440directions. When the TCP window scale option is not used, kernel will 441try to enable fast path immediately when the connection comes into the 442established state, but if the TCP window scale option is used, kernel 443will disable the fast path at first, and try to enable it after kernel 444receives packets. 445 446* TcpExtTCPPureAcks and TcpExtTCPHPAcks 447 448If a packet set ACK flag and has no data, it is a pure ACK packet, if 449kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1, 450if kernel handles it in the slow path, TcpExtTCPPureAcks will 451increase 1. 452 453* TcpExtTCPHPHits 454 455If a TCP packet has data (which means it is not a pure ACK packet), 456and this packet is handled in the fast path, TcpExtTCPHPHits will 457increase 1. 458 459 460TCP abort 461========= 462* TcpExtTCPAbortOnData 463 464It means TCP layer has data in flight, but need to close the 465connection. So TCP layer sends a RST to the other side, indicate the 466connection is not closed very graceful. An easy way to increase this 467counter is using the SO_LINGER option. Please refer to the SO_LINGER 468section of the `socket man page`_: 469 470.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html 471 472By default, when an application closes a connection, the close function 473will return immediately and kernel will try to send the in-flight data 474async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger 475to a positive number, the close function won't return immediately, but 476wait for the in-flight data are acked by the other side, the max wait 477time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0, 478when the application closes a connection, kernel will send a RST 479immediately and increase the TcpExtTCPAbortOnData counter. 480 481* TcpExtTCPAbortOnClose 482 483This counter means the application has unread data in the TCP layer when 484the application wants to close the TCP connection. In such a situation, 485kernel will send a RST to the other side of the TCP connection. 486 487* TcpExtTCPAbortOnMemory 488 489When an application closes a TCP connection, kernel still need to track 490the connection, let it complete the TCP disconnect process. E.g. an 491app calls the close method of a socket, kernel sends fin to the other 492side of the connection, then the app has no relationship with the 493socket any more, but kernel need to keep the socket, this socket 494becomes an orphan socket, kernel waits for the reply of the other side, 495and would come to the TIME_WAIT state finally. When kernel has no 496enough memory to keep the orphan socket, kernel would send an RST to 497the other side, and delete the socket, in such situation, kernel will 498increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger 499TcpExtTCPAbortOnMemory: 500 5011. the memory used by the TCP protocol is higher than the third value of 502the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_: 503 504.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html 505 5062. the orphan socket count is higher than net.ipv4.tcp_max_orphans 507 508 509* TcpExtTCPAbortOnTimeout 510 511This counter will increase when any of the TCP timers expire. In such 512situation, kernel won't send RST, just give up the connection. 513 514* TcpExtTCPAbortOnLinger 515 516When a TCP connection comes into FIN_WAIT_2 state, instead of waiting 517for the fin packet from the other side, kernel could send a RST and 518delete the socket immediately. This is not the default behavior of 519Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option, 520you could let kernel follow this behavior. 521 522* TcpExtTCPAbortFailed 523 524The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is 525satisfied. If an internal error occurs during this process, 526TcpExtTCPAbortFailed will be increased. 527 528.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50 529 530TCP Hybrid Slow Start 531===================== 532The Hybrid Slow Start algorithm is an enhancement of the traditional 533TCP congestion window Slow Start algorithm. It uses two pieces of 534information to detect whether the max bandwidth of the TCP path is 535approached. The two pieces of information are ACK train length and 536increase in packet delay. For detail information, please refer the 537`Hybrid Slow Start paper`_. Either ACK train length or packet delay 538hits a specific threshold, the congestion control algorithm will come 539into the Congestion Avoidance state. Until v4.20, two congestion 540control algorithms are using Hybrid Slow Start, they are cubic (the 541default congestion control algorithm) and cdg. Four snmp counters 542relate with the Hybrid Slow Start algorithm. 543 544.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf 545 546* TcpExtTCPHystartTrainDetect 547 548How many times the ACK train length threshold is detected 549 550* TcpExtTCPHystartTrainCwnd 551 552The sum of CWND detected by ACK train length. Dividing this value by 553TcpExtTCPHystartTrainDetect is the average CWND which detected by the 554ACK train length. 555 556* TcpExtTCPHystartDelayDetect 557 558How many times the packet delay threshold is detected. 559 560* TcpExtTCPHystartDelayCwnd 561 562The sum of CWND detected by packet delay. Dividing this value by 563TcpExtTCPHystartDelayDetect is the average CWND which detected by the 564packet delay. 565 566TCP retransmission and congestion control 567========================================= 568The TCP protocol has two retransmission mechanisms: SACK and fast 569recovery. They are exclusive with each other. When SACK is enabled, 570the kernel TCP stack would use SACK, or kernel would use fast 571recovery. The SACK is a TCP option, which is defined in `RFC2018`_, 572the fast recovery is defined in `RFC6582`_, which is also called 573'Reno'. 574 575The TCP congestion control is a big and complex topic. To understand 576the related snmp counter, we need to know the states of the congestion 577control state machine. There are 5 states: Open, Disorder, CWR, 578Recovery and Loss. For details about these states, please refer page 5 579and page 6 of this document: 580https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf 581 582.. _RFC2018: https://tools.ietf.org/html/rfc2018 583.. _RFC6582: https://tools.ietf.org/html/rfc6582 584 585* TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery 586 587When the congestion control comes into Recovery state, if sack is 588used, TcpExtTCPSackRecovery increases 1, if sack is not used, 589TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP 590stack begins to retransmit the lost packets. 591 592* TcpExtTCPSACKReneging 593 594A packet was acknowledged by SACK, but the receiver has dropped this 595packet, so the sender needs to retransmit this packet. In this 596situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver 597could drop a packet which has been acknowledged by SACK, although it is 598unusual, it is allowed by the TCP protocol. The sender doesn't really 599know what happened on the receiver side. The sender just waits until 600the RTO expires for this packet, then the sender assumes this packet 601has been dropped by the receiver. 602 603* TcpExtTCPRenoReorder 604 605The reorder packet is detected by fast recovery. It would only be used 606if SACK is disabled. The fast recovery algorithm detects recorder by 607the duplicate ACK number. E.g., if retransmission is triggered, and 608the original retransmitted packet is not lost, it is just out of 609order, the receiver would acknowledge multiple times, one for the 610retransmitted packet, another for the arriving of the original out of 611order packet. Thus the sender would find more ACks than its 612expectation, and the sender knows out of order occurs. 613 614* TcpExtTCPTSReorder 615 616The reorder packet is detected when a hole is filled. E.g., assume the 617sender sends packet 1,2,3,4,5, and the receiving order is 6181,2,4,5,3. When the sender receives the ACK of packet 3 (which will 619fill the hole), two conditions will let TcpExtTCPTSReorder increase 6201: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet 6213 is retransmitted but the timestamp of the packet 3's ACK is earlier 622than the retransmission timestamp. 623 624* TcpExtTCPSACKReorder 625 626The reorder packet detected by SACK. The SACK has two methods to 627detect reorder: (1) DSACK is received by the sender. It means the 628sender sends the same packet more than one times. And the only reason 629is the sender believes an out of order packet is lost so it sends the 630packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and 631the sender has received SACKs for packet 2 and 5, now the sender 632receives SACK for packet 4 and the sender doesn't retransmit the 633packet yet, the sender would know packet 4 is out of order. The TCP 634stack of kernel will increase TcpExtTCPSACKReorder for both of the 635above scenarios. 636 637* TcpExtTCPSlowStartRetrans 638 639The TCP stack wants to retransmit a packet and the congestion control 640state is 'Loss'. 641 642* TcpExtTCPFastRetrans 643 644The TCP stack wants to retransmit a packet and the congestion control 645state is not 'Loss'. 646 647* TcpExtTCPLostRetransmit 648 649A SACK points out that a retransmission packet is lost again. 650 651* TcpExtTCPRetransFail 652 653The TCP stack tries to deliver a retransmission packet to lower layers 654but the lower layers return an error. 655 656* TcpExtTCPSynRetrans 657 658The TCP stack retransmits a SYN packet. 659 660DSACK 661===== 662The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report 663duplicate packets to the sender. There are two kinds of 664duplications: (1) a packet which has been acknowledged is 665duplicate. (2) an out of order packet is duplicate. The TCP stack 666counts these two kinds of duplications on both receiver side and 667sender side. 668 669.. _RFC2883 : https://tools.ietf.org/html/rfc2883 670 671* TcpExtTCPDSACKOldSent 672 673The TCP stack receives a duplicate packet which has been acked, so it 674sends a DSACK to the sender. 675 676* TcpExtTCPDSACKOfoSent 677 678The TCP stack receives an out of order duplicate packet, so it sends a 679DSACK to the sender. 680 681* TcpExtTCPDSACKRecv 682 683The TCP stack receives a DSACK, which indicates an acknowledged 684duplicate packet is received. 685 686* TcpExtTCPDSACKOfoRecv 687 688The TCP stack receives a DSACK, which indicate an out of order 689duplicate packet is received. 690 691invalid SACK and DSACK 692====================== 693When a SACK (or DSACK) block is invalid, a corresponding counter would 694be updated. The validation method is base on the start/end sequence 695number of the SACK block. For more details, please refer the comment 696of the function tcp_is_sackblock_valid in the kernel source code. A 697SACK option could have up to 4 blocks, they are checked 698individually. E.g., if 3 blocks of a SACk is invalid, the 699corresponding counter would be updated 3 times. The comment of commit 70018f02545a9a1 ("[TCP] MIB: Add counters for discarded SACK blocks") 701has additional explanation: 702 703* TcpExtTCPSACKDiscard 704 705This counter indicates how many SACK blocks are invalid. If the invalid 706SACK block is caused by ACK recording, the TCP stack will only ignore 707it and won't update this counter. 708 709* TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo 710 711When a DSACK block is invalid, one of these two counters would be 712updated. Which counter will be updated depends on the undo_marker flag 713of the TCP socket. If the undo_marker is not set, the TCP stack isn't 714likely to re-transmit any packets, and we still receive an invalid 715DSACK block, the reason might be that the packet is duplicated in the 716middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo 717will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld 718will be updated. As implied in its name, it might be an old packet. 719 720SACK shift 721========== 722The linux networking stack stores data in sk_buff struct (skb for 723short). If a SACK block acrosses multiple skb, the TCP stack will try 724to re-arrange data in these skb. E.g. if a SACK block acknowledges seq 72510 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and 72615 in skb2 would be moved to skb1. This operation is 'shift'. If a 727SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has 728seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be 729discard, this operation is 'merge'. 730 731* TcpExtTCPSackShifted 732 733A skb is shifted 734 735* TcpExtTCPSackMerged 736 737A skb is merged 738 739* TcpExtTCPSackShiftFallback 740 741A skb should be shifted or merged, but the TCP stack doesn't do it for 742some reasons. 743 744TCP out of order 745================ 746* TcpExtTCPOFOQueue 747 748The TCP layer receives an out of order packet and has enough memory 749to queue it. 750 751* TcpExtTCPOFODrop 752 753The TCP layer receives an out of order packet but doesn't have enough 754memory, so drops it. Such packets won't be counted into 755TcpExtTCPOFOQueue. 756 757* TcpExtTCPOFOMerge 758 759The received out of order packet has an overlay with the previous 760packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge 761packets will also be counted into TcpExtTCPOFOQueue. 762 763TCP PAWS 764======== 765PAWS (Protection Against Wrapped Sequence numbers) is an algorithm 766which is used to drop old packets. It depends on the TCP 767timestamps. For detail information, please refer the `timestamp wiki`_ 768and the `RFC of PAWS`_. 769 770.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17 771.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps 772 773* TcpExtPAWSActive 774 775Packets are dropped by PAWS in Syn-Sent status. 776 777* TcpExtPAWSEstab 778 779Packets are dropped by PAWS in any status other than Syn-Sent. 780 781TCP ACK skip 782============ 783In some scenarios, kernel would avoid sending duplicate ACKs too 784frequently. Please find more details in the tcp_invalid_ratelimit 785section of the `sysctl document`_. When kernel decides to skip an ACK 786due to tcp_invalid_ratelimit, kernel would update one of below 787counters to indicate the ACK is skipped in which scenario. The ACK 788would only be skipped if the received packet is either a SYN packet or 789it has no data. 790 791.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst 792 793* TcpExtTCPACKSkippedSynRecv 794 795The ACK is skipped in Syn-Recv status. The Syn-Recv status means the 796TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is 797waiting for an ACK. Generally, the TCP stack doesn't need to send ACK 798in the Syn-Recv status. But in several scenarios, the TCP stack need 799to send an ACK. E.g., the TCP stack receives the same SYN packet 800repeately, the received packet does not pass the PAWS check, or the 801received packet sequence number is out of window. In these scenarios, 802the TCP stack needs to send ACK. If the ACk sending frequency is higher than 803tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and 804increase TcpExtTCPACKSkippedSynRecv. 805 806 807* TcpExtTCPACKSkippedPAWS 808 809The ACK is skipped due to PAWS (Protect Against Wrapped Sequence 810numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2 811or Time-Wait statuses, the skipped ACK would be counted to 812TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or 813TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK 814would be counted to TcpExtTCPACKSkippedPAWS. 815 816* TcpExtTCPACKSkippedSeq 817 818The sequence number is out of window and the timestamp passes the PAWS 819check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait. 820 821* TcpExtTCPACKSkippedFinWait2 822 823The ACK is skipped in Fin-Wait-2 status, the reason would be either 824PAWS check fails or the received sequence number is out of window. 825 826* TcpExtTCPACKSkippedTimeWait 827 828The ACK is skipped in Time-Wait status, the reason would be either 829PAWS check failed or the received sequence number is out of window. 830 831* TcpExtTCPACKSkippedChallenge 832 833The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines 8343 kind of challenge ACK, please refer `RFC 5961 section 3.2`_, 835`RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these 836three scenarios, In some TCP status, the linux TCP stack would also 837send challenge ACKs if the ACK number is before the first 838unacknowledged number (more strict than `RFC 5961 section 5.2`_). 839 840.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7 841.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9 842.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11 843 844TCP receive window 845================== 846* TcpExtTCPWantZeroWindowAdv 847 848Depending on current memory usage, the TCP stack tries to set receive 849window to zero. But the receive window might still be a no-zero 850value. For example, if the previous window size is 10, and the TCP 851stack receives 3 bytes, the current window size would be 7 even if the 852window size calculated by the memory usage is zero. 853 854* TcpExtTCPToZeroWindowAdv 855 856The TCP receive window is set to zero from a no-zero value. 857 858* TcpExtTCPFromZeroWindowAdv 859 860The TCP receive window is set to no-zero value from zero. 861 862 863Delayed ACK 864=========== 865The TCP Delayed ACK is a technique which is used for reducing the 866packet count in the network. For more details, please refer the 867`Delayed ACK wiki`_ 868 869.. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment 870 871* TcpExtDelayedACKs 872 873A delayed ACK timer expires. The TCP stack will send a pure ACK packet 874and exit the delayed ACK mode. 875 876* TcpExtDelayedACKLocked 877 878A delayed ACK timer expires, but the TCP stack can't send an ACK 879immediately due to the socket is locked by a userspace program. The 880TCP stack will send a pure ACK later (after the userspace program 881unlock the socket). When the TCP stack sends the pure ACK later, the 882TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK 883mode. 884 885* TcpExtDelayedACKLost 886 887It will be updated when the TCP stack receives a packet which has been 888ACKed. A Delayed ACK loss might cause this issue, but it would also be 889triggered by other reasons, such as a packet is duplicated in the 890network. 891 892Tail Loss Probe (TLP) 893===================== 894TLP is an algorithm which is used to detect TCP packet loss. For more 895details, please refer the `TLP paper`_. 896 897.. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01 898 899* TcpExtTCPLossProbes 900 901A TLP probe packet is sent. 902 903* TcpExtTCPLossProbeRecovery 904 905A packet loss is detected and recovered by TLP. 906 907TCP Fast Open description 908========================= 909TCP Fast Open is a technology which allows data transfer before the 9103-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a 911general description. 912 913.. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open 914 915* TcpExtTCPFastOpenActive 916 917When the TCP stack receives an ACK packet in the SYN-SENT status, and 918the ACK packet acknowledges the data in the SYN packet, the TCP stack 919understand the TFO cookie is accepted by the other side, then it 920updates this counter. 921 922* TcpExtTCPFastOpenActiveFail 923 924This counter indicates that the TCP stack initiated a TCP Fast Open, 925but it failed. This counter would be updated in three scenarios: (1) 926the other side doesn't acknowledge the data in the SYN packet. (2) The 927SYN packet which has the TFO cookie is timeout at least once. (3) 928after the 3-way handshake, the retransmission timeout happens 929net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole 930fast open after the handshake. 931 932* TcpExtTCPFastOpenPassive 933 934This counter indicates how many times the TCP stack accepts the fast 935open request. 936 937* TcpExtTCPFastOpenPassiveFail 938 939This counter indicates how many times the TCP stack rejects the fast 940open request. It is caused by either the TFO cookie is invalid or the 941TCP stack finds an error during the socket creating process. 942 943* TcpExtTCPFastOpenListenOverflow 944 945When the pending fast open request number is larger than 946fastopenq->max_qlen, the TCP stack will reject the fast open request 947and update this counter. When this counter is updated, the TCP stack 948won't update TcpExtTCPFastOpenPassive or 949TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the 950TCP_FASTOPEN socket operation and it could not be larger than 951net.core.somaxconn. For example: 952 953setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen)); 954 955* TcpExtTCPFastOpenCookieReqd 956 957This counter indicates how many times a client wants to request a TFO 958cookie. 959 960SYN cookies 961=========== 962SYN cookies are used to mitigate SYN flood, for details, please refer 963the `SYN cookies wiki`_. 964 965.. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies 966 967* TcpExtSyncookiesSent 968 969It indicates how many SYN cookies are sent. 970 971* TcpExtSyncookiesRecv 972 973How many reply packets of the SYN cookies the TCP stack receives. 974 975* TcpExtSyncookiesFailed 976 977The MSS decoded from the SYN cookie is invalid. When this counter is 978updated, the received packet won't be treated as a SYN cookie and the 979TcpExtSyncookiesRecv counter won't be updated. 980 981Challenge ACK 982============= 983For details of challenge ACK, please refer the explanation of 984TcpExtTCPACKSkippedChallenge. 985 986* TcpExtTCPChallengeACK 987 988The number of challenge acks sent. 989 990* TcpExtTCPSYNChallenge 991 992The number of challenge acks sent in response to SYN packets. After 993updates this counter, the TCP stack might send a challenge ACK and 994update the TcpExtTCPChallengeACK counter, or it might also skip to 995send the challenge and update the TcpExtTCPACKSkippedChallenge. 996 997prune 998===== 999When a socket is under memory pressure, the TCP stack will try to 1000reclaim memory from the receiving queue and out of order queue. One of 1001the reclaiming method is 'collapse', which means allocate a big skb, 1002copy the contiguous skbs to the single big skb, and free these 1003contiguous skbs. 1004 1005* TcpExtPruneCalled 1006 1007The TCP stack tries to reclaim memory for a socket. After updates this 1008counter, the TCP stack will try to collapse the out of order queue and 1009the receiving queue. If the memory is still not enough, the TCP stack 1010will try to discard packets from the out of order queue (and update the 1011TcpExtOfoPruned counter) 1012 1013* TcpExtOfoPruned 1014 1015The TCP stack tries to discard packet on the out of order queue. 1016 1017* TcpExtRcvPruned 1018 1019After 'collapse' and discard packets from the out of order queue, if 1020the actually used memory is still larger than the max allowed memory, 1021this counter will be updated. It means the 'prune' fails. 1022 1023* TcpExtTCPRcvCollapsed 1024 1025This counter indicates how many skbs are freed during 'collapse'. 1026 1027examples 1028======== 1029 1030ping test 1031--------- 1032Run the ping command against the public dns server 8.8.8.8:: 1033 1034 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1 1035 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 1036 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms 1037 1038 --- 8.8.8.8 ping statistics --- 1039 1 packets transmitted, 1 received, 0% packet loss, time 0ms 1040 rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms 1041 1042The nstayt result:: 1043 1044 nstatuser@nstat-a:~$ nstat 1045 #kernel 1046 IpInReceives 1 0.0 1047 IpInDelivers 1 0.0 1048 IpOutRequests 1 0.0 1049 IcmpInMsgs 1 0.0 1050 IcmpInEchoReps 1 0.0 1051 IcmpOutMsgs 1 0.0 1052 IcmpOutEchos 1 0.0 1053 IcmpMsgInType0 1 0.0 1054 IcmpMsgOutType8 1 0.0 1055 IpExtInOctets 84 0.0 1056 IpExtOutOctets 84 0.0 1057 IpExtInNoECTPkts 1 0.0 1058 1059The Linux server sent an ICMP Echo packet, so IpOutRequests, 1060IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The 1061server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs, 1062IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply 1063was passed to the ICMP layer via IP layer, so IpInDelivers was 1064increased 1. The default ping data size is 48, so an ICMP Echo packet 1065and its corresponding Echo Reply packet are constructed by: 1066 1067* 14 bytes MAC header 1068* 20 bytes IP header 1069* 16 bytes ICMP header 1070* 48 bytes data (default value of the ping command) 1071 1072So the IpExtInOctets and IpExtOutOctets are 20+16+48=84. 1073 1074tcp 3-way handshake 1075------------------- 1076On server side, we run:: 1077 1078 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000 1079 Listening on [0.0.0.0] (family 0, port 9000) 1080 1081On client side, we run:: 1082 1083 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000 1084 Connection to 192.168.122.251 9000 port [tcp/*] succeeded! 1085 1086The server listened on tcp 9000 port, the client connected to it, they 1087completed the 3-way handshake. 1088 1089On server side, we can find below nstat output:: 1090 1091 nstatuser@nstat-b:~$ nstat | grep -i tcp 1092 TcpPassiveOpens 1 0.0 1093 TcpInSegs 2 0.0 1094 TcpOutSegs 1 0.0 1095 TcpExtTCPPureAcks 1 0.0 1096 1097On client side, we can find below nstat output:: 1098 1099 nstatuser@nstat-a:~$ nstat | grep -i tcp 1100 TcpActiveOpens 1 0.0 1101 TcpInSegs 1 0.0 1102 TcpOutSegs 2 0.0 1103 1104When the server received the first SYN, it replied a SYN+ACK, and came into 1105SYN-RCVD state, so TcpPassiveOpens increased 1. The server received 1106SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2 1107packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK 1108of the 3-way handshake is a pure ACK without data, so 1109TcpExtTCPPureAcks increased 1. 1110 1111When the client sent SYN, the client came into the SYN-SENT state, so 1112TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent 1113ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased 11141, TcpOutSegs increased 2. 1115 1116TCP normal traffic 1117------------------ 1118Run nc on server:: 1119 1120 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1121 Listening on [0.0.0.0] (family 0, port 9000) 1122 1123Run nc on client:: 1124 1125 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1126 Connection to nstat-b 9000 port [tcp/*] succeeded! 1127 1128Input a string in the nc client ('hello' in our example):: 1129 1130 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1131 Connection to nstat-b 9000 port [tcp/*] succeeded! 1132 hello 1133 1134The client side nstat output:: 1135 1136 nstatuser@nstat-a:~$ nstat 1137 #kernel 1138 IpInReceives 1 0.0 1139 IpInDelivers 1 0.0 1140 IpOutRequests 1 0.0 1141 TcpInSegs 1 0.0 1142 TcpOutSegs 1 0.0 1143 TcpExtTCPPureAcks 1 0.0 1144 TcpExtTCPOrigDataSent 1 0.0 1145 IpExtInOctets 52 0.0 1146 IpExtOutOctets 58 0.0 1147 IpExtInNoECTPkts 1 0.0 1148 1149The server side nstat output:: 1150 1151 nstatuser@nstat-b:~$ nstat 1152 #kernel 1153 IpInReceives 1 0.0 1154 IpInDelivers 1 0.0 1155 IpOutRequests 1 0.0 1156 TcpInSegs 1 0.0 1157 TcpOutSegs 1 0.0 1158 IpExtInOctets 58 0.0 1159 IpExtOutOctets 52 0.0 1160 IpExtInNoECTPkts 1 0.0 1161 1162Input a string in nc client side again ('world' in our example):: 1163 1164 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1165 Connection to nstat-b 9000 port [tcp/*] succeeded! 1166 hello 1167 world 1168 1169Client side nstat output:: 1170 1171 nstatuser@nstat-a:~$ nstat 1172 #kernel 1173 IpInReceives 1 0.0 1174 IpInDelivers 1 0.0 1175 IpOutRequests 1 0.0 1176 TcpInSegs 1 0.0 1177 TcpOutSegs 1 0.0 1178 TcpExtTCPHPAcks 1 0.0 1179 TcpExtTCPOrigDataSent 1 0.0 1180 IpExtInOctets 52 0.0 1181 IpExtOutOctets 58 0.0 1182 IpExtInNoECTPkts 1 0.0 1183 1184 1185Server side nstat output:: 1186 1187 nstatuser@nstat-b:~$ nstat 1188 #kernel 1189 IpInReceives 1 0.0 1190 IpInDelivers 1 0.0 1191 IpOutRequests 1 0.0 1192 TcpInSegs 1 0.0 1193 TcpOutSegs 1 0.0 1194 TcpExtTCPHPHits 1 0.0 1195 IpExtInOctets 58 0.0 1196 IpExtOutOctets 52 0.0 1197 IpExtInNoECTPkts 1 0.0 1198 1199Compare the first client-side nstat and the second client-side nstat, 1200we could find one difference: the first one had a 'TcpExtTCPPureAcks', 1201but the second one had a 'TcpExtTCPHPAcks'. The first server-side 1202nstat and the second server-side nstat had a difference too: the 1203second server-side nstat had a TcpExtTCPHPHits, but the first 1204server-side nstat didn't have it. The network traffic patterns were 1205exactly the same: the client sent a packet to the server, the server 1206replied an ACK. But kernel handled them in different ways. When the 1207TCP window scale option is not used, kernel will try to enable fast 1208path immediately when the connection comes into the established state, 1209but if the TCP window scale option is used, kernel will disable the 1210fast path at first, and try to enable it after kernel receives 1211packets. We could use the 'ss' command to verify whether the window 1212scale option is used. e.g. run below command on either server or 1213client:: 1214 1215 nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 ) 1216 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port 1217 tcp 0 0 192.168.122.250:40654 192.168.122.251:9000 1218 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98 1219 1220The 'wscale:7,7' means both server and client set the window scale 1221option to 7. Now we could explain the nstat output in our test: 1222 1223In the first nstat output of client side, the client sent a packet, server 1224reply an ACK, when kernel handled this ACK, the fast path was not 1225enabled, so the ACK was counted into 'TcpExtTCPPureAcks'. 1226 1227In the second nstat output of client side, the client sent a packet again, 1228and received another ACK from the server, in this time, the fast path is 1229enabled, and the ACK was qualified for fast path, so it was handled by 1230the fast path, so this ACK was counted into TcpExtTCPHPAcks. 1231 1232In the first nstat output of server side, fast path was not enabled, 1233so there was no 'TcpExtTCPHPHits'. 1234 1235In the second nstat output of server side, the fast path was enabled, 1236and the packet received from client qualified for fast path, so it 1237was counted into 'TcpExtTCPHPHits'. 1238 1239TcpExtTCPAbortOnClose 1240--------------------- 1241On the server side, we run below python script:: 1242 1243 import socket 1244 import time 1245 1246 port = 9000 1247 1248 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1249 s.bind(('0.0.0.0', port)) 1250 s.listen(1) 1251 sock, addr = s.accept() 1252 while True: 1253 time.sleep(9999999) 1254 1255This python script listen on 9000 port, but doesn't read anything from 1256the connection. 1257 1258On the client side, we send the string "hello" by nc:: 1259 1260 nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000 1261 1262Then, we come back to the server side, the server has received the "hello" 1263packet, and the TCP layer has acked this packet, but the application didn't 1264read it yet. We type Ctrl-C to terminate the server script. Then we 1265could find TcpExtTCPAbortOnClose increased 1 on the server side:: 1266 1267 nstatuser@nstat-b:~$ nstat | grep -i abort 1268 TcpExtTCPAbortOnClose 1 0.0 1269 1270If we run tcpdump on the server side, we could find the server sent a 1271RST after we type Ctrl-C. 1272 1273TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout 1274--------------------------------------------------- 1275Below is an example which let the orphan socket count be higher than 1276net.ipv4.tcp_max_orphans. 1277Change tcp_max_orphans to a smaller value on client:: 1278 1279 sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans" 1280 1281Client code (create 64 connection to server):: 1282 1283 nstatuser@nstat-a:~$ cat client_orphan.py 1284 import socket 1285 import time 1286 1287 server = 'nstat-b' # server address 1288 port = 9000 1289 1290 count = 64 1291 1292 connection_list = [] 1293 1294 for i in range(64): 1295 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1296 s.connect((server, port)) 1297 connection_list.append(s) 1298 print("connection_count: %d" % len(connection_list)) 1299 1300 while True: 1301 time.sleep(99999) 1302 1303Server code (accept 64 connection from client):: 1304 1305 nstatuser@nstat-b:~$ cat server_orphan.py 1306 import socket 1307 import time 1308 1309 port = 9000 1310 count = 64 1311 1312 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1313 s.bind(('0.0.0.0', port)) 1314 s.listen(count) 1315 connection_list = [] 1316 while True: 1317 sock, addr = s.accept() 1318 connection_list.append((sock, addr)) 1319 print("connection_count: %d" % len(connection_list)) 1320 1321Run the python scripts on server and client. 1322 1323On server:: 1324 1325 python3 server_orphan.py 1326 1327On client:: 1328 1329 python3 client_orphan.py 1330 1331Run iptables on server:: 1332 1333 sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP 1334 1335Type Ctrl-C on client, stop client_orphan.py. 1336 1337Check TcpExtTCPAbortOnMemory on client:: 1338 1339 nstatuser@nstat-a:~$ nstat | grep -i abort 1340 TcpExtTCPAbortOnMemory 54 0.0 1341 1342Check orphaned socket count on client:: 1343 1344 nstatuser@nstat-a:~$ ss -s 1345 Total: 131 (kernel 0) 1346 TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0 1347 1348 Transport Total IP IPv6 1349 * 0 - - 1350 RAW 1 0 1 1351 UDP 1 1 0 1352 TCP 14 13 1 1353 INET 16 14 2 1354 FRAG 0 0 0 1355 1356The explanation of the test: after run server_orphan.py and 1357client_orphan.py, we set up 64 connections between server and 1358client. Run the iptables command, the server will drop all packets from 1359the client, type Ctrl-C on client_orphan.py, the system of the client 1360would try to close these connections, and before they are closed 1361gracefully, these connections became orphan sockets. As the iptables 1362of the server blocked packets from the client, the server won't receive fin 1363from the client, so all connection on clients would be stuck on FIN_WAIT_1 1364stage, so they will keep as orphan sockets until timeout. We have echo 136510 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would 1366only keep 10 orphan sockets, for all other orphan sockets, the client 1367system sent RST for them and delete them. We have 64 connections, so 1368the 'ss -s' command shows the system has 10 orphan sockets, and the 1369value of TcpExtTCPAbortOnMemory was 54. 1370 1371An additional explanation about orphan socket count: You could find the 1372exactly orphan socket count by the 'ss -s' command, but when kernel 1373decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel 1374doesn't always check the exactly orphan socket count. For increasing 1375performance, kernel checks an approximate count firstly, if the 1376approximate count is more than tcp_max_orphans, kernel checks the 1377exact count again. So if the approximate count is less than 1378tcp_max_orphans, but exactly count is more than tcp_max_orphans, you 1379would find TcpExtTCPAbortOnMemory is not increased at all. If 1380tcp_max_orphans is large enough, it won't occur, but if you decrease 1381tcp_max_orphans to a small value like our test, you might find this 1382issue. So in our test, the client set up 64 connections although the 1383tcp_max_orphans is 10. If the client only set up 11 connections, we 1384can't find the change of TcpExtTCPAbortOnMemory. 1385 1386Continue the previous test, we wait for several minutes. Because of the 1387iptables on the server blocked the traffic, the server wouldn't receive 1388fin, and all the client's orphan sockets would timeout on the 1389FIN_WAIT_1 state finally. So we wait for a few minutes, we could find 139010 timeout on the client:: 1391 1392 nstatuser@nstat-a:~$ nstat | grep -i abort 1393 TcpExtTCPAbortOnTimeout 10 0.0 1394 1395TcpExtTCPAbortOnLinger 1396---------------------- 1397The server side code:: 1398 1399 nstatuser@nstat-b:~$ cat server_linger.py 1400 import socket 1401 import time 1402 1403 port = 9000 1404 1405 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1406 s.bind(('0.0.0.0', port)) 1407 s.listen(1) 1408 sock, addr = s.accept() 1409 while True: 1410 time.sleep(9999999) 1411 1412The client side code:: 1413 1414 nstatuser@nstat-a:~$ cat client_linger.py 1415 import socket 1416 import struct 1417 1418 server = 'nstat-b' # server address 1419 port = 9000 1420 1421 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1422 s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10)) 1423 s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1)) 1424 s.connect((server, port)) 1425 s.close() 1426 1427Run server_linger.py on server:: 1428 1429 nstatuser@nstat-b:~$ python3 server_linger.py 1430 1431Run client_linger.py on client:: 1432 1433 nstatuser@nstat-a:~$ python3 client_linger.py 1434 1435After run client_linger.py, check the output of nstat:: 1436 1437 nstatuser@nstat-a:~$ nstat | grep -i abort 1438 TcpExtTCPAbortOnLinger 1 0.0 1439 1440TcpExtTCPRcvCoalesce 1441-------------------- 1442On the server, we run a program which listen on TCP port 9000, but 1443doesn't read any data:: 1444 1445 import socket 1446 import time 1447 port = 9000 1448 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1449 s.bind(('0.0.0.0', port)) 1450 s.listen(1) 1451 sock, addr = s.accept() 1452 while True: 1453 time.sleep(9999999) 1454 1455Save the above code as server_coalesce.py, and run:: 1456 1457 python3 server_coalesce.py 1458 1459On the client, save below code as client_coalesce.py:: 1460 1461 import socket 1462 server = 'nstat-b' 1463 port = 9000 1464 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1465 s.connect((server, port)) 1466 1467Run:: 1468 1469 nstatuser@nstat-a:~$ python3 -i client_coalesce.py 1470 1471We use '-i' to come into the interactive mode, then a packet:: 1472 1473 >>> s.send(b'foo') 1474 3 1475 1476Send a packet again:: 1477 1478 >>> s.send(b'bar') 1479 3 1480 1481On the server, run nstat:: 1482 1483 ubuntu@nstat-b:~$ nstat 1484 #kernel 1485 IpInReceives 2 0.0 1486 IpInDelivers 2 0.0 1487 IpOutRequests 2 0.0 1488 TcpInSegs 2 0.0 1489 TcpOutSegs 2 0.0 1490 TcpExtTCPRcvCoalesce 1 0.0 1491 IpExtInOctets 110 0.0 1492 IpExtOutOctets 104 0.0 1493 IpExtInNoECTPkts 2 0.0 1494 1495The client sent two packets, server didn't read any data. When 1496the second packet arrived at server, the first packet was still in 1497the receiving queue. So the TCP layer merged the two packets, and we 1498could find the TcpExtTCPRcvCoalesce increased 1. 1499 1500TcpExtListenOverflows and TcpExtListenDrops 1501------------------------------------------- 1502On server, run the nc command, listen on port 9000:: 1503 1504 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1505 Listening on [0.0.0.0] (family 0, port 9000) 1506 1507On client, run 3 nc commands in different terminals:: 1508 1509 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1510 Connection to nstat-b 9000 port [tcp/*] succeeded! 1511 1512The nc command only accepts 1 connection, and the accept queue length 1513is 1. On current linux implementation, set queue length to n means the 1514actual queue length is n+1. Now we create 3 connections, 1 is accepted 1515by nc, 2 in accepted queue, so the accept queue is full. 1516 1517Before running the 4th nc, we clean the nstat history on the server:: 1518 1519 nstatuser@nstat-b:~$ nstat -n 1520 1521Run the 4th nc on the client:: 1522 1523 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1524 1525If the nc server is running on kernel 4.10 or higher version, you 1526won't see the "Connection to ... succeeded!" string, because kernel 1527will drop the SYN if the accept queue is full. If the nc client is running 1528on an old kernel, you would see that the connection is succeeded, 1529because kernel would complete the 3 way handshake and keep the socket 1530on half open queue. I did the test on kernel 4.15. Below is the nstat 1531on the server:: 1532 1533 nstatuser@nstat-b:~$ nstat 1534 #kernel 1535 IpInReceives 4 0.0 1536 IpInDelivers 4 0.0 1537 TcpInSegs 4 0.0 1538 TcpExtListenOverflows 4 0.0 1539 TcpExtListenDrops 4 0.0 1540 IpExtInOctets 240 0.0 1541 IpExtInNoECTPkts 4 0.0 1542 1543Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time 1544between the 4th nc and the nstat was longer, the value of 1545TcpExtListenOverflows and TcpExtListenDrops would be larger, because 1546the SYN of the 4th nc was dropped, the client was retrying. 1547 1548IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes 1549------------------------------------------------- 1550server A IP address: 192.168.122.250 1551server B IP address: 192.168.122.251 1552Prepare on server A, add a route to server B:: 1553 1554 $ sudo ip route add 8.8.8.8/32 via 192.168.122.251 1555 1556Prepare on server B, disable send_redirects for all interfaces:: 1557 1558 $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0 1559 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0 1560 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0 1561 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0 1562 1563We want to let sever A send a packet to 8.8.8.8, and route the packet 1564to server B. When server B receives such packet, it might send a ICMP 1565Redirect message to server A, set send_redirects to 0 will disable 1566this behavior. 1567 1568First, generate InAddrErrors. On server B, we disable IP forwarding:: 1569 1570 $ sudo sysctl -w net.ipv4.conf.all.forwarding=0 1571 1572On server A, we send packets to 8.8.8.8:: 1573 1574 $ nc -v 8.8.8.8 53 1575 1576On server B, we check the output of nstat:: 1577 1578 $ nstat 1579 #kernel 1580 IpInReceives 3 0.0 1581 IpInAddrErrors 3 0.0 1582 IpExtInOctets 180 0.0 1583 IpExtInNoECTPkts 3 0.0 1584 1585As we have let server A route 8.8.8.8 to server B, and we disabled IP 1586forwarding on server B, Server A sent packets to server B, then server B 1587dropped packets and increased IpInAddrErrors. As the nc command would 1588re-send the SYN packet if it didn't receive a SYN+ACK, we could find 1589multiple IpInAddrErrors. 1590 1591Second, generate IpExtInNoRoutes. On server B, we enable IP 1592forwarding:: 1593 1594 $ sudo sysctl -w net.ipv4.conf.all.forwarding=1 1595 1596Check the route table of server B and remove the default route:: 1597 1598 $ ip route show 1599 default via 192.168.122.1 dev ens3 proto static 1600 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251 1601 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static 1602 1603On server A, we contact 8.8.8.8 again:: 1604 1605 $ nc -v 8.8.8.8 53 1606 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable 1607 1608On server B, run nstat:: 1609 1610 $ nstat 1611 #kernel 1612 IpInReceives 1 0.0 1613 IpOutRequests 1 0.0 1614 IcmpOutMsgs 1 0.0 1615 IcmpOutDestUnreachs 1 0.0 1616 IcmpMsgOutType3 1 0.0 1617 IpExtInNoRoutes 1 0.0 1618 IpExtInOctets 60 0.0 1619 IpExtOutOctets 88 0.0 1620 IpExtInNoECTPkts 1 0.0 1621 1622We enabled IP forwarding on server B, when server B received a packet 1623which destination IP address is 8.8.8.8, server B will try to forward 1624this packet. We have deleted the default route, there was no route for 16258.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP 1626Destination Unreachable" message to server A. 1627 1628Third, generate IpOutNoRoutes. Run ping command on server B:: 1629 1630 $ ping -c 1 8.8.8.8 1631 connect: Network is unreachable 1632 1633Run nstat on server B:: 1634 1635 $ nstat 1636 #kernel 1637 IpOutNoRoutes 1 0.0 1638 1639We have deleted the default route on server B. Server B couldn't find 1640a route for the 8.8.8.8 IP address, so server B increased 1641IpOutNoRoutes. 1642 1643TcpExtTCPACKSkippedSynRecv 1644-------------------------- 1645In this test, we send 3 same SYN packets from client to server. The 1646first SYN will let server create a socket, set it to Syn-Recv status, 1647and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK 1648again, and record the reply time (the duplicate ACK reply time). The 1649third SYN will let server check the previous duplicate ACK reply time, 1650and decide to skip the duplicate ACK, then increase the 1651TcpExtTCPACKSkippedSynRecv counter. 1652 1653Run tcpdump to capture a SYN packet:: 1654 1655 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000 1656 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1657 1658Open another terminal, run nc command:: 1659 1660 nstatuser@nstat-a:~$ nc nstat-b 9000 1661 1662As the nstat-b didn't listen on port 9000, it should reply a RST, and 1663the nc command exited immediately. It was enough for the tcpdump 1664command to capture a SYN packet. A linux server might use hardware 1665offload for the TCP checksum, so the checksum in the /tmp/syn.pcap 1666might be not correct. We call tcprewrite to fix it:: 1667 1668 nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum 1669 1670On nstat-b, we run nc to listen on port 9000:: 1671 1672 nstatuser@nstat-b:~$ nc -lkv 9000 1673 Listening on [0.0.0.0] (family 0, port 9000) 1674 1675On nstat-a, we blocked the packet from port 9000, or nstat-a would send 1676RST to nstat-b:: 1677 1678 nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP 1679 1680Send 3 SYN repeatedly to nstat-b:: 1681 1682 nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done 1683 1684Check snmp counter on nstat-b:: 1685 1686 nstatuser@nstat-b:~$ nstat | grep -i skip 1687 TcpExtTCPACKSkippedSynRecv 1 0.0 1688 1689As we expected, TcpExtTCPACKSkippedSynRecv is 1. 1690 1691TcpExtTCPACKSkippedPAWS 1692----------------------- 1693To trigger PAWS, we could send an old SYN. 1694 1695On nstat-b, let nc listen on port 9000:: 1696 1697 nstatuser@nstat-b:~$ nc -lkv 9000 1698 Listening on [0.0.0.0] (family 0, port 9000) 1699 1700On nstat-a, run tcpdump to capture a SYN:: 1701 1702 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000 1703 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1704 1705On nstat-a, run nc as a client to connect nstat-b:: 1706 1707 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1708 Connection to nstat-b 9000 port [tcp/*] succeeded! 1709 1710Now the tcpdump has captured the SYN and exit. We should fix the 1711checksum:: 1712 1713 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum 1714 1715Send the SYN packet twice:: 1716 1717 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done 1718 1719On nstat-b, check the snmp counter:: 1720 1721 nstatuser@nstat-b:~$ nstat | grep -i skip 1722 TcpExtTCPACKSkippedPAWS 1 0.0 1723 1724We sent two SYN via tcpreplay, both of them would let PAWS check 1725failed, the nstat-b replied an ACK for the first SYN, skipped the ACK 1726for the second SYN, and updated TcpExtTCPACKSkippedPAWS. 1727 1728TcpExtTCPACKSkippedSeq 1729---------------------- 1730To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid 1731timestamp (to pass PAWS check) but the sequence number is out of 1732window. The linux TCP stack would avoid to skip if the packet has 1733data, so we need a pure ACK packet. To generate such a packet, we 1734could create two sockets: one on port 9000, another on port 9001. Then 1735we capture an ACK on port 9001, change the source/destination port 1736numbers to match the port 9000 socket. Then we could trigger 1737TcpExtTCPACKSkippedSeq via this packet. 1738 1739On nstat-b, open two terminals, run two nc commands to listen on both 1740port 9000 and port 9001:: 1741 1742 nstatuser@nstat-b:~$ nc -lkv 9000 1743 Listening on [0.0.0.0] (family 0, port 9000) 1744 1745 nstatuser@nstat-b:~$ nc -lkv 9001 1746 Listening on [0.0.0.0] (family 0, port 9001) 1747 1748On nstat-a, run two nc clients:: 1749 1750 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1751 Connection to nstat-b 9000 port [tcp/*] succeeded! 1752 1753 nstatuser@nstat-a:~$ nc -v nstat-b 9001 1754 Connection to nstat-b 9001 port [tcp/*] succeeded! 1755 1756On nstat-a, run tcpdump to capture an ACK:: 1757 1758 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001 1759 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1760 1761On nstat-b, send a packet via the port 9001 socket. E.g. we sent a 1762string 'foo' in our example:: 1763 1764 nstatuser@nstat-b:~$ nc -lkv 9001 1765 Listening on [0.0.0.0] (family 0, port 9001) 1766 Connection from nstat-a 42132 received! 1767 foo 1768 1769On nstat-a, the tcpdump should have captured the ACK. We should check 1770the source port numbers of the two nc clients:: 1771 1772 nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee 1773 State Recv-Q Send-Q Local Address:Port Peer Address:Port 1774 ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000 1775 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001 1776 1777Run tcprewrite, change port 9001 to port 9000, change port 42132 to 1778port 50208:: 1779 1780 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum 1781 1782Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b:: 1783 1784 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done 1785 1786Check TcpExtTCPACKSkippedSeq on nstat-b:: 1787 1788 nstatuser@nstat-b:~$ nstat | grep -i skip 1789 TcpExtTCPACKSkippedSeq 1 0.0 1790