strange TCP issue on RELENG_7

[Mike Tancsa]

At 04:01 PM 8/22/2008, Bjoern A. Zeeb wrote:
>On Fri, 22 Aug 2008, Mike Tancsa wrote:
>
>>At 03:12 PM 8/22/2008, Bjoern A. Zeeb wrote:
>>
>>>can you make sure you have this?
>>>http://svn.freebsd.org/changeset/base/181596
>>
>>Hi,
>>I do. I am running a GENERIC kernel but with inet6 disabled from yesterday
>>
>>7.0-STABLE #0: Thu Aug 21 10:27:04 EDT 2008
>>
>>and with the patch below as TOE seems to be broken for my workload
>
>okay.
>
>Does netstat -s -p tcp should any obvious error counters raising?
>
>I just try to get the easy things sorted out before anyone starts
>to debug more TCP...

Not sure, what is obvious :)  This host is part of the primary MX 
list for a lot of inbound traffic, so there will be all sorts of 
hosts using all sorts of stacks connecting to it.

netstat -ni shows no errors as do the stats from the em driver (below)

tcp:
         26989994 packets sent
                 17034559 data packets (84842431 bytes)
                 101073 data packets (20893246 bytes) retransmitted
                 4586 data packets unnecessarily retransmitted
                 2 resends initiated by MTU discovery
                 6523153 ack-only packets (333850 delayed)
                 0 URG only packets
                 0 window probe packets
                 1016646 window update packets
                 2314525 control packets
         27848527 packets received
                 15078177 acks (for 78677787 bytes)
                 830948 duplicate acks
                 4689 acks for unsent data
                 11374257 packets (4127132455 bytes) received in-sequence
                 40287 completely duplicate packets (30901526 bytes)
                 921 old duplicate packets
                 352 packets with some dup. data (179712 bytes duped)
                 225759 out-of-order packets (276463593 bytes)
                 8 packets (0 bytes) of data after window
                 0 window probes
                 710635 window update packets
                 298525 packets received after close
                 849 discarded for bad checksums
                 0 discarded for bad header offset fields
                 0 discarded because packet too short
                 5855 discarded due to memory problems
         855019 connection requests
         1091001 connection accepts
         0 bad connection attempts
         10943 listen queue overflows
         79448 ignored RSTs in the windows
         1944666 connections established (including accepts)
         2088626 connections closed (including 516007 drops)
                 840374 connections updated cached RTT on close
                 842332 connections updated cached RTT variance on close
                 60049 connections updated cached ssthresh on close
         1201 embryonic connections dropped
         14026440 segments updated rtt (of 12656642 attempts)
         157370 retransmit timeouts
                 7412 connections dropped by rexmit timeout
         1 persist timeout
                 0 connections dropped by persist timeout
         0 Connections (fin_wait_2) dropped because of timeout
         886 keepalive timeouts
                 13 keepalive probes sent
                 873 connections dropped by keepalive
         110658 correct ACK header predictions
         7810293 correct data packet header predictions
         1114950 syncache entries added
                 63212 retransmitted
                 35592 dupsyn
                 77 dropped
                 1091001 completed
                 0 bucket overflow
                 0 cache overflow
                 4908 reset
                 8260 stale
                 10943 aborted
                 0 badack
                 174 unreach
                 0 zone failures
         1115027 cookies sent
         361 cookies received
         2843 SACK recovery episodes
         2620 segment rexmits in SACK recovery episodes
         3722090 byte rexmits in SACK recovery episodes
         35491 SACK options (SACK blocks) received
         162860 SACK options (SACK blocks) sent
         0 SACK scoreboard overflow


Aug 22 16:13:28 smtp2 kernel: em0: Excessive collisions = 0
Aug 22 16:13:28 smtp2 kernel: em0: Sequence errors = 0
Aug 22 16:13:28 smtp2 kernel: em0: Defer count = 0
Aug 22 16:13:28 smtp2 kernel: em0: Missed Packets = 0
Aug 22 16:13:28 smtp2 kernel: em0: Receive No Buffers = 0
Aug 22 16:13:28 smtp2 kernel: em0: Receive Length Errors = 0
Aug 22 16:13:28 smtp2 kernel: em0: Receive errors = 0
Aug 22 16:13:28 smtp2 kernel: em0: Crc errors = 0
Aug 22 16:13:28 smtp2 kernel: em0: Alignment errors = 0
Aug 22 16:13:28 smtp2 kernel: em0: Collision/Carrier extension errors = 0
Aug 22 16:13:28 smtp2 kernel: em0: RX overruns = 0
Aug 22 16:13:28 smtp2 kernel: em0: watchdog timeouts = 0
Aug 22 16:13:28 smtp2 kernel: em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 
LINK MSIX IRQ = 0
Aug 22 16:13:28 smtp2 kernel: em0: XON Rcvd = 0
Aug 22 16:13:28 smtp2 kernel: em0: XON Xmtd = 0
Aug 22 16:13:28 smtp2 kernel: em0: XOFF Rcvd = 0
Aug 22 16:13:28 smtp2 kernel: em0: XOFF Xmtd = 0
Aug 22 16:13:28 smtp2 kernel: em0: Good Packets Rcvd = 16920559
Aug 22 16:13:28 smtp2 kernel: em0: Good Packets Xmtd = 16025617
Aug 22 16:13:28 smtp2 kernel: em0: TSO Contexts Xmtd = 0
Aug 22 16:13:28 smtp2 kernel: em0: TSO Contexts Failed = 0
Aug 22 16:13:31 smtp2 kernel: em1: Excessive collisions = 0
Aug 22 16:13:31 smtp2 kernel: em1: Sequence errors = 0
Aug 22 16:13:31 smtp2 kernel: em1: Defer count = 0
Aug 22 16:13:31 smtp2 kernel: em1: Missed Packets = 0
Aug 22 16:13:31 smtp2 kernel: em1: Receive No Buffers = 0
Aug 22 16:13:31 smtp2 kernel: em1: Receive Length Errors = 0
Aug 22 16:13:31 smtp2 kernel: em1: Receive errors = 0
Aug 22 16:13:31 smtp2 kernel: em1: Crc errors = 0
Aug 22 16:13:31 smtp2 kernel: em1: Alignment errors = 0
Aug 22 16:13:31 smtp2 kernel: em1: Collision/Carrier extension errors = 0
Aug 22 16:13:31 smtp2 kernel: em1: RX overruns = 0
Aug 22 16:13:31 smtp2 kernel: em1: watchdog timeouts = 0
Aug 22 16:13:31 smtp2 kernel: em1: RX MSIX IRQ = 0 TX MSIX IRQ = 0 
LINK MSIX IRQ = 0
Aug 22 16:13:31 smtp2 kernel: em1: XON Rcvd = 0
Aug 22 16:13:31 smtp2 kernel: em1: XON Xmtd = 0
Aug 22 16:13:31 smtp2 kernel: em1: XOFF Rcvd = 0
Aug 22 16:13:31 smtp2 kernel: em1: XOFF Xmtd = 0
Aug 22 16:13:31 smtp2 kernel: em1: Good Packets Rcvd = 13566035
Aug 22 16:13:31 smtp2 kernel: em1: Good Packets Xmtd = 14313612
Aug 22 16:13:31 smtp2 kernel: em1: TSO Contexts Xmtd = 0
Aug 22 16:13:31 smtp2 kernel: em1: TSO Contexts Failed = 0