General problems with checksums (txcsum/rxcsum) on FreeBSD 8.0?
alan bryan
alan.bryan at yahoo.com
Fri Jan 15 19:35:05 UTC 2010
--- On Fri, 1/15/10, Chuck Swiger <cswiger at mac.com> wrote:
> From: Chuck Swiger <cswiger at mac.com>
> Subject: Re: General problems with checksums (txcsum/rxcsum) on FreeBSD 8.0?
> To: "alan bryan" <alan.bryan at yahoo.com>
> Cc: freebsd-stable at freebsd.org
> Date: Friday, January 15, 2010, 11:21 AM
> Hi--
>
> On Jan 15, 2010, at 11:12 AM, alan bryan wrote:
> [ ... ]
> > I'm currently sifting through a tcpdump in wireshark
> and there are all sorts of messages in there about checksums
> being incorrect - both TCP and UDP.
>
> If you run tcpdump on a machine, it normally will receive
> the traffic being sent from that machine before the
> checksums are computed, especially if HW checksumming is
> being used. For reliable detection of these problems,
> you need to look at the traffic either on a hub or via the
> monitoring or span port of a smart switch, although simply
> glancing at the checksum stats from "netstat -s" on both
> sides should indicate whether significant error rates are
> happening.
>
> Regards,
> --
> -Chuck
Ah - thanks for the explanation as I'm new to tcpdump/wireshark. Need to learn more about those tools still.
Watching "systat -ip 1" gives results like:
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |||||
IP Input IP Output
4534 total packets received 4395 total packets sent
0 - with bad checksums 4395 - generated locally
0 - too short for header 0 - output drops
0 - too short for data 42 output fragments generated
0 - with invalid hlen 0 - fragmentation failed
0 - with invalid length 0 destinations unreachable
0 - with invalid version 0 packets output via raw IP
0 - jumbograms
84 total fragments received UDP Statistics
0 - fragments dropped 3989 total input packets
0 - fragments timed out 0 - too short for header
14 - packets reassembled ok 0 - invalid checksum
0 packets forwarded 0 - no checksum
0 - unreachable dests 0 - invalid length
0 - redirects generated 0 - no socket for dest port
0 option errors 0 - no socket for broadcast
0 unwanted multicasts 0 - socket buffer full
4464 delivered to upper layer 3989 total output packets
The total IP packets for this box should normally be in the hundreds per second - it stays constant at this 4K level.
And the matching netstat -s:
tcp:
47312791 packets sent
28828783 data packets (26317159806 bytes)
206 data packets (633246 bytes) retransmitted
11 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
12364262 ack-only packets (944193 delayed)
0 URG only packets
2518 window probe packets
3212717 window update packets
2903587 control packets
59176152 packets received
34785356 acks (for 26320662084 bytes)
1849829 duplicate acks
0 acks for unsent data
39787782 packets (24426616075 bytes) received in-sequence
601 completely duplicate packets (14975 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
101 out-of-order packets (114828 bytes)
2 packets (0 bytes) of data after window
0 window probes
4411949 window update packets
76 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
905931 connection requests
1098039 connection accepts
0 bad connection attempts
0 listen queue overflows
523 ignored RSTs in the windows
2003969 connections established (including accepts)
2002786 connections closed (including 4056 drops)
1187984 connections updated cached RTT on close
1191288 connections updated cached RTT variance on close
294899 connections updated cached ssthresh on close
0 embryonic connections dropped
34785321 segments updated rtt (of 29019225 attempts)
42636 retransmit timeouts
3512 connections dropped by rexmit timeout
2554 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
23 keepalive timeouts
23 keepalive probes sent
0 connections dropped by keepalive
79387 correct ACK header predictions
16380715 correct data packet header predictions
1098039 syncache entries added
0 retransmitted
0 dupsyn
0 dropped
1098039 completed
0 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
1098039 cookies sent
0 cookies received
2 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks) received
55 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
udp:
79558178 datagrams received
0 with incomplete header
0 with bad data length field
2 with bad checksum
929 with no checksum
1622 dropped due to no socket
27825 broadcast/multicast datagrams undelivered
0 dropped due to full socket buffers
0 not for hashed pcb
79528729 delivered
79523714 datagrams output
0 times multicast source filter matched
sctp:
0 input packets
0 datagrams
0 packets that had data
0 input SACK chunks
0 input DATA chunks
0 duplicate DATA chunks
0 input HB chunks
0 HB-ACK chunks
0 input ECNE chunks
0 input AUTH chunks
0 chunks missing AUTH
0 invalid HMAC ids received
0 invalid secret ids received
0 auth failed
0 fast path receives all one chunk
0 fast path multi-part data
0 output packets
0 output SACKs
0 output DATA chunks
0 retransmitted DATA chunks
0 fast retransmitted DATA chunks
0 FR's that happened more than once to same chunk.
0 intput HB chunks
0 output ECNE chunks
0 output AUTH chunks
0 ip_output error counter
Packet drop statistics:
0 from middle box
0 from end host
0 with data
0 non-data, non-endhost
0 non-endhost, bandwidth rep only
0 not enough for chunk header
0 not enough data to confirm
0 where process_chunk_drop said break
0 failed to find TSN
0 attempt reverse TSN lookup
0 e-host confirms zero-rwnd
0 midbox confirms no space
0 data did not match TSN
0 TSN's marked for Fast Retran
Timeouts:
0 iterator timers fired
0 T3 data time outs
0 window probe (T3) timers fired
0 INIT timers fired
0 sack timers fired
0 shutdown timers fired
0 heartbeat timers fired
0 a cookie timeout fired
0 an endpoint changed its cookiesecret
0 PMTU timers fired
0 shutdown ack timers fired
0 shutdown guard timers fired
0 stream reset timers fired
0 early FR timers fired
0 an asconf timer fired
0 auto close timer fired
0 asoc free timers expired
0 inp free timers expired
0 packet shorter than header
0 checksum error
0 no endpoint for port
0 bad v-tag
0 bad SID
0 no memory
0 number of multiple FR in a RTT window
0 RFC813 allowed sending
0 RFC813 does not allow sending
0 max burst dosn't allow sending
0 look ahead tells us no memory in interface
0 numbers of window probes sent
0 times an output error to clamp down on next user send.
0 times sctp_senderrors were caused from a user
0 number of in data drops due to chunk limit reached
0 number of in data drops due to rwnd limit reached
0 times a ECN reduced the cwnd
0 used express lookup via vtag
0 collision in express lookup.
0 times the sender ran dry of user data on primary
0 same for above
0 sacks the slow way
0 window update only sacks sent
0 sends with sinfo_flags !=0
0 unordered sends
0 sends with EOF flag set
0 sends with ABORT flag set
0 times protocol drain called
0 times we did a protocol drain
0 times recv was called with peek
0 cached chunks used
0 cached stream oq's used
0 unread messages abandonded by close
0 send burst avoidance, already max burst inflight to net
0 send cwnd full avoidance, already max burst inflight to net
0 number of map array over-runs via fwd-tsn's
ip:
153027955 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
17594719 fragments received
19 fragments dropped (dup or out of space)
11 fragments dropped after timeout
2948502 packets reassembled ok
138381640 packets for this host
14 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
0 packets not forwardable
55 packets received for unknown multicast group
0 redirects sent
126477794 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
1732555 output datagrams fragmented
10345284 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header
icmp:
1622 calls to icmp_error
0 errors not generated in response to an icmp message
Output histogram:
echo reply: 2
destination unreachable: 1622
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 multicast echo requests ignored
0 multicast timestamp requests ignored
Input histogram:
destination unreachable: 14
echo: 2
2 message responses generated
0 invalid return addresses
0 no return routes
ICMP address mask responses are disabled
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
ip6:
352688 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
352688 packets for this host
0 packets forwarded
0 packets not forwardable
0 redirects sent
352692 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
1 output packet discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
0 multicast packets which we don't join
Input histogram:
TCP: 352688
Mbuf statistics:
352688 one mbuf
0 one ext mbuf
0 two or more ext mbuf
0 packets whose headers are not continuous
0 tunneling packets that can't find gif
0 packets discarded because of too many headers
0 failures of source address selection
352687 forward cache hit
0 forward cache miss
Source addresses selection rule applied:
50384 first candidate
50384 same address
icmp6:
0 calls to icmp6_error
0 errors not generated in response to an icmp6 message
0 errors not generated because of rate limitation
Output histogram:
multicast listener report: 4
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Histogram of error messages to be generated:
0 no route
0 administratively prohibited
0 beyond scope
0 address unreachable
0 port unreachable
0 packet too big
0 time exceed transit
0 time exceed reassembly
0 erroneous header field
0 unrecognized next header
0 unrecognized option
0 redirect
0 unknown
0 message responses generated
0 messages with too many ND options
0 messages with bad ND options
0 bad neighbor solicitation messages
0 bad neighbor advertisement messages
0 bad router solicitation messages
0 bad router advertisement messages
0 bad redirect messages
0 path MTU changes
rip6:
0 messages received
0 checksum calcurations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output
Now for the NFS (FreeBSD 8.0) server side:
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |
IP Input IP Output
5388 total packets received 4717 total packets sent
0 - with bad checksums 4717 - generated locally
0 - too short for header 0 - output drops
0 - too short for data 65 output fragments generated
0 - with invalid hlen 0 - fragmentation failed
0 - with invalid length 0 destinations unreachable
0 - with invalid version 0 packets output via raw IP
0 - jumbograms
808 total fragments received UDP Statistics
0 - fragments dropped 4714 total input packets
0 - fragments timed out 0 - too short for header
137 - packets reassembled ok 0 - invalid checksum
0 packets forwarded 0 - no checksum
0 - unreachable dests 0 - invalid length
0 - redirects generated 0 - no socket for dest port
0 option errors 0 - no socket for broadcast
0 unwanted multicasts 0 - socket buffer full
4717 delivered to upper layer 4714 total output packets
Again - packet volume should be in the hundreds right now with bursts during traffic - but this is a constant. There is no growth in disk usage over time - nothing is actually getting written out anywhere. If I kill everything on the FreeBSD 7.0 client the packets still transfer until the client is rebooted. Then everything is usually OK again and then often serveral hrs later the FreeBSD 8.0 server loses networking or simply reboots.
And to complete the stats, here's netstat -s for the FreeBSD 8.0 NFS server:
tcp:
11241344 packets sent
8436896 data packets (24959317163 bytes)
100 data packets (112180 bytes) retransmitted
1 data packet unnecessarily retransmitted
0 resends initiated by MTU discovery
2282609 ack-only packets (6239 delayed)
0 URG only packets
0 window probe packets
518530 window update packets
3305 control packets
17239617 packets received
10845777 acks (for 24959571358 bytes)
560 duplicate acks
0 acks for unsent data
10437746 packets (4690003575 bytes) received in-sequence
17 completely duplicate packets (11070 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
39 out-of-order packets (43478 bytes)
0 packets (0 bytes) of data after window
0 window probes
1130377 window update packets
132 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
3 connection requests
3310 connection accepts
1 bad connection attempt
0 listen queue overflows
0 ignored RSTs in the windows
3313 connections established (including accepts)
3302 connections closed (including 1 drop)
2233 connections updated cached RTT on close
2233 connections updated cached RTT variance on close
21 connections updated cached ssthresh on close
0 embryonic connections dropped
8077952 segments updated rtt (of 6908327 attempts)
10 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
10 keepalive timeouts
9 keepalive probes sent
1 connection dropped by keepalive
1341332 correct ACK header predictions
5196626 correct data packet header predictions
3316 syncache entries added
18 retransmitted
2 dupsyn
0 dropped
3310 completed
0 bucket overflow
0 cache overflow
0 reset
6 stale
0 aborted
0 badack
0 unreach
0 zone failures
3316 cookies sent
0 cookies received
18 SACK recovery episodes
89 segment rexmits in SACK recovery episodes
101595 byte rexmits in SACK recovery episodes
640 SACK options (SACK blocks) received
77 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
udp:
91334533 datagrams received
0 with incomplete header
0 with bad data length field
8 with bad checksum
16 with no checksum
254 dropped due to no socket
4339 broadcast/multicast datagrams undelivered
56 dropped due to full socket buffers
0 not for hashed pcb
91329876 delivered
91316517 datagrams output
0 times multicast source filter matched
sctp:
0 input packets
0 datagrams
0 packets that had data
0 input SACK chunks
0 input DATA chunks
0 duplicate DATA chunks
0 input HB chunks
0 HB-ACK chunks
0 input ECNE chunks
0 input AUTH chunks
0 chunks missing AUTH
0 invalid HMAC ids received
0 invalid secret ids received
0 auth failed
0 fast path receives all one chunk
0 fast path multi-part data
0 output packets
0 output SACKs
0 output DATA chunks
0 retransmitted DATA chunks
0 fast retransmitted DATA chunks
0 FR's that happened more than once to same chunk
0 intput HB chunks
0 output ECNE chunks
0 output AUTH chunks
0 ip_output error counter
Packet drop statistics:
0 from middle box
0 from end host
0 with data
0 non-data, non-endhost
0 non-endhost, bandwidth rep only
0 not enough for chunk header
0 not enough data to confirm
0 where process_chunk_drop said break
0 failed to find TSN
0 attempt reverse TSN lookup
0 e-host confirms zero-rwnd
0 midbox confirms no space
0 data did not match TSN
0 TSN's marked for Fast Retran
Timeouts:
3 iterator timers fired
0 T3 data time outs
0 window probe (T3) timers fired
0 INIT timers fired
0 sack timers fired
0 shutdown timers fired
0 heartbeat timers fired
0 a cookie timeout fired
0 an endpoint changed its cookiesecret
0 PMTU timers fired
0 shutdown ack timers fired
0 shutdown guard timers fired
0 stream reset timers fired
0 early FR timers fired
0 an asconf timer fired
0 auto close timer fired
0 asoc free timers expired
0 inp free timers expired
0 packet shorter than header
0 checksum error
0 no endpoint for port
0 bad v-tag
0 bad SID
0 no memory
0 number of multiple FR in a RTT window
0 RFC813 allowed sending
0 RFC813 does not allow sending
0 times max burst prohibited sending
0 look ahead tells us no memory in interface
0 numbers of window probes sent
0 times an output error to clamp down on next user send
0 times sctp_senderrors were caused from a user
0 number of in data drops due to chunk limit reached
0 number of in data drops due to rwnd limit reached
0 times a ECN reduced the cwnd
0 used express lookup via vtag
0 collision in express lookup
0 times the sender ran dry of user data on primary
0 same for above
0 sacks the slow way
0 window update only sacks sent
0 sends with sinfo_flags !=0
0 unordered sends
0 sends with EOF flag set
0 sends with ABORT flag set
0 times protocol drain called
0 times we did a protocol drain
0 times recv was called with peek
0 cached chunks used
0 cached stream oq's used
0 unread messages abandonded by close
0 send burst avoidance, already max burst inflight to net
0 send cwnd full avoidance, already max burst inflight to net
0 number of map array over-runs via fwd-tsn's
ip:
122321395 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
16507934 fragments received
19 fragments dropped (dup or out of space)
970 fragments dropped after timeout
2761830 packets reassembled ok
108575291 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
0 packets not forwardable
0 packets received for unknown multicast group
0 redirects sent
102560932 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
3269107 output datagrams fragmented
19499266 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header
icmp:
254 calls to icmp_error
0 errors not generated in response to an icmp message
Output histogram:
echo reply: 1132
destination unreachable: 254
0 messages with bad code fields
0 messages less than the minimum length
0 messages with bad checksum
0 messages with bad length
0 multicast echo requests ignored
0 multicast timestamp requests ignored
Input histogram:
echo reply: 6
echo: 1132
1132 message responses generated
0 invalid return addresses
0 no return routes
ICMP address mask responses are disabled
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with wrong TTL
0 messages received with bad checksum
0 V1/V2 membership queries received
0 V3 membership queries received
0 membership queries received with invalid field(s)
0 general queries received
0 group queries received
0 group-source queries received
0 group-source queries dropped
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 V3 reports received without Router Alert
0 membership reports sent
ip6:
166 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
0 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
166 packets for this host
0 packets forwarded
0 packets not forwardable
0 redirects sent
0 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 packets that violated scope rules
166 multicast packets which we don't join
Input histogram:
ICMP6: 166
Mbuf statistics:
0 one mbuf
166 one ext mbuf
0 two or more ext mbuf
0 packets whose headers are not continuous
0 tunneling packets that can't find gif
0 packets discarded because of too many headers
0 failures of source address selection
Source addresses selection rule applied:
icmp6:
0 calls to icmp6_error
0 errors not generated in response to an icmp6 message
0 errors not generated because of rate limitation
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Histogram of error messages to be generated:
0 no route
0 administratively prohibited
0 beyond scope
0 address unreachable
0 port unreachable
0 packet too big
0 time exceed transit
0 time exceed reassembly
0 erroneous header field
0 unrecognized next header
0 unrecognized option
0 redirect
0 unknown
0 message responses generated
0 messages with too many ND options
0 messages with bad ND options
0 bad neighbor solicitation messages
0 bad neighbor advertisement messages
0 bad router solicitation messages
0 bad router advertisement messages
0 bad redirect messages
0 path MTU changes
rip6:
0 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
0 delivered
0 datagrams output
More information about the freebsd-stable
mailing list