Date: Fri, 17 Mar 2023 12:31:37 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270285 Bug ID: 270285 Summary: Network issue with very small frames (tcp, padded) Product: Base System Version: 12.3-STABLE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: email@example.com Hi experts, we are using freebsd under the hood of a pfsense firewall (latest CE release, which builds on 12.3 Stable). Recently, we activated a bunch of servers in a new very fast (virtualized) environment, which is connected to the internal network with 10GBit. The pfsense firewall has 1GBit NICs on LAN and WAN side and is a physical machine. We now encountered an issue which is probably resulting from an overload situation or MTU issue (while all MTU values are 1500 everywhere): When transmitting a medium large file over http (apache http server running on Ubuntu on the server side), the traffic sometimes, not always contains a very small frame which is corrupting the output. We can see in the packet capture taken on the LAN side of the firewall that normally all data frames are 1514 bytes, but all of a sudden, a 60 byte frame (marked as PSH,ACK) arrives which contains a very small tcp segment data of 4 bytes only. This frame is padded with two 0 bytes at the end AFTER the payload. Extract from wireshark: Transmission Control Protocol, Src Port: 80, Dst Port: 37710, Seq: 553466, Ack: 188, Len: 4Frame 136159: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Ethernet II, Src: e2:84:72:d3:14:5c (e2:84:72:d3:14:5c), Dst: IETF-VRRP-VRID_04 (00:00:5e:00:01:04) Destination: IETF-VRRP-VRID_04 (00:00:5e:00:01:04) Source: e2:84:72:d3:14:5c (e2:84:72:d3:14:5c) Type: IPv4 (0x0800) Padding: 0000 Internet Protocol Version 4, Src: 192.168.25.41, Dst: 80.xxx.xxx.xxx 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 44 Identification: 0x594c (22860) Flags: 0x40, Don't fragment ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 64 Protocol: TCP (6) Header Checksum: 0x37c2 [validation disabled] [Header checksum status: Unverified] Source Address: 192.168.25.41 Destination Address: 80.xx.xx.xx Transmission Control Protocol, Src Port: 80, Dst Port: 37710, Seq: 553466, Ack: 188, Len: 4 Source Port: 80 Destination Port: 37710 [Stream index: 77] [Conversation completeness: Complete, WITH_DATA (47)] [TCP Segment Len: 4] Sequence Number: 553466 (relative sequence number) Sequence Number (raw): 2934325866 [Next Sequence Number: 553470 (relative sequence number)] Acknowledgment Number: 188 (relative ack number) Acknowledgment number (raw): 2019009064 0101 .... = Header Length: 20 bytes (5) Flags: 0x018 (PSH, ACK) Window: 42153 [Calculated window size: 42153] [Window size scaling factor: -2 (no window scaling used)] Checksum: 0xc1c8 [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [SEQ/ACK analysis] TCP payload (4 bytes) [Reassembled PDU in frame: 137331] TCP segment data (4 bytes) 0000 00 00 5e 00 01 04 e2 84 72 d3 14 5c 08 00 45 00 0010 00 2c 59 4c 40 00 40 06 37 c2 c0 a8 19 29 50 XX 0020 XX XX 00 50 93 4e ae e6 42 6a 78 57 a2 28 50 18 0030 a4 a9 c1 c8 00 00 ee 86 11 a2 00 00 The real payload is only 4 bytes, (0xee, 0x86, 0x11, 0xa2), After that, two bytes are appended (sort of padding, resulting in a total of 60 bytes). We do not know why this padding occurs, but it is copied into the forwarded frame, leading to a currupted output. Any idea is welcome how to solve this. The padding seems to be the underlying reason of the data congestion, but we do not know if there is a way to prevent the padding on the sending side or if simply there is a bug in the tcp code of Freebsd which forwards this padded bytes as content. Traffic between the internal machines is not affected for whatever reason (mostly Linux). The congestion issues do not occur on each transfer, but only sometimes. Most of the time, those small frames are not sent for whatever reason. I found in the cap file that there have been TCP window full / TCP window update messages, but these are We have tested always with the same file, but the frame sizes vary on each transfer. Padding seems to be a well-known process in order to make small packets have the minimum length of 64 bytes, as I have read in some articles. So the tcp code should be able to handle this from my point of view. I can of course provide the .cap file with the defect flow if that makes sense. But my C knowledge is very limited, so I am not able to debug the kernel (and as always, this happens in production, is not reproducable in testing, and we do not have the identical network setup for test). Best regards, Marcus -- You are receiving this mail because: You are the assignee for the bug.