From nobody Fri Mar 17 12:31:37 2023 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PdNmB1LPXz3yK1M for ; Fri, 17 Mar 2023 12:31:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PdNm96Nj2z3mM6 for ; Fri, 17 Mar 2023 12:31:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679056297; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=F1MTlr5cGarFOttzpKwo4BGhBspmANfMyL/kNHq1LFk=; b=tGHVcj/00PtEwKWreEYRGt4zbt5iy8u1RT0Jufip9eoHws/uh5E0zkDvsYNAmZZ3Au0sTk LNVGCbbgMt9C7sfl6tUafDkN19be6urAdkeyhn6i5g4354wsn21aMFgsqQ4C+BKHbFIcfo 0nwXNNFmkkGKA49s8nj1sjAP3bbdLmkRVofmQZ6BJ3zRUe3US0wAoDUupnnKaDeWCw7BIz WOcFOJcdnZfEuJcz380bLMttV+iDnkej7/jde8TRHQSCUeQFPsQ47rb9sawmARQb131n0D ZQqr6sferaGAMOQmKj8zQ7jH2lt7zRISlIOu//PzZwEN4TuouYrsGC0oHYtGaA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679056297; a=rsa-sha256; cv=none; b=EeOPNnMeoeJgjynzueTBSSnHwxq6UVvjMxREYSIzbwZ3QJwvHh4xreZSicX1bWhr5Ffgi1 MLg1MOtvuasNJK92n2DBn1iJKnhptMeoUtEpdQJJNwqR6PubC0u9s7aLhKb+lg/2QIqQQs 0BLB2UEkovr1TRCRHSRuwp4y4vhStQcjExv6HiIKNX8ug+ckB5lUwLCpNCISdfcURe/qd+ DBmCSYpyvIEm/c/cxz9x2gMQfPeXaZCGkwqdXkXC217m9fmvQBoocWJrWq/TKg7kqQ0nkj +qD9XlHiOlAVLbevzq7tsUabYK5lakMjhYN11a47lo4hDQY1XUXPca0YHx1UcA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PdNm95RkWzQLP for ; Fri, 17 Mar 2023 12:31:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 32HCVbL5099750 for ; Fri, 17 Mar 2023 12:31:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 32HCVb2w099749 for bugs@FreeBSD.org; Fri, 17 Mar 2023 12:31:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 270285] Network issue with very small frames (tcp, padded) Date: Fri, 17 Mar 2023 12:31:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mhaarmann@midoco.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270285 Bug ID: 270285 Summary: Network issue with very small frames (tcp, padded) Product: Base System Version: 12.3-STABLE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: mhaarmann@midoco.de Hi experts, we are using freebsd under the hood of a pfsense firewall (latest CE releas= e, which builds on 12.3 Stable). Recently, we activated a bunch of servers in a new very fast (virtualized) environment, which is connected to the internal network with 10GBit. The pfsense firewall has 1GBit NICs on LAN and WAN side and is a physical machi= ne. We now encountered an issue which is probably resulting from an overload situation or MTU issue (while all MTU values are 1500 everywhere): When transmitting a medium large file over http (apache http server running= on Ubuntu on the server side), the traffic sometimes, not always contains a ve= ry small frame which is corrupting the output. We can see in the packet capture taken on the LAN side of the firewall that normally all data frames are 1514 bytes, but all of a sudden, a 60 byte fra= me (marked as PSH,ACK) arrives which contains a very small tcp segment data of= 4 bytes only. This frame is padded with two 0 bytes at the end AFTER the payl= oad. Extract from wireshark: Transmission Control Protocol, Src Port: 80, Dst Port: 37710, Seq: 553466, = Ack: 188, Len: 4Frame 136159: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Ethernet II, Src: e2:84:72:d3:14:5c (e2:84:72:d3:14:5c), Dst: IETF-VRRP-VRI= D_04 (00:00:5e:00:01:04) Destination: IETF-VRRP-VRID_04 (00:00:5e:00:01:04) Source: e2:84:72:d3:14:5c (e2:84:72:d3:14:5c) Type: IPv4 (0x0800) Padding: 0000 Internet Protocol Version 4, Src: 192.168.25.41, Dst: 80.xxx.xxx.xxx 0100 .... =3D Version: 4 .... 0101 =3D Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 44 Identification: 0x594c (22860) Flags: 0x40, Don't fragment ...0 0000 0000 0000 =3D Fragment Offset: 0 Time to Live: 64 Protocol: TCP (6) Header Checksum: 0x37c2 [validation disabled] [Header checksum status: Unverified] Source Address: 192.168.25.41 Destination Address: 80.xx.xx.xx Transmission Control Protocol, Src Port: 80, Dst Port: 37710, Seq: 553466, = Ack: 188, Len: 4 Source Port: 80 Destination Port: 37710 [Stream index: 77] [Conversation completeness: Complete, WITH_DATA (47)] [TCP Segment Len: 4] Sequence Number: 553466 (relative sequence number) Sequence Number (raw): 2934325866 [Next Sequence Number: 553470 (relative sequence number)] Acknowledgment Number: 188 (relative ack number) Acknowledgment number (raw): 2019009064 0101 .... =3D Header Length: 20 bytes (5) Flags: 0x018 (PSH, ACK) Window: 42153 [Calculated window size: 42153] [Window size scaling factor: -2 (no window scaling used)] Checksum: 0xc1c8 [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [SEQ/ACK analysis] TCP payload (4 bytes) [Reassembled PDU in frame: 137331] TCP segment data (4 bytes) 0000 00 00 5e 00 01 04 e2 84 72 d3 14 5c 08 00 45 00 0010 00 2c 59 4c 40 00 40 06 37 c2 c0 a8 19 29 50 XX 0020 XX XX 00 50 93 4e ae e6 42 6a 78 57 a2 28 50 18 0030 a4 a9 c1 c8 00 00 ee 86 11 a2 00 00 The real payload is only 4 bytes, (0xee, 0x86, 0x11, 0xa2), After that, two bytes are appended (sort of padding, resulting in a total of 60 bytes). We do not know why this padding occurs, but it is copied into the forwarded frame, leading to a currupted output. Any idea is welcome how to solve this. The padding seems to be the underlyi= ng reason of the data congestion, but we do not know if there is a way to prev= ent the padding on the sending side or if simply there is a bug in the tcp code= of Freebsd which forwards this padded bytes as content. Traffic between the internal machines is not affected for whatever reason (mostly Linux).=20 The congestion issues do not occur on each transfer, but only sometimes.=20 Most of the time, those small frames are not sent for whatever reason. I found in the cap file that there have been TCP window full / TCP window update messages, but these are=20 We have tested always with the same file, but the frame sizes vary on each transfer. Padding seems to be a well-known process in order to make small packets have the minimum length of 64 bytes, as I have read in some articles. So the tcp code should be able to handle this from my point of view. I can of course provide the .cap file with the defect flow if that makes se= nse. But my C knowledge is very limited, so I am not able to debug the kernel (a= nd as always, this happens in production, is not reproducable in testing, and = we do not have the identical network setup for test). Best regards, Marcus --=20 You are receiving this mail because: You are the assignee for the bug.=