From nobody Sun Sep 12 11:12:24 2021 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E857517C1319 for ; Sun, 12 Sep 2021 11:12:35 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4H6n5H0l9Xz4qxh for ; Sun, 12 Sep 2021 11:12:34 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800:0:0:0:0:a135]) by mx0.gentlemail.de (8.15.2/8.15.2) with ESMTP id 18CBCQab014387 for ; Sun, 12 Sep 2021 13:12:26 +0200 (CEST) (envelope-from freebsd@omnilan.de) X-Authentication-Warning: mx0.gentlemail.de: Host ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800:0:0:0:0:a135] claimed to be mh0.gentlemail.de Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id E96E8D62 for ; Sun, 12 Sep 2021 13:12:25 +0200 (CEST) To: "net@FreeBSD.org" From: Harry Schmalzbauer Subject: TCP6 regression for MTU path on stable/13 Organization: OmniLAN Message-ID: <8e4f78e5-0717-8002-5364-44df5c8d7dad@omnilan.de> Date: Sun, 12 Sep 2021 13:12:24 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4H6n5H0l9Xz4qxh X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@omnilan.de designates 2a00:e10:2800::a130 as permitted sender) smtp.mailfrom=freebsd@omnilan.de X-Spamd-Result: default: False [-1.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[omnilan.de]; NEURAL_SPAM_SHORT(1.00)[1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_DN_EQ_ADDR_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:61157, ipnet:2a00:e10:2800::/38, country:DE]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hello, on one of my production stable/13 setups, MTU definitions from the routing table aren't respected (anymore?). I always had jumbo frames enabled and set a fixed MTU for routed destinations. No issues with stable/13 until april I guess (but I never checked for 'too big" ICMP6 messages before). After upgrading from stable/13~april -> stable/13-august, TCP6 connections suffer from massive MTU induced performance drops. No issue for ICMP4 For ICMP6, there seems to be a miscalculation anywhere. Let's start with TCP6: 42:c9:f9:fc:82:02 > 96:07:e9:f9:fc:85, ethertype IPv6 (0x86dd), length 1294: 2003:a:f43:84a2::1 > 2003:a:f43:84a2::10: ICMP6, packet too big, mtu 1492, length 1240 in response to 96:07:e9:f9:fc:85 > 42:c9:f9:fc:82:02, ethertype IPv6 (0x86dd), length 1770: 2003:a:f43:84a2::10.22 > 2003:a:47f:6ba1::3:1.55102: Flags [.], seq 2417:4101, ack 108, win 1030, options [nop,nop,TS val 4168798600 ecr 1552727989], length 1684 42:c9:f9:fc:82:02 is the next hop at 2003:a:f43:84a2::1, which is routing the packets. 96:07:e9:f9:fc:85 is a SSH server at 2003:a:f43:84a2::10, which responds to a shell command. 2003:a:47f:6ba1::3:1(.55102) is the SSH-client. 96:07:e9:f9:fc:85 / 2003:a:f43:84a2::10 transmitts a packet with length 1770 to 2003:a:47f:6ba1::3:1. Which mustn't happen, since: route -6 get 2003:a:47f:6ba1::3:1    route to: 2003:a:47f:6ba1::3:1 destination: 2003:a:47f:6ba1::        mask: ffff:ffff:ffff:ffff::     gateway: cupid         fib: 0   interface: ht0hsm       flags:  recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight expire        0         0         0         0      1492         1 0 With TSO enabled on if_igb(4) (i350), this results in unusable low connection bandwidth. Somehow not yet tracked, the 'packet too big' seem to get lost, and some timeout leads to retransmissions... Result is a *throughput of around 100kBit/s!* With TSO disabled, the problem still shows up, but impact is by far not that big. Here's an example of a - probably not directly related - ICMP6 oddity, I accidentally noticed during hunting the recently nitroduced issue: ping -s 1453 -D 2003:a:47f:6ba1::3:1 PING6(1501=40+8+1453 bytes) 2003:a:f43:84a2::2:130 --> 2003:a:47f:6ba1::3:1 ping: sendmsg: Message too long ping -s 1452 -D 2003:a:47f:6ba1::3:1 PING6(1500=40+8+1452 bytes) 2003:a:f43:84a2::2:130 --> 2003:a:47f:6ba1::3:1 no reply, because: 96:07:e9:f9:fc:85 > 42:c9:f9:fc:82:02, ethertype IPv6 (0x86dd), length 1514: 2003:a:f43:84a2::2:130 > 2003:a:47f:6ba1::3:1: ICMP6, echo request, seq 0, length 1460 42:c9:f9:fc:82:02 > 96:07:e9:f9:fc:85, ethertype IPv6 (0x86dd), length 1294: 2003:a:f43:84a2::1 > 2003:a:f43:84a2::2:130: ICMP6, packet too big, mtu 1492, length 1240 Altering mtu for the v6 route doesn't influence ICMP6 sendmsg behaviour at all, limit is always 1500 bytes, no matter what mtu I define! This is also true for stable/12 from long ago! Will try to further track it down, but in case anybody has an idea, what change during the last view months in stable/13 could have caused this real-world problem regarding resulting TCP6 throughput, I'm happy to start testing at that point. Thanks, -harry