[Bug 234624] FreeBSD 12.0 TX network regression from 11.2 - potentially caused by TSO
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Sat Jan 5 00:06:28 UTC 2019
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234624
Bug ID: 234624
Summary: FreeBSD 12.0 TX network regression from 11.2 -
potentially caused by TSO
Product: Base System
Version: 12.0-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: misc
Assignee: bugs at FreeBSD.org
Reporter: freebsd at aoeu.uk
Hi,
Immediately after upgrading from FreeBSD 11.2 to 12.0, network TX performance
dropped significantly. Seems to be related to TSO, or at least disabling it
resolves the problem for me. Setup is a physical FreeBSD 12 host connected to
a mixture of physical and virtual Debian and Ubuntu machines via a 10gbit
switch.
23:58:18 root at filer:/root/ uname -a
FreeBSD filer.<censored> 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC
amd64
iperf3 tests from physical FreeBSD to virtual Debian:
23:32:39 root at filer:/root/ iperf3 -c 10.21.10.4
Connecting to host 10.21.10.4, port 5201
[ 5] local 10.21.10.2 port 16129 connected to 10.21.10.4 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 70.0 KBytes 572 Kbits/sec 8 17.5 KBytes
[ 5] 1.00-2.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes
[ 5] 2.00-3.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes
[ 5] 3.00-4.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes
[ 5] 4.00-5.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes
[ 5] 5.00-6.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes
[ 5] 6.00-7.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes
[ 5] 7.00-8.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes
[ 5] 8.00-9.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes
[ 5] 9.00-10.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 411 KBytes 336 Kbits/sec 86 sender
[ 5] 0.00-10.00 sec 376 KBytes 308 Kbits/sec receiver
iperf Done.
23:32:51 root at filer:/root/ iperf3 -c 10.21.10.4 -R
Connecting to host 10.21.10.4, port 5201
Reverse mode, remote host 10.21.10.4 is sending
[ 5] local 10.21.10.2 port 63402 connected to 10.21.10.4 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.15 GBytes 9.84 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.86 Gbits/sec
[ 5] 3.00-4.00 sec 1.13 GBytes 9.71 Gbits/sec
[ 5] 4.00-5.00 sec 1.13 GBytes 9.73 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.88 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.84 Gbits/sec receiver
iperf Done.
Turning off TSO:
23:33:17 root at filer:/root/ ifconfig ix0 -tso
Re-running iperf3:
23:33:57 root at filer:/root/ iperf3 -c 10.21.10.4
Connecting to host 10.21.10.4, port 5201
[ 5] local 10.21.10.2 port 60137 connected to 10.21.10.4 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.09 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec receiver
iperf Done.
23:34:12 root at filer:/root/ iperf3 -c 10.21.10.4 -R
Connecting to host 10.21.10.4, port 5201
Reverse mode, remote host 10.21.10.4 is sending
[ 5] local 10.21.10.2 port 31621 connected to 10.21.10.4 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.14 GBytes 9.81 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.84 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec receiver
iperf Done.
Network configuration is not entirely trivial. From /etc/rc.conf (ipv6 global
addresses censored):
ipv6_activate_all_interfaces="YES"
ifconfig_igb0="up mtu 9000"
ifconfig_igb1="up mtu 9000"
ifconfig_ix0="up mtu 9000"
ifconfig_ix1="up mtu 9000"
cloned_interfaces="lagg0 lagg1 vlan10 vlan11 vlan30 vlan34 vlan35"
ifconfig_lagg0="laggproto failover laggport ix0 laggport igb0"
ifconfig_lagg1="laggproto failover laggport ix1 laggport igb1"
ifconfig_vlan10="vlan 10 vlandev lagg0 mtu 9000"
ifconfig_vlan10_alias0="inet 10.21.10.2/24"
ifconfig_vlan10_ipv6="inet6 fd9c:bee6:9d33:10::2 prefixlen 64"
ifconfig_vlan11="vlan 11 vlandev lagg1 mtu 9000"
ifconfig_vlan11_alias0="inet 10.21.11.2/24"
ifconfig_vlan11_ipv6="inet6 fd9c:bee6:9d33:11::2 prefixlen 64"
ifconfig_vlan30="vlan 30 vlandev lagg0 mtu 9000"
ifconfig_vlan30_alias0="inet 10.21.30.3/24"
ifconfig_vlan30_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64"
ifconfig_vlan34="vlan 34 vlandev lagg1 mtu 9000"
ifconfig_vlan34_alias0="inet 10.21.34.2/24"
ifconfig_vlan34_ipv6="inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64"
ifconfig_vlan35="vlan 35 vlandev lagg1 mtu 1500"
ifconfig_vlan35_alias0="inet 10.21.35.3/24"
ifconfig_vlan35_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64"
defaultrouter="10.21.34.1"
ipv6_defaultrouter="2a01:xxxx:xxxx:xxxx::1"
23:46:03 root at filer:/root/ ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:94
media: Ethernet autoselect (10Gbase-Twinax
<full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:96
media: Ethernet autoselect (10Gbase-Twinax
<full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:94
hwaddr bc:5f:f4:c7:d5:58
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:96
hwaddr bc:5f:f4:c7:d5:56
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:94
inet6 fe80::a236:9fff:fe72:3594%lagg0 prefixlen 64 scopeid 0x6
laggproto failover lagghash l2,l3,l4
laggport: ix0 flags=5<MASTER,ACTIVE>
laggport: igb0 flags=0<>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:96
inet6 fe80::a236:9fff:fe72:3596%lagg1 prefixlen 64 scopeid 0x7
laggproto failover lagghash l2,l3,l4
laggport: ix1 flags=5<MASTER,ACTIVE>
laggport: igb1 flags=0<>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:94
inet6 fd9c:bee6:9d33:10::2 prefixlen 64
inet6 fe80::a236:9fff:fe72:3594%vlan10 prefixlen 64 scopeid 0x8
inet 10.21.10.2 netmask 0xffffff00 broadcast 10.21.10.255
groups: vlan
vlan: 10 vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:96
inet6 fd9c:bee6:9d33:11::2 prefixlen 64
inet6 fe80::a236:9fff:fe72:3596%vlan11 prefixlen 64 scopeid 0x9
inet 10.21.11.2 netmask 0xffffff00 broadcast 10.21.11.255
groups: vlan
vlan: 11 vlanpcp: 0 parent interface: lagg1
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:94
inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64
inet6 fe80::a236:9fff:fe72:3594%vlan30 prefixlen 64 scopeid 0xa
inet 10.21.30.3 netmask 0xffffff00 broadcast 10.21.30.255
groups: vlan
vlan: 30 vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan34: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:96
inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64
inet6 fe80::a236:9fff:fe72:3596%vlan34 prefixlen 64 scopeid 0xb
inet 10.21.34.2 netmask 0xffffff00 broadcast 10.21.34.255
groups: vlan
vlan: 34 vlanpcp: 0 parent interface: lagg1
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan35: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:72:35:96
inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64
inet6 fe80::a236:9fff:fe72:3596%vlan35 prefixlen 64 scopeid 0xc
inet 10.21.35.3 netmask 0xffffff00 broadcast 10.21.35.255
groups: vlan
vlan: 35 vlanpcp: 0 parent interface: lagg1
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
23:53:21 root at filer:/root/ netstat -I vlan10
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs
Coll
vlan1 9000 <Link#8> a0:36:9f:72:35:94 1957078 0 0 2672967 6
0
vlan1 - fd9c:bee6:9d3 fd9c:bee6:9d33:10 451401 - - 372777 -
-
vlan1 - fe80::%vlan10 fe80::a236:9fff:f 0 - - 4 -
-
vlan1 - 10.21.10.0/24 10.21.10.2 1505540 - - 2301252 -
-
23:53:25 root at filer:/root/ netstat -I lagg0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs
Coll
lagg0 9000 <Link#6> a0:36:9f:72:35:94 6043751 0 0 4156297 15
0
lagg0 - fe80::%lagg0/ fe80::a236:9fff:f 0 - - 3 -
-
23:53:31 root at filer:/root/ netstat -I ix0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs
Coll
ix0 9000 <Link#1> a0:36:9f:72:35:94 6043052 0 0 4156335 0
0
No seemingly relevant warnings or messages in dmesg or /var/log/messages.
23:56:26 root at filer:/root/ dmesg | grep "igb\|ix\|lagg"
acpi0: Power Button (fixed)
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe020-0xe03f mem
0xf0500000-0xf05fffff,0xf0704000-0xf0707fff irq 16 at device 0.0 on pci1
ix0: using 2048 tx descriptors and 2048 rx descriptors
ix0: msix_init qsets capped at 16
ix0: pxm cpus: 4 queue msgs: 63 admincnt: 1
ix0: using 4 rx queues 4 tx queues
ix0: Using MSIX interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: a0:36:9f:72:35:94
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe000-0xe01f mem
0xf0400000-0xf04fffff,0xf0700000-0xf0703fff irq 17 at device 0.1 on pci1
ix1: using 2048 tx descriptors and 2048 rx descriptors
ix1: msix_init qsets capped at 16
ix1: pxm cpus: 4 queue msgs: 63 admincnt: 1
ix1: using 4 rx queues 4 tx queues
ix1: Using MSIX interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: a0:36:9f:72:35:96
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xb000-0xb01f mem
0xf0c00000-0xf0c7ffff,0xf0c80000-0xf0c83fff irq 18 at device 0.0 on pci5
igb0: attach_pre capping queues at 4
igb0: using 1024 tx descriptors and 1024 rx descriptors
igb0: msix_init qsets capped at 4
igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb0: using 4 rx queues 4 tx queues
igb0: Using MSIX interrupts with 5 vectors
igb0: allocated for 4 tx_queues
igb0: allocated for 4 rx_queues
igb0: Ethernet address: bc:5f:f4:c7:d5:58
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xa000-0xa01f mem
0xf0900000-0xf097ffff,0xf0980000-0xf0983fff irq 19 at device 0.0 on pci9
igb1: attach_pre capping queues at 4
igb1: using 1024 tx descriptors and 1024 rx descriptors
igb1: msix_init qsets capped at 4
igb1: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb1: using 4 rx queues 4 tx queues
igb1: Using MSIX interrupts with 5 vectors
igb1: allocated for 4 tx_queues
igb1: allocated for 4 rx_queues
igb1: Ethernet address: bc:5f:f4:c7:d5:56
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
ix0: link state changed to UP
lagg0: IPv6 addresses on ix0 have been removed before adding it as a member to
prevent IPv6 address scope violation.
lagg0: link state changed to UP
lagg0: IPv6 addresses on igb0 have been removed before adding it as a member to
prevent IPv6 address scope violation.
ix1: link state changed to UP
lagg1: IPv6 addresses on ix1 have been removed before adding it as a member to
prevent IPv6 address scope violation.
lagg1: link state changed to UP
lagg1: IPv6 addresses on igb1 have been removed before adding it as a member to
prevent IPv6 address scope violation.
ix0: link state changed to DOWN
lagg0: link state changed to DOWN
ix1: link state changed to DOWN
lagg1: link state changed to DOWN
ix0: link state changed to UP
lagg0: link state changed to UP
ix1: link state changed to UP
lagg1: link state changed to UP
igb0: link state changed to UP
igb1: link state changed to UP
I'll perform some further tests over the weekend, see if failing over to the
1gbit igb interfaces or testing against a non-vlan-tagged interface makes any
difference. Let me know if any further info could be useful.
Kind regards,
Nick
--
You are receiving this mail because:
You are the assignee for the bug.
More information about the freebsd-bugs
mailing list