7.2-RELEASE performance of em + 9k MTU + tso on ESB2 chipset

[Case van Rij]

While investigating a performance issue on a 7.2-RELEASE system I
noticed iperf performance
on systems with an ESB2 chipset (supermicro X7DBU onboard em) is worse
when TSO is
enabled at 9000 MTU:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:30:48:33:43:4e
        inet 10.53.13.161 netmask 0xffff0000 broadcast 10.53.255.255
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active

em0 at pci0:4:0:0: class=0x020000 card=0x000015d9 chip=0x10968086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'PRO/1000 EB Network Connection'

ad005# /usr/local/bin/iperf -c 10.53.13.236 -l 512k -w 1024k -N -t 30
------------------------------------------------------------
Client connecting to 10.53.13.236, TCP port 5001
TCP window size: 32.0 KByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 10.53.13.161 port 56621 connected with 10.53.13.236 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  2.69 GBytes    769 Mbits/sec

ad005# ifconfig em0 -tso
ad005# /usr/local/bin/iperf -c 10.53.13.236 -l 512k -w 1024k -N -t 30
------------------------------------------------------------
Client connecting to 10.53.13.236, TCP port 5001
TCP window size: 32.0 KByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 10.53.13.161 port 54876 connected with 10.53.13.236 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  3.26 GBytes    933 Mbits/sec

I looked through the archive and didn't see anything related to this,
but I'm curious
if this is a known issue.  I have not yet verified if the behaviour is
the same on 8.0

Curiously, this doesn't seem to be a problem (or much less so) on 82546EB

qaI-LSI-A-2# ifconfig em0 10.53.138.12 netmask 255.255.0.0 mtu 9000 tso
qaI-LSI-A-2# iperf -c 10.53.138.11 -l 512k -w 1024k -N -t 30
------------------------------------------------------------
Client connecting to 10.53.138.11, TCP port 5001
TCP window size: 1.01 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 10.53.138.12 port 49694 connected with 10.53.138.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  3.29 GBytes    942 Mbits/sec

qaI-LSI-A-2# ifconfig em0 10.53.138.12 netmask 255.255.0.0 mtu 9000 -tso
qaI-LSI-A-2# iperf -c 10.53.138.11 -l 512k -w 1024k -N -t 30
------------------------------------------------------------
Client connecting to 10.53.138.11, TCP port 5001
TCP window size: 1.01 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 10.53.138.12 port 52284 connected with 10.53.138.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  3.27 GBytes    935 Mbits/sec

Thanks,
Case