Abysmal re(4) performance under 8.1-STABLE (mid-August)

[Ulrich Spörlein]

Hello Pyun,

On this new server, I cannot get more than ~280kByte/s up/downstream out of
re(4) without any tweaking.

re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
        ether 00:21:85:63:74:34
        inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1 
        inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet autoselect (100baseTX <half-duplex>)
        status: active

re0 at pci0:2:0:0: class=0x020000 card=0x368c1462 chip=0x816810ec rev=0x01 hdr=0x00
    vendor     = 'Realtek Semiconductor'
    device     = 'Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111/8111c)'
    class      = network
    subclass   = ethernet

re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2
re0: Using 1 MSI messages
re0: Chip rev. 0x38000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
re0: Ethernet address: 00:21:85:63:74:34
re0: [FILTER]


It's interesting to note, that re0 only negotiates half-duplex, where
linux will negotiate full-duplex (and gets ~10MByte/s as expected).

Next, I disabled almost all options, except that I cannot disable
VLAN_MTU, VLAN_HWCSUM. I also forced the adapter into full-duplex.

# ifconfig re0 -vlanmtu
# ifconfig re0 -vlanhwcsum
ifconfig: -vlanhwcsum: bad value
# ifconfig re0
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=88<VLAN_MTU,VLAN_HWCSUM>
        ether 00:21:85:63:74:34
        inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1 
        inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
        status: active

If I then immediately start the test-download, I get a ~2MByte/s spike,
which quickly returns to something around 250kByte/s.

Booting with
hw.pci.enable_msix=0
hw.pci.enable_msi=0

I can get almost up to 400kByte/s, but this may be coincidence.

So this is usually as far as it gets:

re0  in    190.504 KB/s        246.136 KB/s           66.709 MB
     out    10.290 KB/s         12.985 KB/s            6.076 MB

But then I ran tcpdump in another session, and it looks like the ssh traffic on
the upstream of the interface will make the downloads running in another window
go faster:

re0  in    805.961 KB/s          1.577 MB/s          116.523 MB
     out   222.940 KB/s        502.045 KB/s           19.267 MB

Any ideas?

Uli