Abysmal re(4) performance under 8.1-STABLE (mid-August)

[Pyun YongHyeon]

On Sat, Nov 6, 2010 at 2:37 AM, Ulrich Spörlein <uqs at spoerlein.net> wrote:
> Hello Pyun,
>
> On this new server, I cannot get more than ~280kByte/s up/downstream out of
> re(4) without any tweaking.
>
> re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
>        ether 00:21:85:63:74:34
>        inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1
>        inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191
>        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
>        media: Ethernet autoselect (100baseTX <half-duplex>)
>        status: active
>

It seems the link was resolved to half-duplex. Does link partner
also agree on the resolved speed/duplex?

> re0 at pci0:2:0:0: class=0x020000 card=0x368c1462 chip=0x816810ec rev=0x01 hdr=0x00
>    vendor     = 'Realtek Semiconductor'
>    device     = 'Gigabit Ethernet NIC(NDIS 6.0) (RTL8168/8111/8111c)'
>    class      = network
>    subclass   = ethernet
>
> re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2
> re0: Using 1 MSI messages
> re0: Chip rev. 0x38000000
> re0: MAC rev. 0x00000000
> miibus0: <MII bus> on re0
> rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
> rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
> re0: Ethernet address: 00:21:85:63:74:34
> re0: [FILTER]
>
>
> It's interesting to note, that re0 only negotiates half-duplex, where
> linux will negotiate full-duplex (and gets ~10MByte/s as expected).
>
> Next, I disabled almost all options, except that I cannot disable
> VLAN_MTU, VLAN_HWCSUM. I also forced the adapter into full-duplex.
>
> # ifconfig re0 -vlanmtu
> # ifconfig re0 -vlanhwcsum
> ifconfig: -vlanhwcsum: bad value

I'm sure this has nothing to do that this issue.
If you want to disable checksum offloading of VLAN
interface, use vlan interface instead of parent interface
of the VLAN interface(i.e. ifconfig vlan0 -txcsum -rxcsum).
And you can't disable VLAN_MTU on re(4). There is no
reason to disable supporting VLAN oversized frames.

> # ifconfig re0
> re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=88<VLAN_MTU,VLAN_HWCSUM>
>        ether 00:21:85:63:74:34
>        inet6 fe80::221:85ff:fe63:7434%re0 prefixlen 64 scopeid 0x1
>        inet 46.4.12.147 netmask 0xffffffc0 broadcast 46.4.12.191
>        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
>        media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
>        status: active
>

This time, it seems  you used forced media configuration
instead of auto. It still shows duplex mismatch so it's
normal to see poor performance. What makes me wonder
is why you have duplex mismatch?
Did you use forced media configuration on link partner?
What happens when you use different switch?

> If I then immediately start the test-download, I get a ~2MByte/s spike,
> which quickly returns to something around 250kByte/s.
>
> Booting with
> hw.pci.enable_msix=0
> hw.pci.enable_msi=0
>
> I can get almost up to 400kByte/s, but this may be coincidence.
>
> So this is usually as far as it gets:
>
> re0  in    190.504 KB/s        246.136 KB/s           66.709 MB
>     out    10.290 KB/s         12.985 KB/s            6.076 MB
>
> But then I ran tcpdump in another session, and it looks like the ssh traffic on
> the upstream of the interface will make the downloads running in another window
> go faster:
>
> re0  in    805.961 KB/s          1.577 MB/s          116.523 MB
>     out   222.940 KB/s        502.045 KB/s           19.267 MB
>
> Any ideas?
>
> Uli
>