How to troubleshoot slow network performance with Realtek 8169s?

[Juha Saarinen]

I have a 3.06GHz Intel P4 server running 5.4-RELEASE-p3, with the
following two NICs:

fxp0: <Intel 82801BA/CAM (ICH2/3) Pro/100 Ethernet> port 0xdf00-0xdf3f mem 0xfea
fc000-0xfeafcfff irq 20 at device 8.0 on pci2
miibus0: <MII bus> on fxp0
inphy0: <i82562ET 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:d0:b7:c3:0f:20
re0: <RealTek 8169S Single-chip Gigabit Ethernet> port 0xd800-0xd8ff mem 0xfeaff
800-0xfeaff8ff irq 22 at device 10.0 on pci2
miibus1: <MII bus> on re0
rgephy0: <RTL8169S/8110S media interface> on miibus1
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX
-FDX, auto
re0: Ethernet address: 00:08:a1:3c:72:bb

The Intel NIC is built into the motherboard, and the Realtek 8169s
sits in a 32-bit PCI slot.

Performance for the Realtek adapter is slow though: around 25Mbit/s
when serving clients with Fast Ethernet NICs, and around 250Mbit/s for
Gigabit Ethernet ones.

Having tried out the card in Windows boxes and seen the expected
600Mbit/s, I expected something similar with FreeBSD 5.4. The odd
thing is that you get approx. a quarter of the link speed with both
100TX and 1000TX but not more under FreeBSD 5.4.

The environment a switched network with known good cabling, standard
size Ethernet frames, media autoselect and a kernel with
DEVICE_POLLING and HZ=2000 which re(4) supports according to the
polling(4) man page.

#ifconfig re0
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=58<VLAN_MTU,VLAN_HWTAGGING,POLLING>
<snip>
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active

Enabling/disabling polling makes no real difference, apart from taking
the edge of performance somewhat.

#sysctl -a | grep polling
kern.polling.idlepoll_sleeping: 1
kern.polling.stalled: 0
kern.polling.suspect: 367
kern.polling.phase: 0
kern.polling.enable: 1
kern.polling.handlers: 1
kern.polling.residual_burst: 0
kern.polling.pending_polls: 0
kern.polling.lost_polls: 782
kern.polling.short_ticks: 731
kern.polling.reg_frac: 20
kern.polling.user_frac: 50
kern.polling.poll_in_trap: 0
kern.polling.idle_poll: 0
kern.polling.burst_max: 150
kern.polling.each_burst: 5
kern.polling.burst: 110

The re(4) man page doesn't have any troubleshooting hints and Googling
didn't turn up anything useful.

What else could I look for?

-- 

Juha