can't get linerate TCP iperf between two Intel PRO/1000 82544GC cards

[Chris Tracy]

Hi,

I have been attempting to get iperf to generate a line-rate TCP flow
(~989Mbps) across a GigE link but can only get a maximum of around
912Mbps.

The bottleneck appears to be on the sending side, as top shows the CPU
for the iperf process at 50% (machine is a dual Xeon 2.4GHz with SMP
and HTT enabled) and the WCPU column goes to 99%.

Presumably the CPU is busy computing checksums, so I took a look at
if_em.c to find out more about what kind of hardware checksum
offloading features might be available, and I found this:

         if (adapter->hw.mac_type >= em_82543) {
                 if (ifp->if_capenable & IFCAP_TXCSUM)
                         ifp->if_hwassist = EM_CHECKSUM_FEATURES;

I wasn't sure which chipset I had, so I checked the output of pciconf:

# pciconf -v -l |grep 8254
     device   = '82544GC Gigabit Ethernet Controller (LOM)'

I then tried an ifconfig em0, and it indeed shows the rxcsum and
txcsum options listed in the output.  However, executing something
like 'ifconfig em0 txcsum' doesn't seem to help with the CPU issue.

I took a look at /usr/src/sys/dev/em/README and didn't see the GC
flavor of this card mentioned anywhere, nor any details or known
limitations with regard to checksum offloading.  I'm assuming GC means
Gigabit Copper since these are the 1000bT versions.

I also have some 82544EI/GC cards which I haven't gotten around to
testing yet because of a jumbo frame issue on one of the layer3
devices in between the hosts with those cards.

At this point, all that I can think of doing is to try and add some
debug statements to the driver so that I might be able to get a better
idea of what's going on with the hardware checksum...maybe it is
already offloading correctly and the CPU bottleneck is being caused by
something else?

I can provide additional details if needed, but please include my
email in the reply as I'm not subscribed to the list.  Any help would
be most appreciated.  I've had no problems doing 989Mbps in Linux, but
I'd really prefer to run FreeBSD.

Thanks,
-Chris