asymetric speeds over gigE link

[Tom Judge]

root wrote:
> Summary:  Using iperf to measure TCP net speed between a linux and
> freebsd box over gigE, I see significant speed difference depending on
> the data direction.  Pushing data from the freebsd box to the linux box,
> I average about 500Gb/s.  Pushing data from the linux box to the freebsd
> box, I see about 300Gb/s.  It would seem that there is something
> limiting either the transmit side of the linux box or the recv side of
> the freebsd box. Any suggestions on how to narrow down who's tuning
> needs more work would also be welcome.  Just for grins, I tried upping
> the txqueuelen on the linux box to 1500, but no help.
> 
> I'd like to get closer to the theoretical speed of this link (I'd be
> happy if I could get 600-700 both ways), but the real puzzler is the
> difference.  The client is the linux box @192.168.1.104 and the iperf
> server is the freebsd box @ 192.168.1.15 .  Neither box is cpu or net
> busy during testing.  Both systems only have one nic.  I played with
> larger window sizes, but it only had a minor affect.  I went with the
> Intel Pro/1000's based on their good rep in both the linux and freebsd
> communities. 
> 
> Client connecting to 192.168.1.15, TCP port 5001
> TCP window size:   128 KByte (default)
> ------------------------------------------------------------
> [  6] local 192.168.1.104 port 34788 connected with 192.168.1.15 port 5001
> [  6]  0.0-10.0 sec    369 MBytes    309 Mbits/sec
> [  5] local 192.168.1.104 port 5001 connected with 192.168.1.15 port 52963
> [  5]  0.0-10.0 sec    597 MBytes    500 Mbits/sec
> 
> 
> Switch is the Netgear GS105 (5 port, supposedly wire speed, cables are
> Belkin 5e), both systems are on this switch.
> 
> FreeBSD box:
> FreeBSD 6.1-RELEASE-p10
> 1.9Ghz Athlon / 1 gig of main mem
> Abit/nforce2 MB+chipset (onboard nic disabled in bios)
> Intel Pro/1000GT NIC
> sysctl.conf:
> kern.ipc.maxsockbuf=8192000
> net.inet.tcp.sendspace=262144
> net.inet.tcp.recvspace=262144
> sysctl reports:
> kern.ipc.nmbjumbo16: 0
> kern.ipc.nmbjumbo9: 0
> kern.ipc.nmbjumbop: 0
> kern.ipc.nmbclusters: 25600
> net.inet.tcp.rfc1323: 1
> 
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         options=b<RXCSUM,TXCSUM,VLAN_MTU>
>         inet6 fe80::20e:cff:feda:1a3c%em0 prefixlen 64 scopeid 0x1
>         inet 192.168.1.15 netmask 0xffffff00 broadcast 192.168.1.255
>         ether 00:0e:0c:da:1a:3c
>         media: Ethernet autoselect (1000baseTX <full-duplex>)
>         status: active
> 
> Linux box:
> Linux emperor 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686
> GNU/Linux
> Soyo Dragon+ MB/ 1.9 Ghz Athlon/ 1 gig main mem.
> I run  tcp_tune.sh at boot on the linux box
> #!/bin/bash
> echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_rmem
> echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_wmem
> echo "131072 262144 8192000" > /proc/sys/net/ipv4/tcp_mem
> 
> /proc/sys/net/ipv4/tcp_window_scaling is 1
> /proc/sys/net/ipv4/tcp_timestamps is 1
> 
> eth1      Link encap:Ethernet  HWaddr 00:0E:0C:DA:1A:3B
>           inet addr:192.168.1.104  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::20e:cff:feda:1a3b/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:8139539 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:5638407 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:147638720 (140.7 MiB)  TX bytes:1109677958 (1.0 GiB)
>           Base address:0xd800 Memory:e2020000-e2040000
> 
> Settings for eth1:
>         Supported ports: [ TP ]
>         Supported link modes:   10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Full
>         Supports auto-negotiation: Yes
>         Advertised link modes:  10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Full
>         Advertised auto-negotiation: Yes
>         Speed: 1000Mb/s
>         Duplex: Full
>         Port: Twisted Pair
>         PHYAD: 0
>         Transceiver: internal
>         Auto-negotiation: on
>         Supports Wake-on: umbg
>         Wake-on: g
>         Current message level: 0x00000007 (7)
>         Link detected: yes
> 



Have you tried upping the MTU, that is if the cards and switch you are 
using support it.  I have seen significant speed increases (FreeBSD - 
FreeBSD) in some scenario's twice the through put with an MTU of 8192.

I have also seen 700Mb/s sustained FreeBSD - FreeBSD using the openssh 
HPN patch set and no extra tuning of the network stack.  Which makes me 
think that maybe the linux stack needs some tuning?

Just some ideas

Tom