local network throughput issues

[Jeremy Chadwick]

On Sat, Aug 16, 2008 at 02:43:11PM -0500, Rick C. Petty wrote:
> Hello.  I've been having some serious network throughput issues recently.
> Prior to these issues, I was running 7.0-STABLE from a few months back.  I
> recently build a world/kernel from csup(1) on 2008-Jul-25 and started
> noticing the issues.  I'm currently on a RELENG_7 from 2008-Aug-02 and am
> building another one today.
> 
> The server is running samba, dnsmasq, an NFS server, and is the gateway for
> a couple of freebsd machines.  I first noticed the issue when I was unable
> to play mp3s served over samba.  I was getting a throughput of about 3
> KBytes/sec over my gigabit switched network.  After a few weeks of trying
> different versions of samba and watching tcpdump/trafshow, I decided to try
> another test:
> 
> % ssh gateway cat /dev/zero | dd of=/dev/null
> load: 0.00  cmd: dd 68286 [runnable] 0.00u 0.00s 0% 844k
> 3680+0 records in
> 3680+0 records out
> 1884160 bytes transferred in 512.856684 secs (3674 bytes/sec)
> ^C3680+0 records in
> 3680+0 records out
> 1884160 bytes transferred in 513.392858 secs (3670 bytes/sec)
> Killed by signal 2.
> 
> Again, not even 4 KB/s throughput over ssh.  What's weird is that not all
> network traffic is slow.  I'm able to download at almost my full DSL speed
> (7168 Kbps) and here's the strange bit:  NFS is fast!  From the same
> machine as above, I have an NFS mountpoint:
> 
> % dd if=/nfs/some_large_file of=/dev/null
> 55980+0 records in
> 55980+0 records out
> 28661760 bytes transferred in 0.147159 secs (194767476 bytes/sec)
> 
> Which is what I would expect for a gigabit network.  Even if I perform the
> same test over ssh:
> 
> % ssh workstation cat /nfs/some_large_file | dd of=/dev/null
> 55980+0 records in
> 55980+0 records out
> 28661760 bytes transferred in 2.791392 secs (10267909 bytes/sec)
> 
> This is reasonable considering ssh overhead.  The kernel config for the
> server contains:
> 
> include GENERIC
> ident           DDB
> options KDB
> options KDB_TRACE
> options DDB
> options DDB_NUMSYM
> 
> My internal network is on 172.23.20.x and the DSL modem is connected to the
> same NIC on 192.168.0.1:
> 
> # ifconfig nfe0
> nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=10b<RXCSUM,TXCSUM,VLAN_MTU,TSO4>
>         ether 00:15:f2:17:0c:20
>         inet 172.23.20.1 netmask 0xffffff00 broadcast 172.23.20.255
>         inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255
>         media: Ethernet autoselect (1000baseTX <full-duplex,flag0,flag1>)
>         status: active
> 
> # ipfw show
> 00100    62968     8170100 allow ip from any to any via lo0
> 00110        0           0 deny ip from any to 127.0.0.0/8
> 00120        0           0 deny ip from 127.0.0.0/8 to any
> 00130 25444031 23419621364 divert 8668 ip from any to any via 192.168.0.3
> 00140        0           0 deny ip from not 172.23.20.0/24 to 172.23.20.0/24 dst-port 137-149,445
> 00150 20639141 15067620538 allow tcp from any to any established
> 00160       21       10683 allow ip from any to any frag
> 00170    79791     4696696 allow tcp from any to any setup
> 65530  5713367  8864760207 allow ip from any to any
> 65535        0           0 deny ip from any to any
> 
> Other than that, I'm not doing anything out of the ordinary.  Why is NFS
> behaving correctly and why are ssh/smbd connections so slow?  I've pasted
> my dmesg output below.  I've used this configuration for years and it
> wasn't until a recent RELENG_7 upgrade that I've had any problems.  The box
> was 99-100% idle during those tests and I don't see an interrupt storm or
> anything funny like that.  Any ideas?

1) Please provide netstat -in output.

2) NFS (unless you're explicitly disabling it) is UDP-based, while SSH
and Samba are TCP-based.  Your nfe0 device has TSO4 enabled on it, so
I'm left wondering if the TCP offloading support for your nfe(4) device
is broken.

Can you try disabling it by adding -tso to your ifconfig_nfe0 line in
/etc/rc.conf?  If you're using DHCP on that interface, that may pose
somewhat of a problem.

3) Can you disable the firewall (disable ipfw entirely) and see if the
problem continues?

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |