local network throughput issues

[Rick C. Petty]

On Sat, Aug 16, 2008 at 03:15:35PM -0700, Jeremy Chadwick wrote:
> 
> 1) Please provide netstat -in output.

Name    Mtu Network       Address              Ipkts Ierrs    Opkts Oerrs  Coll
nfe0   1500 <Link#1>      00:15:f2:17:0c:20 14714298     0 14821857     0     0
nfe0   1500 172.23.20.0/2 172.23.20.1        2144815     - 12438012     -     -
nfe0   1500 192.168.0.0/2 192.168.0.3        9502416     -  2357751     -     -
lo0   16384 <Link#2>                           31707     0    31707     0     0
lo0   16384 fe80:2::1/64  fe80:2::1                0     -        0     -     -
lo0   16384 ::1/128       ::1                      6     -        6     -     -
lo0   16384 127.0.0.0/8   127.0.0.1            31588     -    31588     -     -
tun11  1500 <Link#3>                          985877     0   890589     0     0
tun11  1500 172.23.23.20/ 172.23.23.20        714240     -   713237     -     -
tun0   1500 <Link#4>                           22456     0    21872     0     0
tun0   1500 172.16.1.0/32 172.16.1.0               0     -        0     -     -

> 2) NFS (unless you're explicitly disabling it) is UDP-based, while SSH
> and Samba are TCP-based.  Your nfe0 device has TSO4 enabled on it, so
> I'm left wondering if the TCP offloading support for your nfe(4) device
> is broken.
> 
> Can you try disabling it by adding -tso to your ifconfig_nfe0 line in
> /etc/rc.conf?  If you're using DHCP on that interface, that may pose
> somewhat of a problem.

Yes, that seems to have made all the difference in the world:

% ssh gateway cat /dev/zero | dd of=/dev/null
load: 0.08  cmd: ssh 68698 [runnable] 1.20u 0.25s 11% 3020k
94384+0 records in
94384+0 records out
48324608 bytes transferred in 5.314707 secs (9092619 bytes/sec)
load: 0.08  cmd: ssh 68698 [runnable] 1.81u 0.33s 15% 3020k
147664+0 records in
147664+0 records out
75603968 bytes transferred in 7.652768 secs (9879297 bytes/sec)

So I'm thinking TSO wasn't an option in the older 7-stable I was running
and now it is, but the support for it is broken.

Your comment about DHCP, would that affect dhcpd or dhclient?  This is my
"server" machine so I don't run dhclient on it.  I hardcode the IP
connecting to the DSL modem.  I'm currently hardcoding all the other
machines also so I should be okay.

> 3) Can you disable the firewall (disable ipfw entirely) and see if the
> problem continues?

Well the firewall is primarily for NAT and port forwarding.  There's
nothing special about it.  It looks like the TSO disabling fixed my
problems.  Thank you for the suggestion!

-- Rick C. Petty