SSH and gigabit NICs

[Alex Zbyslaw]

The setup:
   Both machines FreeBSD/i386 5.4
   a) AMD64 machine with on-board Marvel Gigabit NIC
   b) Athlon XP with cheap SMC Gigabit NIC (also Marvel)
   Cabling is brand new Cat5e.  Have tried various different cables of 
different lengths to no effect.

To rule out problems with a cheap switch, I have just wired the NICs 
together.

To benchmark, I had a huge bz2 file (430Mb) which I copied with scp and 
ftp from machine a to machine b.

On cheap NetGear 100Mb cards, the transfers both took ~40 seconds which 
is ~80Mbit.

On the new Gigabit hardware, ftp drops to 17 seconds, but scp takes 
longer!  Out of the box (no tweaking of any relevant parameters) it now 
takes over 53 seconds.

After tweaking tons of stuff, I can make scp take maybe 43 seconds, but 
at those settings, ftp takes well over a minute!

What is going on?  I know that 17 seconds for the ftp is hardly stellar 
(200+Mbit or so) but for £50 I could live with that.  But for ssh to get 
slower just boggles.  These days, almost anything you do over a network 
ends up using ssh -- specifically I was hoping to make rsyncs faster -- 
but for them to get slower?

I've seen odd ssh network behaviour on other boxen.  A couple old Linux 
servers were 2-3 times slower for ssh than ftp, but I put it down to 
oldness and Linux and general weirdness.  They were on a 100Mbit network.

When I monitor with "systat -ifstat" I can see ftp keeping up a 
reasonably regular transfer rate, but when I watch the ssh, it yoyos up 
and down wildly, but still never gets above about 80Mbit.  Both machines 
have plenty of idle CPU and the ssh is not compressed.

Does anyone have a clue what might be going on?


So far I have tried:
    HZ=1000 on both machines.  No effect.

    various net.inet.tcp.recvspace and net.inet.tcp.recvspace values on 
both machines.  About 4096 (down from the default of 32768) makes ssh 
work "best", but stuffs ftp.  65536 improves ftp a bit but ssh goes up 
to 53 seconds (~64Mbit).

    MTU values of 5-9000.  ~6000 the scp seems to start a bit faster 
(maybe 100Mbit) but soon drops back into the 60s.

Before anyone asks, the driver doesn't seem to support polling.

--Alex, baffled and really quite annoyed.