Terrible ix performance

[Lawrence Stewart]

On 07/03/13 14:28, Outback Dingo wrote:
> Ive got a high end storage server here, iperf shows decent network io
> 
> iperf -i 10 -t 20 -c 10.0.96.1 -w 2.5M -l 2.5M
> ------------------------------------------------------------
> Client connecting to 10.0.96.1, TCP port 5001
> TCP window size: 2.50 MByte (WARNING: requested 2.50 MByte)
> ------------------------------------------------------------
> [  3] local 10.0.96.2 port 34753 connected with 10.0.96.1 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  9.78 GBytes  8.40 Gbits/sec
> [  3] 10.0-20.0 sec  8.95 GBytes  7.69 Gbits/sec
> [  3]  0.0-20.0 sec  18.7 GBytes  8.05 Gbits/sec

Given that iperf exercises the ixgbe driver (ix), network path and TCP,
I would suggest that your subject is rather misleading ;)

> the card has a 3 meter twinax cable from cisco connected to it, going
> through a fujitsu switch. We have tweaked various networking, and kernel
> sysctls, however from a sftp and nfs session i cant get better then 100MBs
> from a zpool with 8 mirrored vdevs. We also have an identical box that will
> get 1.4Gbs with a 1 meter cisco twinax cables that writes 2.4Gbs compared
> to reads only 1.4Gbs...

I take it the RTT between both hosts is very low i.e. sub 1ms?

> does anyone have an idea of what the bottle neck could be?? This is a
> shared storage array with dual LSI controllers connected to 32 drives via
> an enclosure, local dd and other tests show the zpool performs quite well.
> however as soon as we introduce any type of protocol, sftp, samba, nfs
> performance plummets. Im quite puzzled and have run out of ideas. so now
> curiousity has me........ its loading the ix driver and working but not up
> to speed,

ssh (and sftp by extension) aren't often tuned for high speed operation.
Are you running with the HPN patch applied or a new enough FreeBSD that
has the patch included? Samba and NFS are both likely to need tuning for
multi-Gbps operation.

Cheers,
Lawrence