Terrible ix performance

[Scott Long]

On Jul 2, 2013, at 10:28 PM, Outback Dingo <outbackdingo at gmail.com> wrote:

> Ive got a high end storage server here, iperf shows decent network io
> 
> 
> the card has a 3 meter twinax cable from cisco connected to it, going
> through a fujitsu switch. We have tweaked various networking, and kernel
> sysctls, however from a sftp and nfs session i cant get better then 100MBs
> from a zpool with 8 mirrored vdevs. We also have an identical box that will
> get 1.4Gbs with a 1 meter cisco twinax cables that writes 2.4Gbs compared
> to reads only 1.4Gbs...
> 
> does anyone have an idea of what the bottle neck could be?? This is a
> shared storage array with dual LSI controllers connected to 32 drives via
> an enclosure, local dd and other tests show the zpool performs quite well.
> however as soon as we introduce any type of protocol, sftp, samba, nfs
> performance plummets. Im quite puzzled and have run out of ideas. so now
> curiousity has me........ its loading the ix driver and working but not up
> to speed,
> it is feasible it should be using the ixgbe driver??

Try turning off interrupt moderation.  Add the following to /boot/loader.conf

hw.ixgbe.enable_aim=0

NAS workloads are extremely sensitive to latency, and interrupt moderation
only adds latency.  We tune some other things as well at Netflix and manage
to get quite good performance, though with a fairly different workload.  Let
me know if this does or does not work for you.

Thanks,
Scott