NFS server bottlenecks

[Nikolay Denev]

On Oct 20, 2012, at 10:45 PM, Outback Dingo <outbackdingo at gmail.com> wrote:

> On Sat, Oct 20, 2012 at 3:28 PM, Ivan Voras <ivoras at freebsd.org> wrote:
>> On 20 October 2012 14:45, Rick Macklem <rmacklem at uoguelph.ca> wrote:
>>> Ivan Voras wrote:
>> 
>>>> I don't know how to interpret the rise in context switches; as this is
>>>> kernel code, I'd expect no context switches. I hope someone else can
>>>> explain.
>>>> 
>>> Don't the mtx_lock() calls spin for a little while and then context
>>> switch if another thread still has it locked?
>> 
>> Yes, but are in-kernel context switches also counted? I was assuming
>> they are light-weight enough not to count.
>> 
>>> Hmm, I didn't look, but were there any tests using UDP mounts?
>>> (I would have thought that your patch would mainly affect UDP mounts,
>>> since that is when my version still has the single LRU queue/mutex.
>> 
>> Another assumption - I thought UDP was the default.
>> 
>>> As I think you know, my concern with your patch would be correctness
>>> for UDP, not performance.)
>> 
>> Yes.
> 
> Ive got a similar box config here, with 2x 10GB intel nics, and 24 2TB
> drives on an LSI controller.
> Im watching the thread patiently, im kinda looking for results, and
> answers, Though Im also tempted to
> run benchmarks on my system also see if i get similar results I also
> considered that netmap might be one
> but not quite sure if it would help NFS, since its to hard to tell if
> its a network bottle neck, though it appears
> to be network related.
> 
>> _______________________________________________
>> freebsd-hackers at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"

Doesn't look like network issue to me. From my observations it's more like some overhead in nfs and arc.
The boxes easily push 10G with simple iperf test.
Running two iperf test over each port of the dual ported 10G nics gives 960MB/sec regardles which machine is the server.
Also, I've seen over 960Gb/sec over NFS with this setup, but I can't understand what type of workload was able to do this.
At some point I was able to do this with simple dd, then after a reboot I was no longer to push this traffic.
I'm thinking something like ARC/kmem fragmentation might be the issue?