NFS server bottlenecks

[Nikolay Denev]

On Oct 13, 2012, at 5:05 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote:

> I wrote:
>> Oops, I didn't get the "readahead" option description
>> quite right in the last post. The default read ahead
>> is 1, which does result in "rsize * 2", since there is
>> the read + 1 readahead.
>> 
>> "rsize * 16" would actually be for the option "readahead=15"
>> and for "readahead=16" the calculation would be "rsize * 17".
>> 
>> However, the example was otherwise ok, I think? rick
> 
> I've attached the patch drc3.patch (it assumes drc2.patch has already been
> applied) that replaces the single mutex with one for each hash list
> for tcp. It also increases the size of NFSRVCACHE_HASHSIZE to 200.
> 
> These patches are also at:
>  http://people.freebsd.org/~rmacklem/drc2.patch
>  http://people.freebsd.org/~rmacklem/drc3.patch
> in case the attachments don't get through.
> 
> rick
> ps: I haven't tested drc3.patch a lot, but I think it's ok?

drc3.patch applied and build cleanly and shows nice improvement!

I've done a quick benchmark using iozone over the NFS mount from the Linux host.

drc2.pach (but with NFSRVCACHE_HASHSIZE=500)

	TEST WITH 8K
	-------------------------------------------------------------------------------------------------
        Auto Mode
        Using Minimum Record Size 8 KB
        Using Maximum Record Size 8 KB
        Using minimum file size of 2097152 kilobytes.
        Using maximum file size of 2097152 kilobytes.
        O_DIRECT feature enabled
        SYNC Mode. 
        OPS Mode. Output is in operations per second.
        Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o -O -i 0 -i 1 -i 2
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152       8    1919    1914     2356     2321    2335    1706                                                          

	TEST WITH 1M
	-------------------------------------------------------------------------------------------------
        Auto Mode
        Using Minimum Record Size 1024 KB
        Using Maximum Record Size 1024 KB
        Using minimum file size of 2097152 kilobytes.
        Using maximum file size of 2097152 kilobytes.
        O_DIRECT feature enabled
        SYNC Mode. 
        OPS Mode. Output is in operations per second.
        Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o -O -i 0 -i 1 -i 2
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    1024      73      64      477      486     496      61                                                          


drc3.patch

	TEST WITH 8K
	-------------------------------------------------------------------------------------------------
        Auto Mode
        Using Minimum Record Size 8 KB
        Using Maximum Record Size 8 KB
        Using minimum file size of 2097152 kilobytes.
        Using maximum file size of 2097152 kilobytes.
        O_DIRECT feature enabled
        SYNC Mode. 
        OPS Mode. Output is in operations per second.
        Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o -O -i 0 -i 1 -i 2
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152       8    2108    2397     3001     3013    3010    2389                                                          


	TEST WITH 1M
	-------------------------------------------------------------------------------------------------
        Auto Mode
        Using Minimum Record Size 1024 KB
        Using Maximum Record Size 1024 KB
        Using minimum file size of 2097152 kilobytes.
        Using maximum file size of 2097152 kilobytes.
        O_DIRECT feature enabled
        SYNC Mode. 
        OPS Mode. Output is in operations per second.
        Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o -O -i 0 -i 1 -i 2
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    1024      80      79      521      536     528      75                                                          


Also with drc3 the CPU usage on the server is noticeably lower. Most of the time I could see only the geom{g_up}/{g_down} threads,
and a few nfsd threads, before that nfsd's were much more prominent.

I guess under bigger load the performance improvement can be bigger.

I'll run some more tests with heavier loads this week.

Thanks,
Nikolay