Bad ZFS - NFS interaction? [ was: NFS server bottlenecks ]
Nikolay Denev
ndenev at gmail.com
Mon Oct 15 07:28:50 UTC 2012
On Oct 13, 2012, at 6:22 PM, Nikolay Denev <ndenev at gmail.com> wrote:
>
> On Oct 13, 2012, at 5:05 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
>
>> I wrote:
>>> Oops, I didn't get the "readahead" option description
>>> quite right in the last post. The default read ahead
>>> is 1, which does result in "rsize * 2", since there is
>>> the read + 1 readahead.
>>>
>>> "rsize * 16" would actually be for the option "readahead=15"
>>> and for "readahead=16" the calculation would be "rsize * 17".
>>>
>>> However, the example was otherwise ok, I think? rick
>>
>> I've attached the patch drc3.patch (it assumes drc2.patch has already been
>> applied) that replaces the single mutex with one for each hash list
>> for tcp. It also increases the size of NFSRVCACHE_HASHSIZE to 200.
>>
>> These patches are also at:
>> http://people.freebsd.org/~rmacklem/drc2.patch
>> http://people.freebsd.org/~rmacklem/drc3.patch
>> in case the attachments don't get through.
>>
>> rick
>> ps: I haven't tested drc3.patch a lot, but I think it's ok?
>
> drc3.patch applied and build cleanly and shows nice improvement!
>
> I've done a quick benchmark using iozone over the NFS mount from the Linux host.
>
> drc2.pach (but with NFSRVCACHE_HASHSIZE=500)
>
> TEST WITH 8K
> -------------------------------------------------------------------------------------------------
> Auto Mode
> Using Minimum Record Size 8 KB
> Using Maximum Record Size 8 KB
> Using minimum file size of 2097152 kilobytes.
> Using maximum file size of 2097152 kilobytes.
> O_DIRECT feature enabled
> SYNC Mode.
> OPS Mode. Output is in operations per second.
> Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o -O -i 0 -i 1 -i 2
> Time Resolution = 0.000001 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> random random bkwd record stride
> KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
> 2097152 8 1919 1914 2356 2321 2335 1706
>
> TEST WITH 1M
> -------------------------------------------------------------------------------------------------
> Auto Mode
> Using Minimum Record Size 1024 KB
> Using Maximum Record Size 1024 KB
> Using minimum file size of 2097152 kilobytes.
> Using maximum file size of 2097152 kilobytes.
> O_DIRECT feature enabled
> SYNC Mode.
> OPS Mode. Output is in operations per second.
> Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o -O -i 0 -i 1 -i 2
> Time Resolution = 0.000001 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> random random bkwd record stride
> KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
> 2097152 1024 73 64 477 486 496 61
>
>
> drc3.patch
>
> TEST WITH 8K
> -------------------------------------------------------------------------------------------------
> Auto Mode
> Using Minimum Record Size 8 KB
> Using Maximum Record Size 8 KB
> Using minimum file size of 2097152 kilobytes.
> Using maximum file size of 2097152 kilobytes.
> O_DIRECT feature enabled
> SYNC Mode.
> OPS Mode. Output is in operations per second.
> Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o -O -i 0 -i 1 -i 2
> Time Resolution = 0.000001 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> random random bkwd record stride
> KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
> 2097152 8 2108 2397 3001 3013 3010 2389
>
>
> TEST WITH 1M
> -------------------------------------------------------------------------------------------------
> Auto Mode
> Using Minimum Record Size 1024 KB
> Using Maximum Record Size 1024 KB
> Using minimum file size of 2097152 kilobytes.
> Using maximum file size of 2097152 kilobytes.
> O_DIRECT feature enabled
> SYNC Mode.
> OPS Mode. Output is in operations per second.
> Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o -O -i 0 -i 1 -i 2
> Time Resolution = 0.000001 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> random random bkwd record stride
> KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
> 2097152 1024 80 79 521 536 528 75
>
>
> Also with drc3 the CPU usage on the server is noticeably lower. Most of the time I could see only the geom{g_up}/{g_down} threads,
> and a few nfsd threads, before that nfsd's were much more prominent.
>
> I guess under bigger load the performance improvement can be bigger.
>
> I'll run some more tests with heavier loads this week.
>
> Thanks,
> Nikolay
>
>
If anyone is interested here's a FlameGraph generated using DTrace and
Brendan Gregg's tools from https://github.com/brendangregg/FlameGraph :
https://home.totalterror.net/freebsd/goliath-kernel.svg
It was sampled during Oracle database restore from Linux host over the nfs mount.
Currently all IO on the dataset that the linux machine writes is stuck, simple ls in the directory
hangs for maybe 10-15 minutes and then eventually completes.
Looks like some weird locking issue.
[*] http://dtrace.org/blogs/brendan/2011/12/16/flame-graphs/
P.S.: The machine runs with drc3.patch for the NFS server.
P.S.2: The nfsd server is configured for vfs.nfsd.maxthreads=200, maybe that's too much?
More information about the freebsd-fs
mailing list