An order of magnitude higher IOPS needed with ZFS than UFS

[Attila Nagy]

Hi,

On 06/12/13 01:21, Kenneth D. Merry wrote:
> On Tue, Jun 11, 2013 at 17:20:09 -0400, Rick Macklem wrote:
>>
>> ken@ recently committed a change to the new NFS server to add file
>> handle affinity support to it. He reported that he had found that,
>> without file handle affinity, that ZFS's sequential reading heuristic
>> broke badly (or something like that, you can probably find the email
>> thread or maybe he will chime in).
> That is correct.  The problem, if the I/O is sequential, is that simultaneous
> requests for adjacent blocks in a file will get farmed out to different
The IO is pretty much random, and the files aren't so big either (mean 
size around 400k).

> threads in the NFS server.  These can easily go down into ZFS out of order,
> and make the ZFS prefetch code think that the file is not being read
> sequentially.  It blows away the zfetch stream, and you wind up with a lot
> of I/O bandwidth getting used (with a lot of prefetching done and then
> re-done), but not much performance.
I've tried disabling prefetch, without any noticeable effects.
>
> Linux clients are more likely than FreeBSD and MacOS clients to queue a lot
> of reads to the server.
The clients are also FreeBSD (8.3 and 7.2 mostly). Running NFSv3 of course.
>
>> Anyhow, you could try switching the FreeBSD 9 system to use the old
>> NFS server (assuming your clients are doing NFSv3 mounts) and see if
>> that has a significant effect. (For FreeBSD9, the old server has file
>> handle affinity, but the new server does not.)
> If using the old NFS server helps, then the FHA code for the new server
> will help as well.  Perhaps more, because the default FHA tuning parameters
> have changed somewhat and parallel writes are now possible.
>
> If you want to try out the FHA changes in stable/9, I just MFCed them,
> change 251641.
>
Sure, I will try both 251641 and the old nfsd.

Thanks,