Performance difference between UFS and ZFS with NFS

Tue Nov 19 19:38:36 UTC 2013

Wow, those are great mount options, I use em too :)

Well, this is very interesting on the +3x access/getattrs with ZFS.

I'll report back my findings as I'm going down a similar road, albeit not home dirs but rendering using AE and C4D on many clients.

Until then hoping some one chime in on this with some added nuggets.

- aurf

On Nov 19, 2013, at 11:11 AM, Eric Browning wrote:

> Locking is set to locallocks, cache folders and similar folders are redirected to the local hard drive.  All applications run just fine including Adobe CS6 and MS 2011 apps.
> 
> This is my client NFS conf:
> nfs.client.mount.options = noatime,nobrowse,tcp,vers=3,rsize=32768,wsize=32768,readahead=0,acregmax=3600,acdirmax=3600,locallocks,inet,noquota,nfc
> nfs.client.statfs_rate_limit = 5
> nfs.client.access_for_getattr = 1
> nfs.client.is_mobile = 0
> 
> I'm sure this is more complex than it needs to be and I can probably get rid of most of this now, forcing nfc did cure some unicode issues between mac and freebsd. Packets are not being fragmented and there are only one or two errors here and there despite traversing vlans through the core router, MSS is set at 1460.
> 
> One thing Rick M suggested is actually trying these entire setup on a UFS system.  I tested by copying my home folder to another server with a UFS system and ran it for like 45 minutes and compared it to another 45 minute jaunt on the main file server and I had about 3x less Access and Getattrs on UFS than I had on ZFS.  Seeing this prompted me to move one server over to a UFS raid and since doing that it's like day and night performance-wise. 
> 
> Server's NFS is set to 256 threads ARC is currently only at 46G of 56G total and NFS is 9.9G on the ZFS server and CPU usage is 878%.  On the UFS server NFS is the same 256 threads and 9.9G but as I look at it with currently 52 users logged in NFS is at CPU 0.00% usage.
> 
> This is the server NFS configs from rc.conf
> ## NFS Server
> rpcbind_enable="YES"
> nfs_server_enable="YES"
> mountd_flags="-r -l"
> nfsd_enable="YES"
> mountd_enable="YES"
> rpc_lockd_enable="NO"
> rpc_statd_enable="NO"
> nfs_server_flags="-t -n 256"
> nfsv4_server_enable="NO"
> nfsuserd_enable="YES"
> 
> UFS Server mem stats:
> Mem: 49M Active, 56G Inact, 3246M Wired, 1434M Cache, 1654M Buf, 1002M Free
> ARC: 1884K Total, 149K MFU, 1563K MRU, 16K Anon, 56K Header, 99K Other
> Swap: 4096M Total, 528K Used, 4095M Free
> 
> ZFS mem stats:
> Mem: 3180K Active, 114M Inact, 60G Wired, 1655M Buf, 2412M Free
> ARC: 46G Total, 26G MFU, 13G MRU, 3099K Anon, 4394M Header, 4067M Other
> Swap: 4096M Total, 4096M Free
> 
> 
> 
> On Tue, Nov 19, 2013 at 11:25 AM, aurfalien <aurfalien at gmail.com> wrote:
> Curious.
> 
> Do you have NFS locking enabled client side?
> 
> Most likely you do as Mac Mail will not run w/o locks, nor will Adobe prefs like temp cache. etc...
> 
> So being this is prolly the case, could it be a mem pressure issue and not enough RAM?
> 
> So NFS locks take up RAM as does ARC.  What are your mem stats and swap stats during the 700% (yikes) experience?
> 
> - aurf
> 
> On Nov 19, 2013, at 10:19 AM, Eric Browning wrote:
> 
>> Aurf,
>> 
>> I ran those two commands and it doesn't seem to have made a difference.  Usage is still above 700% and it still takes 30s to list a directory.  The time to list is proportional to the number of users logged in.  On UFS with all students logged in and hammering away at their files there is no noticeable speed decrease.
>> 
>> 
>> On Tue, Nov 19, 2013 at 11:12 AM, aurfalien <aurfalien at gmail.com> wrote:
>> 
>> On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote:
>> 
>> > Eric Browning wrote:
>> >> Some background:
>> >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ 3Ghz,
>> >> -64GB ram each server
>> >> -Four Intel DC S3700 800GB SSDs for primary storage, each server.
>> >> -FreeBSD 9 stable as of 902503
>> >> -ZFS v28 and later updated to feature flags (v29?)
>> >> -LSI 9200-8i controller
>> >> -Intel I350T4 nic (only one port being used currently) using all four
>> >> in
>> >> LACP overtaxed the server's NFS queue from what we found out making
>> >> the
>> >> server basically unusable.
>> >>
>> >> There is definitely something going on between NFS and ZFS when used
>> >> as a
>> >> file server (random workload) for mac home directories.  They do not
>> >> jive
>> >> well at all and pretty much drag down these beefy servers and cause
>> >> 20-30
>> >> second delays when just attempting to list a directory on Mac 10.7,
>> >> 10.8
>> >> clients although throughput seems fast when copying files.
>> >>
>> >> This server's NFS was sitting north of 700% (7+ cores) all day long
>> >> when
>> >> using ZFSv28 raidz1. I have also tried stripe, compression on/off,
>> >> sync
>> >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC.
>> >> I've
>> >> tried just 100% stock settings in loader.conf and and some
>> >> recommended
>> >> tuning from various sources on the freebsd lists and other sites
>> >> including
>> >> the freebsd handbook.
>> >>
>> >> This is my mountpoint creation:
>> >> zfs create -o mountpoint=/users -o sharenfs=on -o
>> >> casesensitivity=insensitive -o aclmode=passthrough -o compression=lz4
>> >> -o
>> >> atime=off -o aclinherit=passthrough tank/users
>> >>
>> >> This last weekend I switched one of these servers over to a UFS raid
>> >> 0
>> >> setup and NFS now only eats about 36% of one core during the initial
>> >> login
>> >> phase of 150-ish users over about 10 minutes and sits under 1-3%
>> >> during
>> >> normal usage and directories all list instantly even when drilling
>> >> down 10
>> >> or so directories on the client's home files. The same NFS config on
>> >> server
>> >> and clients are still active.
>> >>
>> >> Right now I'm going to have to abandon ZFS until it works with NFS.
>> >> I
>> >> don't want to get into a finger pointing game, I'd just like to help
>> >> get
>> >> this fixed, I have one old i386 server I can try things out on if
>> >> that
>> >> helps and it's already on 9 stable and ZFS v28.
>> >>
>> > Btw, in previous discussions with Eric on this, he provided nfsstat
>> > output that seemed to indicate most of his RPC load from the Macs
>> > were Access and Getattr RPCs.
>> >
>> > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a
>> > significant part of this issue. I know nothing about ZFS, but I believe
>> > it does always have ACLs enabled and presumably needs to check the
>> > ACL for each VOP_ACCESSX().
>> >
>> > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and
>> > VOP_GETATTR() can look at these?
>> 
>> Indeed.  However couldn't one simply disable ACL mode via;
>> 
>> zfs set aclinherit=discard pool/dataset
>> zfs set aclmode=discard pool/dataset
>> 
>> Eric, mind setting these and see?
>> 
>> Mid/late this week I'll be doing a rather large render farm test amongst our Mac fleet against ZFS.
>> 
>> Will reply to this thread with outcome when I'm done.  Should be interesting.
>> 
>> - aurf
>> 
>> >
>> > rick
>> >
>> >> Thanks,
>> >> --
>> >> Eric Browning
>> >> Systems Administrator
>> >> 801-984-7623
>> >>
>> >> Skaggs Catholic Center
>> >> Juan Diego Catholic High School
>> >> Saint John the Baptist Middle
>> >> Saint John the Baptist Elementary
>> >> _______________________________________________
>> >> freebsd-fs at freebsd.org mailing list
>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>> >>
>> > _______________________________________________
>> > freebsd-fs at freebsd.org mailing list
>> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>> 
>> 
>> 
>> 
>> -- 
>> Eric Browning
>> Systems Administrator
>> 801-984-7623
>> 
>> Skaggs Catholic Center
>> Juan Diego Catholic High School
>> Saint John the Baptist Middle
>> Saint John the Baptist Elementary
> 
> 
> 
> 
> -- 
> Eric Browning
> Systems Administrator
> 801-984-7623
> 
> Skaggs Catholic Center
> Juan Diego Catholic High School
> Saint John the Baptist Middle
> Saint John the Baptist Elementary