Performance difference between UFS and ZFS with NFS
Eric Browning
ericbrowning at skaggscatholiccenter.org
Thu Nov 21 23:04:38 UTC 2013
Just as a bit of a followup I had 163 kids all logged in at once today and
nfsd usage was only 1-5%
@Aurf
How are your results with your AE and C4D clients going?
On Tue, Nov 19, 2013 at 12:38 PM, aurfalien <aurfalien at gmail.com> wrote:
> Wow, those are great mount options, I use em too :)
>
> Well, this is very interesting on the +3x access/getattrs with ZFS.
>
> I'll report back my findings as I'm going down a similar road, albeit not
> home dirs but rendering using AE and C4D on many clients.
>
> Until then hoping some one chime in on this with some added nuggets.
>
> - aurf
>
> On Nov 19, 2013, at 11:11 AM, Eric Browning wrote:
>
> Locking is set to locallocks, cache folders and similar folders are
> redirected to the local hard drive. All applications run just fine
> including Adobe CS6 and MS 2011 apps.
>
> This is my client NFS conf:
> nfs.client.mount.options =
> noatime,nobrowse,tcp,vers=3,rsize=32768,wsize=32768,readahead=0,acregmax=3600,acdirmax=3600,locallocks,inet,noquota,nfc
> nfs.client.statfs_rate_limit = 5
> nfs.client.access_for_getattr = 1
> nfs.client.is_mobile = 0
>
> I'm sure this is more complex than it needs to be and I can probably get
> rid of most of this now, forcing nfc did cure some unicode issues between
> mac and freebsd. Packets are not being fragmented and there are only one or
> two errors here and there despite traversing vlans through the core router,
> MSS is set at 1460.
>
> One thing Rick M suggested is actually trying these entire setup on a UFS
> system. I tested by copying my home folder to another server with a UFS
> system and ran it for like 45 minutes and compared it to another 45 minute
> jaunt on the main file server and I had about 3x less Access and Getattrs
> on UFS than I had on ZFS. Seeing this prompted me to move one server over
> to a UFS raid and since doing that it's like day and night
> performance-wise.
>
> Server's NFS is set to 256 threads ARC is currently only at 46G of 56G
> total and NFS is 9.9G on the ZFS server and CPU usage is 878%. On the UFS
> server NFS is the same 256 threads and 9.9G but as I look at it with
> currently 52 users logged in NFS is at CPU 0.00% usage.
>
> This is the server NFS configs from rc.conf
> ## NFS Server
> rpcbind_enable="YES"
> nfs_server_enable="YES"
> mountd_flags="-r -l"
> nfsd_enable="YES"
> mountd_enable="YES"
> rpc_lockd_enable="NO"
> rpc_statd_enable="NO"
> nfs_server_flags="-t -n 256"
> nfsv4_server_enable="NO"
> nfsuserd_enable="YES"
>
> UFS Server mem stats:
> Mem: 49M Active, 56G Inact, 3246M Wired, 1434M Cache, 1654M Buf, 1002M Free
> ARC: 1884K Total, 149K MFU, 1563K MRU, 16K Anon, 56K Header, 99K Other
> Swap: 4096M Total, 528K Used, 4095M Free
>
> ZFS mem stats:
> Mem: 3180K Active, 114M Inact, 60G Wired, 1655M Buf, 2412M Free
> ARC: 46G Total, 26G MFU, 13G MRU, 3099K Anon, 4394M Header, 4067M Other
> Swap: 4096M Total, 4096M Free
>
>
>
> On Tue, Nov 19, 2013 at 11:25 AM, aurfalien <aurfalien at gmail.com> wrote:
>
>> Curious.
>>
>> Do you have NFS locking enabled client side?
>>
>> Most likely you do as Mac Mail will not run w/o locks, nor will Adobe
>> prefs like temp cache. etc...
>>
>> So being this is prolly the case, could it be a mem pressure issue and
>> not enough RAM?
>>
>> So NFS locks take up RAM as does ARC. What are your mem stats and swap
>> stats during the 700% (yikes) experience?
>>
>> - aurf
>>
>> On Nov 19, 2013, at 10:19 AM, Eric Browning wrote:
>>
>> Aurf,
>>
>> I ran those two commands and it doesn't seem to have made a difference.
>> Usage is still above 700% and it still takes 30s to list a directory. The
>> time to list is proportional to the number of users logged in. On UFS with
>> all students logged in and hammering away at their files there is no
>> noticeable speed decrease.
>>
>>
>> On Tue, Nov 19, 2013 at 11:12 AM, aurfalien <aurfalien at gmail.com> wrote:
>>
>>>
>>> On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote:
>>>
>>> > Eric Browning wrote:
>>> >> Some background:
>>> >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ 3Ghz,
>>> >> -64GB ram each server
>>> >> -Four Intel DC S3700 800GB SSDs for primary storage, each server.
>>> >> -FreeBSD 9 stable as of 902503
>>> >> -ZFS v28 and later updated to feature flags (v29?)
>>> >> -LSI 9200-8i controller
>>> >> -Intel I350T4 nic (only one port being used currently) using all four
>>> >> in
>>> >> LACP overtaxed the server's NFS queue from what we found out making
>>> >> the
>>> >> server basically unusable.
>>> >>
>>> >> There is definitely something going on between NFS and ZFS when used
>>> >> as a
>>> >> file server (random workload) for mac home directories. They do not
>>> >> jive
>>> >> well at all and pretty much drag down these beefy servers and cause
>>> >> 20-30
>>> >> second delays when just attempting to list a directory on Mac 10.7,
>>> >> 10.8
>>> >> clients although throughput seems fast when copying files.
>>> >>
>>> >> This server's NFS was sitting north of 700% (7+ cores) all day long
>>> >> when
>>> >> using ZFSv28 raidz1. I have also tried stripe, compression on/off,
>>> >> sync
>>> >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC.
>>> >> I've
>>> >> tried just 100% stock settings in loader.conf and and some
>>> >> recommended
>>> >> tuning from various sources on the freebsd lists and other sites
>>> >> including
>>> >> the freebsd handbook.
>>> >>
>>> >> This is my mountpoint creation:
>>> >> zfs create -o mountpoint=/users -o sharenfs=on -o
>>> >> casesensitivity=insensitive -o aclmode=passthrough -o compression=lz4
>>> >> -o
>>> >> atime=off -o aclinherit=passthrough tank/users
>>> >>
>>> >> This last weekend I switched one of these servers over to a UFS raid
>>> >> 0
>>> >> setup and NFS now only eats about 36% of one core during the initial
>>> >> login
>>> >> phase of 150-ish users over about 10 minutes and sits under 1-3%
>>> >> during
>>> >> normal usage and directories all list instantly even when drilling
>>> >> down 10
>>> >> or so directories on the client's home files. The same NFS config on
>>> >> server
>>> >> and clients are still active.
>>> >>
>>> >> Right now I'm going to have to abandon ZFS until it works with NFS.
>>> >> I
>>> >> don't want to get into a finger pointing game, I'd just like to help
>>> >> get
>>> >> this fixed, I have one old i386 server I can try things out on if
>>> >> that
>>> >> helps and it's already on 9 stable and ZFS v28.
>>> >>
>>> > Btw, in previous discussions with Eric on this, he provided nfsstat
>>> > output that seemed to indicate most of his RPC load from the Macs
>>> > were Access and Getattr RPCs.
>>> >
>>> > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a
>>> > significant part of this issue. I know nothing about ZFS, but I believe
>>> > it does always have ACLs enabled and presumably needs to check the
>>> > ACL for each VOP_ACCESSX().
>>> >
>>> > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and
>>> > VOP_GETATTR() can look at these?
>>>
>>> Indeed. However couldn't one simply disable ACL mode via;
>>>
>>> zfs set aclinherit=discard pool/dataset
>>> zfs set aclmode=discard pool/dataset
>>>
>>> Eric, mind setting these and see?
>>>
>>> Mid/late this week I'll be doing a rather large render farm test amongst
>>> our Mac fleet against ZFS.
>>>
>>> Will reply to this thread with outcome when I'm done. Should be
>>> interesting.
>>>
>>> - aurf
>>>
>>> >
>>> > rick
>>> >
>>> >> Thanks,
>>> >> --
>>> >> Eric Browning
>>> >> Systems Administrator
>>> >> 801-984-7623
>>> >>
>>> >> Skaggs Catholic Center
>>> >> Juan Diego Catholic High School
>>> >> Saint John the Baptist Middle
>>> >> Saint John the Baptist Elementary
>>> >> _______________________________________________
>>> >> freebsd-fs at freebsd.org mailing list
>>> >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>>> >>
>>> > _______________________________________________
>>> > freebsd-fs at freebsd.org mailing list
>>> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>>>
>>>
>>
>>
>> --
>> Eric Browning
>> Systems Administrator
>> 801-984-7623
>>
>> Skaggs Catholic Center
>> Juan Diego Catholic High School
>> Saint John the Baptist Middle
>> Saint John the Baptist Elementary
>>
>>
>>
>
>
> --
> Eric Browning
> Systems Administrator
> 801-984-7623
>
> Skaggs Catholic Center
> Juan Diego Catholic High School
> Saint John the Baptist Middle
> Saint John the Baptist Elementary
>
>
>
--
Eric Browning
Systems Administrator
801-984-7623
Skaggs Catholic Center
Juan Diego Catholic High School
Saint John the Baptist Middle
Saint John the Baptist Elementary
More information about the freebsd-fs
mailing list