Regarding regular zfs

[Adam McDougall]

On 04/05/13 06:17, Joar Jegleim wrote:
> Hi FreeBSD !
> 
> I've already sent this one to questions at freebsd.org, but realised this list
> would be a better option.
> 
> So I've got this setup where we have a storage server delivering about
> 2 million jpeg's as a backend for a website ( it's ~1TB of data)
> The storage server is running zfs and every 15 minutes it does a zfs
> send to a 'slave', and our proxy will fail over to the slave if the
> main storage server goes down .
> I've got this script that initially zfs send's a whole zfs volume, and
> for every send after that only sends the diff . So after the initial zfs
> send, the diff's usually take less than a minute to send over.
> 
> I've had increasing problems on the 'slave', it seem to grind to a
> halt for anything between 5-20 seconds after every zfs receive . Everything
> on the server halts / hangs completely.
> 
> I've had a couple go's on trying to solve / figure out what's
> happening without luck, and this 3rd time I've invested even more time
> on the problem .
> 
> To sum it up:
> -Server was initially on 8.2-RELEASE
> -I've set some sysctl variables such as:
> 
> # 16GB arc_max ( server got 30GB of ram, but had a couple 'freeze'
> situations, suspect zfs.arc ate too much memory)
> vfs.zfs.arc_max=17179869184

What is vm.kmem_size?  I suggest setting it to 2x physical ram (60G) but
higher won't hurt because recent-ish kernels will properly cap it at 2x.
 This will allow more space in kmem to reduce ARC fragmentation in
memory which can cause stalls or panics.  Also, I would set
vfs.zfs.arc_max almost as high as your physical memory (or not at all)
unless you need to reserve some for non-zfs usage.  With ZFS from the
latest year or two, I generally don't adjust anything other than making
sure vm.kmem_size is sufficiently high on all systems, and some systems
I use arc_max.  Rarely arc_min but in an attempt to encourage ARC use.
By avoiding ARC fragmentation, I generally feel more ARC is better than
less.

Also keep an eye on kstat.zfs.misc.arcstats.size when you run zfs recv
to see how it behaves.

> 
> # 8.2 default to 30 here, setting it to 5 which is default from 8.3 and
> onwards
> vfs.zfs.txg.timeout="5"
> 
> # Set TXG write limit to a lower threshold.  This helps "level out"
> # the throughput rate (see "zpool iostat").  A value of 256MB works well
> # for systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on
> # disks which have 64 MB cache. <<BR>>
> # NOTE: in <v28, this tunable is called 'vfs.zfs.txg.write_limit_override'.
> #vfs.zfs.txg.write_limit_override=1073741824 # for 8.2
> vfs.zfs.write_limit_override=1073741824 # for 8.3 and above
> 
> -I've implemented mbuffer for the zfs send / receive operations. With
> mbuffer the sync went a lot faster, but still got the same symptoms
> when the zfs receive is done, the hang / unresponsiveness returns for
> 5-20 seconds
> -I've upgraded to 8.3-RELEASE ( + zpool upgrade and zfs upgrade to
> V28), same symptoms
> -I've upgraded to 9.1-RELEASE, still same symptoms
> 

I think it is good to be on 9.x to get ZFS fixes before 8.