ZFS server has gone crazy slow

[Peter Eriksson]

Yes, this is a know “feature” of ZFS. When filesystems are nearly full (or near quota limits) it will tune down the write transaction sizes which causes I/O (in order to make sure it doesn’t write more than the quota, and try to opimize for storage instead of latency/speed) to inflate and things basically grinds to a halt… 

Another fun thing that might happen is if you reboot your server and happen to have a lot of queued up writes in the ZIL (for example if you did a “zfs destroy -d -r POOL at snapshots” (deferred(background) destroys of snapshots) and do a hard reboot while it’s busy it will “write out” those queued transactions at filesystem mount time during the boot sequence - which may take a long time (many hours if your have big pools and some of them have filesystems (datasets) that are nearly full - ours pools are 100-400TB). Taking snapshots will also grind to a crawl if you are near quota limits on filesystems. Fun fun :-)

(We have modified our rc.d startup scripts so we do the “zfs mount -a” part in the background so the rest of the system doesn’t have to wait for it (so we can get a ssh login prompt and login and check the progress of the filesystem mounts. (And then “nfs exports”, nfsd and samba also waits in the background for that to finish before starting up. A bit of a hack but it works (a real parallel server manager with dependencies would have been better :-)

- Peter

> On 11 Apr 2020, at 21:26, Chris Ross <cross+freebsd at distal.com> wrote:
> 
> 
> 
>> On Apr 11, 2020, at 14:10, Ireneusz Pluta/wp.pl <ipluta at wp.pl> wrote:
>> W dniu 2020-04-11 o 20:02, Chris Ross pisze:
>>> And, I’m not sure how to check SMART values.  This is a Dell server, and I’m 95% sure I have it’s RAID controller (Dell PERC 6, mfi0) just set to provide JBOD for the two disks.  Will the smartmontools package be useful with disks on an mfi controller?
>> Does the controller expose these jbods as /dev/da? devices to the system ?
> 
> mfid devices.  And, googling showed that I might want to use mfip to allow me to access the drives for smartctl, and I’ll work on that more in a bit..
> 
> But, Eugene was kind enough to point out something that should’ve been obvious, that I’d filled my ZFS volumes.  I had a snapshot of everything from a while ago, and one of the filesystems contains large data files.  Deleting the snapshot on that volume freed things back to functional.
> 
> Thanks all.
> 
>      - Chris
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"