8.1R ZFS almost locking up system

[Dan Nelson]

In the last episode (Aug 21), Tim Bishop said:
> I've had a problem on a FreeBSD 8.1R system for a few weeks. It seems
> that ZFS gets in to an almost unresponsive state. Last time it did it
> (two weeks ago) I couldn't even log in, although the system was up, this
> time I could manage a reboot but couldn't stop any applications (they
> were likely hanging on I/O).

Could your pool be very close to full?  Zfs will throttle itself when it's
almost out of disk space.  I know it's "saved" me from filling up my
filesystems a couple times :)

> A few items from top, including zfskern:
> 
>   PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
>     5 root        4  -8    -     0K    60K zio->i  0  54:38  3.47% zfskern
> 91775     70      1  44    0 53040K 31144K tx->tx  1   2:11  0.00% postgres
> 39661 tdb         1  44    0 55776K 32968K tx->tx  0   0:39  0.00% mutt
> 14828 root        1  47    0 14636K  1572K tx->tx  1   0:03  0.00% zfs
> 11188 root        1  51    0 14636K  1572K tx->tx  0   0:03  0.00% zfs
> 
> At some point during this process my zfs snapshots have been failing to
> complete:
> 
> root        5  0.8  0.0     0    60  ??  DL    7Aug10  54:43.83 [zfskern]
> root     8265  0.0  0.0 14636  1528  ??  D    10:00AM   0:03.12 zfs snapshot -r pool0 at 2010-08-21_10:00:01--1d
> root    11188  0.0  0.1 14636  1572  ??  D    11:00AM   0:02.93 zfs snapshot -r pool0 at 2010-08-21_11:00:01--1d
> root    14828  0.0  0.1 14636  1572  ??  D    12:00PM   0:03.04 zfs snapshot -r pool0 at 2010-08-21_12:00:00--1d
> root    17862  0.0  0.1 14636  1572  ??  D     1:00PM   0:01.96 zfs snapshot -r pool0 at 2010-08-21_13:00:01--1d
> root    20986  0.0  0.1 14636  1572  ??  D     2:00PM   0:02.07 zfs snapshot -r pool0 at 2010-08-21_14:00:01--1d

procstat -k on some of these processes might help to pinpoint what part of
the zfs code they're all waiting in.

-- 
	Dan Nelson
	dnelson at allantgroup.com