8.1R ZFS almost locking up system

Tim Bishop tim at bishnet.net
Mon Aug 23 23:37:20 UTC 2010


On Sat, Aug 21, 2010 at 05:24:29PM -0500, Dan Nelson wrote:
> In the last episode (Aug 21), Tim Bishop said:
> > I've had a problem on a FreeBSD 8.1R system for a few weeks. It seems
> > that ZFS gets in to an almost unresponsive state. Last time it did it
> > (two weeks ago) I couldn't even log in, although the system was up, this
> > time I could manage a reboot but couldn't stop any applications (they
> > were likely hanging on I/O).
> 
> Could your pool be very close to full?  Zfs will throttle itself when it's
> almost out of disk space.  I know it's "saved" me from filling up my
> filesystems a couple times :)

It's not close to full, so I don't think that's the issue.

> > A few items from top, including zfskern:
> > 
> >   PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
> >     5 root        4  -8    -     0K    60K zio->i  0  54:38  3.47% zfskern
> > 91775     70      1  44    0 53040K 31144K tx->tx  1   2:11  0.00% postgres
> > 39661 tdb         1  44    0 55776K 32968K tx->tx  0   0:39  0.00% mutt
> > 14828 root        1  47    0 14636K  1572K tx->tx  1   0:03  0.00% zfs
> > 11188 root        1  51    0 14636K  1572K tx->tx  0   0:03  0.00% zfs
> > 
> > At some point during this process my zfs snapshots have been failing to
> > complete:
> > 
> > root        5  0.8  0.0     0    60  ??  DL    7Aug10  54:43.83 [zfskern]
> > root     8265  0.0  0.0 14636  1528  ??  D    10:00AM   0:03.12 zfs snapshot -r pool0 at 2010-08-21_10:00:01--1d
> > root    11188  0.0  0.1 14636  1572  ??  D    11:00AM   0:02.93 zfs snapshot -r pool0 at 2010-08-21_11:00:01--1d
> > root    14828  0.0  0.1 14636  1572  ??  D    12:00PM   0:03.04 zfs snapshot -r pool0 at 2010-08-21_12:00:00--1d
> > root    17862  0.0  0.1 14636  1572  ??  D     1:00PM   0:01.96 zfs snapshot -r pool0 at 2010-08-21_13:00:01--1d
> > root    20986  0.0  0.1 14636  1572  ??  D     2:00PM   0:02.07 zfs snapshot -r pool0 at 2010-08-21_14:00:01--1d
> 
> procstat -k on some of these processes might help to pinpoint what part of
> the zfs code they're all waiting in.

I'll do that. Thanks for the pointer :-)

Tim.

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x5AE7D984


More information about the freebsd-stable mailing list