Recurring problem: processes block accessing UFS file system

[Kris Kennaway]

On Mon, Nov 21, 2005 at 05:54:09PM -0600, Greg Rivers wrote:
> I've recently put up three busy email relay hosts running 6.0-STABLE. 
> Performance is excellent except for a nagging critical issue that keeps 
> cropping up.
> 
> /var/spool is its own file system mounted on a geom stripe of four BSD 
> partitions (details below).  Once every two or three days all the 
> processes accessing /var/spool block forever in disk wait.  All three 
> machines suffer this problem.  No diagnostic messages are generated and 
> the machines continue running fine otherwise, but a reboot is required to 
> clear the condition.  This problem occurs during normal operation, but is 
> particularly likely to occur during a backup when dump makes a snapshot.
> 
> There doesn't appear to be a problem with gstripe, as gstripe status is 
> "UP" and I can read the raw device just fine while processes continue to 
> block on the file system.  I tried running a kernel with WITNESS and 
> DIAGNOSTIC, but these options shed no light.
> 
> If I catch the problem early enough I can break successfully into kdb; 
> otherwise, if too many processes stack up, the machine hangs going into 
> kdb and must be power-cycled.

Make sure you have KDB_STOP_NMI in your kernel.

> I obtained the following process listing and traces from kdb.  I traced
> mksnap_ffs which was blocked in "ufs", and two random sendmail processes
> that were blocked in "ufs" and "suspfs" respectively.

Looks like a UFS snapshot deadlock.  Are you running something like
dump -L on this filesystem, or making other use of snapshots?  fsck -B
also uses them, but shouldn't be running except at boot time.

You should take this up with Kirk McKusick <freebsd at McKusick.COM> - in
the meantime you can work around it by not making use of UFS
snapshots.

Kris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20051121/9ed4981f/attachment.bin