panic: spin lock held too long (RELENG_8 from today)

[Jeremy Chadwick]

On Wed, Aug 17, 2011 at 05:01:05PM -0700, Chip Camden wrote:
> Quoth Jeremy Chadwick on Wednesday, 17 August 2011:
> > > 
> > > I'm also getting similar panics on 8.2-STABLE.  Locks up everything and I
> > > have to power off.  Once, I happened to be looking at the console when it
> > > happened and copied dow the following:
> > > 
> > > Sleeping thread (tif 100037, pid 0) owns a non-sleepable lock
> > > panic: sleeping thread
> > > cpuid=1
> > 
> > No idea, might be relevant to the thread.
> > 
> > > Another time I got:
> > > 
> > > lock order reversal:
> > > 1st 0xffffff000593e330 snaplk (snaplk) @ /usr/src/sys/kern/vfr_vnops.c:296
> > > 2nd 0xffffff0005e5d578 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:1587
> > > 
> > > I didn't copy down the traceback.
> > 
> > "snaplk" refers to UFS snapshots.  The above must have been typed in
> > manually as well, due to some typos in filenames as well.
> > 
> > Either this is a different problem, or if everyone in this thread is
> > doing UFS snapshots (dump -L, mksnap_ffs, etc.) and having this problem
> > happen then I recommend people stop using UFS snapshots.  I've ranted
> > about their unreliability in the past (years upon years ago -- still
> > seems valid) and just how badly they can "wedge" a system.  This is one
> > of the many (MANY!) reasons why we use rsnapshot/rsync instead.  The
> > atime clobbering issue is the only downside.
> > 
> 
> If I'm doing UFS snapshots, I didn't know it.

The backtrace indicates that a UFS snapshot is being made -- which
causes the state to be set to string "snaplk", which is then honoured in
vfs_vnops.c.

You can see for yourself: grep -r snaplk /usr/src/sys.

So yes, I'm inclined to believe something on your system is doing UFS
snapshot generation.  Whether or not other people are doing it as well
is a different story.

> Yes, everything was copied manually because it only displays on the
> console and the keyboard does not respond after that point.  So I
> copied first to paper, then had to decode my lousy handwriting to put
> it in an email.  Sorry for the scribal errors.

That sounds more or less like what I saw with UFS snapshots: the system
would go catatonic in one way or another.  It wouldn't "hard lock" (as
in if you had powered it off, etc.), it would "live lock" (as in the
kernel was wedged or held up/spinning doing something).

I never saw a panic as a result of UFS snapshots, only what I described
here.

TL;DR -- Your system appears to be making UFS snapshots, and that
situation is possibly (likely?) unrelated to the sleeping thread issue
you see that causes a panic.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |