``Fatal double fault'' when running nightly jobs, perhaps ZFS-related

[Tim Stewart]

Hello,

I have a FreeBSD 8.2-RELEASE-p2 system that uses ZFS, including booting. 
  The kernel is locally compiled and is just the GENERIC configuration 
with DTrace enabled and a MFI driver patch from the mailing list post at 
[1].

The system panicked with a ``Fatal double fault'' at 3:09 AM one night 
(typed in manually from a screenshot, but proofread):


Fatal double fault
rip = 0xffffffff805d3eeb
rsp = 0xffffff848585b000
rbp = 0xffffff848585b020
cpuid = 0; apic id = 20
panic: double fault
cpuid = 0
KDB: stack backtrace
#0 0xffffffff80618d3e at kdb_backtrace+0x5e
#1 0xffffffff805e4d47 at panic+0x187
#2 0xffffffff808dc834 at dblfault_handler+0xa4
#3 0xffffffff808c53ad at Xdblfault+0xad
Uptime: 3d11h11m37s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
Sleeping thread (tid 100141, pid 5) owns a non-sleepable lock


It seems likely that the activity that prompted the fault is related to 
the nightly periodic jobs.  Others have suggested that it may be related 
to /etc/periodic/security/100.chksetuid (see [2] and [3]), as it does a 
find on every filesystem not marked as `nosuid.'  Indeed, I have a ZFS 
dataset containing 71+ million files that (at the time) was not marked 
as nosuid.

I have not been able to replicate the issue since and it has only 
happened once.  I have set `setuid=off' on the large ZFS dataset so that 
100.chksetuid will no longer traverse it every night.

Any clue as to what is happening here?  I don't have a kernel core dump 
since I'm using ZFS for swap, though I can change this if it helps 
troubleshoot the issue in the event of another crash.

Thanks for any help you can provide,

-- 
-TimS
Tim Stewart


References:

[1] http://lists.freebsd.org/pipermail/freebsd-scsi/2011-March/004839.html

[2] http://lists.freebsd.org/pipermail/freebsd-bugs/2011-March/043781.html

[3] http://forums.freebsd.org/showthread.php?t=23919