[6.1-PRERELEASE/amd64] Kernel panic during heavy UFS traffic

[Jason Harmening]

I managed to get a backtrace from a panic that occurred during bgfsck:

dev = ar0s1f, block = 22958088, fs = /usr
panic: ffs_blkfree: freeing free frag
cpuid = 0
KDB: stack backtrace:
kdb_backtrace() at kdb_backtrace+0x37
panic() at panic+0x1d1
ffs_blkfree() at ffs_blkfree+0x43f
sysctl_ffs_fsck() at sysctl_ffs_fsck+0x316
sysctl_root() at sysctl_root+0x13f
userland_sysctl() at userland_sysctl+0x131
__sysctl() at __sysctl+0xd8
syscall() at syscall+0x404
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x8006e91fc, rsp = 
0x7ffffffee858, rbp =
 0x3 ---
Uptime: 2m59s
Dumping 1023 MB (2 chunks)


However, I've been unable to get a backtrace for the panics that occur when I 
try to mount my DVD-RAM.  For some reason, I've so far only been able to 
reproduce this panic when trying to mount the disk while the X server is 
running (by clicking the KDE disk icon rather than mounting from the command 
ine).  Because of this, I'd like to be able to have the kernel generate a 
crash dump and automatically reboot.  Without debug symbols or dumping 
enabled, the system will automatically reboot.  However, every time I've 
managed to reproduce the panic WITH a debug kernel and WITH a dumpdev entry 
in /etc/rc.conf, the system simply freezes.  After I reset, there is no dump 
in /var/crash.  Currently I have the following in my kernel configuration:

makeoptions     DEBUG=-g
options         KDB
options         KDB_UNATTENDED

And in /etc/rc.conf:

dumpdev="AUTO"


I've tried rebooting in single-user mode and manually retrieving the dump from 
swap using savecore, but it reports "no dump found".  I have 1GB of physical 
RAM, 2GB of swap, and ~7GB free on /var, so space shouldn't be an issue.  
Explicitly specifying my swap partition "/dev/ar0s1b" instead of "AUTO" in 
rc.conf does not fix the problem, nor does removing everything except the 
"makeoptions" line from the kernel config.  I've also tried various 
combinations of KDB, KDB_UNATTENDED, KDB_TRACE, and DDB in the kernel config. 
Also note that while the above backtrace indicates dumping was done, no dump 
was actually generated.  Is there something I'm missing?


On Thursday 16 March 2006 13:54, Kris Kennaway wrote:
> On Thu, Mar 16, 2006 at 12:45:07PM -0600, Jason Harmening wrote:
> > Last night I ran into a series of kernel panics that seemed to be related
> > to heavy UFS traffic.  I ran into two consecutive panics when trying to
> > mount a UFS-formatted DVD-RAM as a regular user (though not when I
> > mounted it as root).  The system seemed to actually succeed in mounting
> > the disk, as it was marked dirty after the ensuing panic.  Upon rebooting
> > after the second panic, I saw another two consecutive panics which
> > happened whenever I tried to do something fairly disk-intensive (e.g.
> > starting the X server + KDE) while the bgfsck was still running from the
> > last panic.  Ultimately I rebooted in single-user mode, ran fsck
> > manually, and have experienced no further panics.  I suspect these panics
> > may be related to UFS deadlocks, as in all cases the application that was
> > attempting disk access hung for several seconds before the panic,
> > followed by a few seconds of total system hang, followed by the automatic
> > reboot.
> >
> > I'm running 6.1-PRELEASE/amd64 from 12 March on an Athlon 64 x2 (SMP)
> > with SCHED_ULE+PREEMPTION--dangerous combination I know, but it's been
> > rock solid for months until now.  If anyone is interested, I'll try to
> > reproduce this panic with a dump/backtrace.  It may be one of the UFS
> > deadlock issues that's already under investigation for 6.1-RELEASE.
>
> Yeah, we need a trace.
>
> Kris