[6.1-PRERELEASE/amd64] Kernel panic during heavy UFS traffic

[Kris Kennaway]

On Sat, Mar 18, 2006 at 05:12:12PM -0600, Jason Harmening wrote:
> I managed to get a backtrace from a panic that occurred during bgfsck:
> 
> dev = ar0s1f, block = 22958088, fs = /usr
> panic: ffs_blkfree: freeing free frag
> cpuid = 0
> KDB: stack backtrace:
> kdb_backtrace() at kdb_backtrace+0x37
> panic() at panic+0x1d1
> ffs_blkfree() at ffs_blkfree+0x43f
> sysctl_ffs_fsck() at sysctl_ffs_fsck+0x316
> sysctl_root() at sysctl_root+0x13f
> userland_sysctl() at userland_sysctl+0x131
> __sysctl() at __sysctl+0xd8
> syscall() at syscall+0x404
> Xfast_syscall() at Xfast_syscall+0xa8
> --- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x8006e91fc, rsp = 
> 0x7ffffffee858, rbp =
>  0x3 ---
> Uptime: 2m59s
> Dumping 1023 MB (2 chunks)

Thanks, I saw this panic a few times too but not after subsequent
updates.  Since you seem to be able to reproduce this easily, can you
try to update to BETA4 or later and see if it's still a problem?

> However, I've been unable to get a backtrace for the panics that occur when I 
> try to mount my DVD-RAM.  For some reason, I've so far only been able to 
> reproduce this panic when trying to mount the disk while the X server is 
> running (by clicking the KDE disk icon rather than mounting from the command 
> ine).  Because of this, I'd like to be able to have the kernel generate a 
> crash dump and automatically reboot.  Without debug symbols or dumping 
> enabled, the system will automatically reboot.  However, every time I've 
> managed to reproduce the panic WITH a debug kernel and WITH a dumpdev entry 
> in /etc/rc.conf, the system simply freezes.  After I reset, there is no dump 
> in /var/crash.  Currently I have the following in my kernel configuration:

Yeah, panics while in X often don't fare very well.  Unfortunately
there's no real solution other than trying to reproduce from the
console (note that you can still run X applications from a vty by
setting DISPLAY, although I don't know if you can trigger the KDE
mounting from the command line).  Perhaps you'll need to figure out
what KDE is doing (e.g. is it mounting with special flags?) and repeat
it by hand.

> I've tried rebooting in single-user mode and manually retrieving the dump from 
> swap using savecore, but it reports "no dump found".  I have 1GB of physical 
> RAM, 2GB of swap, and ~7GB free on /var, so space shouldn't be an issue.  
> Explicitly specifying my swap partition "/dev/ar0s1b" instead of "AUTO" in 
> rc.conf does not fix the problem, nor does removing everything except the 
> "makeoptions" line from the kernel config.  I've also tried various 
> combinations of KDB, KDB_UNATTENDED, KDB_TRACE, and DDB in the kernel config. 
> Also note that while the above backtrace indicates dumping was done, no dump 
> was actually generated.  Is there something I'm missing?

savecore runs after bg fsck, so are you sure it's not there?

Kris

P.S. Don't top-post, it confuses the logical flow of the email and
loses context.

> On Thursday 16 March 2006 13:54, Kris Kennaway wrote:
> > On Thu, Mar 16, 2006 at 12:45:07PM -0600, Jason Harmening wrote:
> > > Last night I ran into a series of kernel panics that seemed to be related
> > > to heavy UFS traffic.  I ran into two consecutive panics when trying to
> > > mount a UFS-formatted DVD-RAM as a regular user (though not when I
> > > mounted it as root).  The system seemed to actually succeed in mounting
> > > the disk, as it was marked dirty after the ensuing panic.  Upon rebooting
> > > after the second panic, I saw another two consecutive panics which
> > > happened whenever I tried to do something fairly disk-intensive (e.g.
> > > starting the X server + KDE) while the bgfsck was still running from the
> > > last panic.  Ultimately I rebooted in single-user mode, ran fsck
> > > manually, and have experienced no further panics.  I suspect these panics
> > > may be related to UFS deadlocks, as in all cases the application that was
> > > attempting disk access hung for several seconds before the panic,
> > > followed by a few seconds of total system hang, followed by the automatic
> > > reboot.
> > >
> > > I'm running 6.1-PRELEASE/amd64 from 12 March on an Athlon 64 x2 (SMP)
> > > with SCHED_ULE+PREEMPTION--dangerous combination I know, but it's been
> > > rock solid for months until now.  If anyone is interested, I'll try to
> > > reproduce this panic with a dump/backtrace.  It may be one of the UFS
> > > deadlock issues that's already under investigation for 6.1-RELEASE.
> >
> > Yeah, we need a trace.
> >
> > Kris
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-bugs/attachments/20060318/a53af6ab/attachment.pgp