Panic in ffs_blkfree()

[Scott Long]

Kirk and all,

Recently a panic started appearing on mine and other's systems while
under  load.  I have an SMP system with all UFS1+softupdates
filesystems, 256MB of RAM, etc.  I'm working to build a debug kernel and
reproduce the panic so I can give a better stack trace, but until then
this is what I have.  Building a release triggers this pretty
consistently, though it takes a couple of hours.  Before my last kernel
update, I was running from a kernel+world from mid-April with no
problems.

panic: bremfree: removing a buffer not on a queue
panic messages:
---
panic: ffs_blkfree: freeing free block
cpuid = 1; lapic.id = 01000000
boot() called on cpu#1

syncing disks, buffers remaining... panic: bremfree: removing a buffer 
not on a queue
cpuid = 1; lapic.id = 01000000
boot() called on cpu#1
Uptime: 5h52m18s
Dumping 255 MB
  16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
---
#0  0xc01f4edb in doadump ()
(kgdb) bt
#0  0xc01f4edb in doadump ()
#1  0xc01f5438 in boot ()
#2  0xc01f579f in panic ()
#3  0xc0234270 in bremfreel ()
#4  0xc02341a5 in bremfree ()
#5  0xc023d598 in vop_stdfsync ()
#6  0xc01bfb90 in spec_fsync ()
#7  0xc01bf228 in spec_vnoperate ()
#8  0xc02bda5d in ffs_sync ()
#9  0xc02490bb in sync ()
#10 0xc01f5041 in boot ()
#11 0xc01f579f in panic ()
#12 0xc02a59ca in ffs_blkfree ()
#13 0xc02b59bf in indir_trunc ()
#14 0xc02b5455 in handle_workitem_freeblocks ()
#15 0xc02b25ba in process_worklist_item ()
#16 0xc02b22e0 in softdep_process_worklist ()
#17 0xc024593e in sched_sync ()
#18 0xc01e21de in fork_exit ()
(kgdb)