FreeBSD 7.0-STABLE Jul 23: panic: ffs_blkfree: freeing free frag

Scott Lambert lambert at lambertfam.org
Wed Nov 26 11:32:04 PST 2008


On Mon, Nov 24, 2008 at 10:31:24PM -0600, Scott Lambert wrote:
> I have a box I am using for hosting jailed web servers.  I did a
> test move of a jail from a FreeBSD 6 box to the FreeBSD 7 server,
> web1.hosting.  It took forever, 30 minutes to be exact, to create the
> jail with the 3GB image file and restore the data from the FreeBSD 6 box
> into it.
> 
> I created a test archive of the running jail with ezjail-admin on the
> FreeBSD 6 box and scp'd it to web1.hosting.  That took 5 minutes.  (I
> was timing all of this to estimate how long it would really take later.)
> 
> Once the archive was on web1.hosting, I created the new jail using the
> archive to populate it.
> 
> sudo ezjail-admin create -a test_host_tcworks_net-200811241856.40.tar.gz \
>       -s 3G -i testhost.tcworks.net 192.168.1.238
> 
> That step took 40 minutes.  According to 'systat -vm 1', da0 tended to
> show around 90% utilization, da1 was about 23% and MB/s was about 1.6
> for both during the creation of the jail.
> 
> After about 20 to 40 minutes of ensuring that the jail was working
> properly with the compat6x libs, I decided to erase the test jail and
> get ready for doing the transfer for real during the next maintenance
> window.
> 
> Just before the box stopped responding to me, I had run: 
> sudo ezjail-admin delete -w testhost.tcworks.net 
> 
> It might have been about 30 seconds after that I noticed it wasn't
> responding.
> 
> FreeBSD web1.hosting.tcworks.net 7.0-STABLE FreeBSD
> 7.0-STABLE #1: Wed Jul 23 03:09:31 CDT 2008
> root at web1.hosting.tcworks.net:/usr/obj/usr/src/sys/GENERIC i386
>
> I changed the fs = part of this line to avoid privacy issues for my
> customer whose web host jail I've been working with.  I just changed the
> hostname:
> 
> dev = md14, block = 1320903, fs = /home/ezjails/testhost.tcworks.net
> 
> 21:23:17 Mon Nov 24 # kgdb /boot/kernel/kernel vmcore.0 
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-marcel-freebsd"...
> 
> Unread portion of the kernel message buffer:
> dev = md14, block = 1320903, fs = /home/ezjails/testhost.tcworks.net
> panic: ffs_blkfree: freeing free frag
> cpuid = 0
> Uptime: 124d14h34m47s
> Physical memory: 1011 MB
> Dumping 199 MB: 184 168 152 136 120 104 88 72 56 40 24 8
> 

<snip reading symbols>

> #0  doadump () at pcpu.h:195
> 195     pcpu.h: No such file or directory.
>         in pcpu.h
> (kgdb) bt
> #0  doadump () at pcpu.h:195
> #1  0xc077fd27 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
> #2  0xc077ffe9 in panic (fmt=Variable "fmt" is not available.
> ) at /usr/src/sys/kern/kern_shutdown.c:572
> #3  0xc09667a3 in ffs_blkfree (ump=0xc865d400, fs=0xc4946800, 
>     devvp=0xc7dc4880, bno=1320903, size=2048, inum=336842)
>     at /usr/src/sys/ufs/ffs/ffs_alloc.c:1918
> #4  0xc097a1af in handle_workitem_freeblocks (freeblks=0xc5aab600, flags=0)
>     at /usr/src/sys/ufs/ffs/ffs_softdep.c:2764
> #5  0xc097b1b8 in process_worklist_item (mp=0xc4859598, flags=Variable "flags" is not available.
> )
>     at /usr/src/sys/ufs/ffs/ffs_softdep.c:962
> #6  0xc097cc62 in softdep_process_worklist (mp=0xc4859598, full=0)
>     at /usr/src/sys/ufs/ffs/ffs_softdep.c:845
> #7  0xc097f687 in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:756
> #8  0xc075d479 in fork_exit (callout=0xc097f210 <softdep_flush>, arg=0x0, 
>     frame=0xe44edd38) at /usr/src/sys/kern/kern_fork.c:781
> #9  0xc0a6e170 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205
> (kgdb) 

My non-developer eye thinks this looks like some of the other
ffs_blkfree panics in the PR database which tend to involve deleteing
large amounts of data in FreeBSD 5 and 6 and 7-CURRENT.

http://www.freebsd.org/cgi/query-pr-summary.cgi?text=ffs_blkfree

Some refer to "freeing free block", some to "freeing free frag".

Would disabling soft-updates be a good temporary workaround?  I can't
afford to attempt to reproduce the problem, on this box, until I get
some production stuff shifted.  I am going to try on a different box,
but everything about that box is different except that it uses a
gmirror.

Thanks for your help and Happy Thanksgiving, for those of you who
celebrate it!
 
-- 
Scott Lambert                    KC5MLE                       Unix SysAdmin
lambert at lambertfam.org



More information about the freebsd-stable mailing list