Kernel panic on 7.0-RELEASE

[Phillip Hocking]

Hello all, I keep getting a kernel panic every Saturday night, so I 
figured I would go through the core dump.

# uname -a
FreeBSD xx.fsklaw.com 7.0-RELEASE FreeBSD 7.0-RELEASE #1: Wed Apr 23 
08:01:10 PDT 2008     xx at to-new.fsklaw.com:/usr/obj/usr/src/sys/TEST  amd64



to# kgdb kernel.symbols /var/crash/vmcore.2
[GDB will not be able to debug user-mode threads: 
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address    = 0x250
fault code        = supervisor read data, page not present
instruction pointer    = 0x8:0xffffffff80482a5d
stack pointer            = 0x10:0xffffffffb184b570
frame pointer            = 0x10:0xffffff003f9cf6a0
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 37920 (bsdtar)
trap number        = 12
panic: page fault
cpuid = 1
Uptime: 7d2h23m33s
Physical memory: 2034 MB
Dumping 415 MB: 400 384 368 352 336 320 304 288 272 256 240 224 208 192 
176 160 144 128 112 96 80 64 48 32 16

#0  doadump () at pcpu.h:194
194        __asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) where
#0  doadump () at pcpu.h:194
#1  0x0000000000000004 in ?? ()
#2  0xffffffff8048e0d9 in boot (howto=260) at 
/usr/src/sys/kern/kern_shutdown.c:409
#3  0xffffffff8048e4dd in panic (fmt=0x104 <Address 0x104 out of 
bounds>) at /usr/src/sys/kern/kern_shutdown.c:563
#4  0xffffffff80786f44 in trap_fatal (frame=0xffffff003f9cf6a0, 
eva=18446742975518566608) at /usr/src/sys/amd64/amd64/trap.c:724
#5  0xffffffff80787315 in trap_pfault (frame=0xffffffffb184b4c0, 
usermode=0) at /usr/src/sys/amd64/amd64/trap.c:641
#6  0xffffffff80787c58 in trap (frame=0xffffffffb184b4c0) at 
/usr/src/sys/amd64/amd64/trap.c:410
#7  0xffffffff8076d88e in calltrap () at 
/usr/src/sys/amd64/amd64/exception.S:169
#8  0xffffffff80482a5d in _mtx_lock_sleep (m=0xffffff009662c410, 
tid=18446742975265175200, opts=Variable "opts" is not available.
) at /usr/src/sys/kern/kern_mutex.c:335
#9  0xffffffff804f5978 in vfs_vmio_release (bp=0xffffffff9a329220) at 
/usr/src/sys/kern/vfs_bio.c:1530
#10 0xffffffff804f77f4 in getnewbuf (slpflag=0, slptimeo=0, 
size=Variable "size" is not available.
) at /usr/src/sys/kern/vfs_bio.c:1847
#11 0xffffffff804f8801 in getblk (vp=0xffffff000b71c5d0, blkno=294994, 
size=16384, slpflag=0, slptimeo=0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/vfs_bio.c:2602
#12 0xffffffff806a3d5b in ffs_balloc_ufs2 (vp=0xffffff000b71c5d0, 
startoffset=Variable "startoffset" is not available.
) at /usr/src/sys/ufs/ffs/ffs_balloc.c:844
#13 0xffffffff806bf6e5 in ffs_write (ap=0xffffffffb184ba10) at 
/usr/src/sys/ufs/ffs/ffs_vnops.c:720
#14 0xffffffff807e04e5 in VOP_WRITE_APV (vop=0xffffffff80a6b0e0, 
a=0xffffffffb184ba10) at vnode_if.c:691
#15 0xffffffff8051ae41 in vn_write (fp=0xffffff00284771e0, 
uio=0xffffffffb184bb00, active_cred=Variable "active_cred" is not available.
) at vnode_if.h:373
#16 0xffffffff804c6468 in dofilewrite (td=0xffffff003f9cf6a0, fd=3, 
fp=0xffffff00284771e0, auio=0xffffffffb184bb00, offset=Variable "offset" 
is not available.
) at file.h:254
#17 0xffffffff804c676e in kern_writev (td=0xffffff003f9cf6a0, fd=3, 
auio=0xffffffffb184bb00) at /usr/src/sys/kern/sys_generic.c:401
#18 0xffffffff804c67ec in write (td=0xffffff009662c410, 
uap=0xffffff003f9cf6a0) at /usr/src/sys/kern/sys_generic.c:317
#19 0xffffffff80787597 in syscall (frame=0xffffffffb184bc70) at 
/usr/src/sys/amd64/amd64/trap.c:852
#20 0xffffffff8076da9b in Xfast_syscall () at 
/usr/src/sys/amd64/amd64/exception.S:290
#21 0x0000000800a66d3c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

Note the uptime, this happens every Saturday for the last three weeks. I 
have a cronjob that tar/bzips a drive and scps it to another box, but 
that runs nightly at midnight on the cronjob, and this happens weekly 
but only on Saturday, same cronjob same data. I figured if it happened 
again, I would hit all you leet-sauce developers up. Hopefully you all 
can suggest what to upgrade to or how to further troubleshoot this 
issue. Thanks a bundle y'all, love the beastie!