kern/124670: large file operation on RAID cause many GEOM errors - crash

Wed Jun 25 14:09:38 UTC 2008

I've got two more cores today which I tried to debug..though I'm not getting a lot of information from
them. This all seems a bit random..as if there are bad blocks on the hard-drive and its just
randome whether the file hit them or not. but that's just speculation..could also be a software/hardware timing incompatibility.

[<code>]
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
panic: vinvalbuf: dirty bufs
cpuid = 0
Uptime: 26m36s
Physical memory: 627 MB
Dumping 114 MB: 99 83 67 51 35 19 3

#0  doadump () at pcpu.h:195
195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:195
#1  0xc0754457 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0754719 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc07cb2c6 in bufobj_invalbuf (bo=0xc370cc70, flags=1, td=0xc3c3e210, 
    slpflag=0, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1075
#4  0xc07cb5f2 in vinvalbuf (vp=0xc370cbb0, flags=1, td=0xc3c3e210, slpflag=0, 
    slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1143
#5  0xc07cb6bf in vgonel (vp=0xc370cbb0) at /usr/src/sys/kern/vfs_subr.c:2514
#6  0xc07cb9a7 in vgone (vp=0xc370cbb0) at /usr/src/sys/kern/vfs_subr.c:2472
#7  0xc06e7c05 in devfs_delete (dm=0xc36771c0, de=0xc3672000, vp_locked=0)
    at /usr/src/sys/fs/devfs/devfs_devs.c:265
#8  0xc06e80cd in devfs_populate_loop (dm=0xc36771c0, cleanup=0)
    at /usr/src/sys/fs/devfs/devfs_devs.c:384
#9  0xc06e840e in devfs_populate (dm=0xc36771c0)
    at /usr/src/sys/fs/devfs/devfs_devs.c:485
#10 0xc06ebe6e in devfs_lookup (ap=0xdaae19b8)
    at /usr/src/sys/fs/devfs/devfs_vnops.c:601
#11 0xc0a5f256 in VOP_LOOKUP_APV (vop=0xc0b71680, a=0xdaae19b8)
    at vnode_if.c:99
#12 0xc07c28f1 in lookup (ndp=0xdaae1b80) at vnode_if.h:57
#13 0xc07c35ff in namei (ndp=0xdaae1b80) at /usr/src/sys/kern/vfs_lookup.c:219
#14 0xc07d9fd7 in vn_open_cred (ndp=0xdaae1b80, flagp=0xdaae1c78, cmode=0, 
    cred=0xc39ba200, fp=0xc3b3f990) at /usr/src/sys/kern/vfs_vnops.c:188
---Type <return> to continue, or q <return> to quit---
#15 0xc07da2a3 in vn_open (ndp=0xdaae1b80, flagp=0xdaae1c78, cmode=0, 
    fp=0xc3b3f990) at /usr/src/sys/kern/vfs_vnops.c:94
#16 0xc07d7f27 in kern_open (td=0xc3c3e210, 
    path=0x2887e109 <Address 0x2887e109 out of bounds>, pathseg=UIO_USERSPACE, 
    flags=1, mode=0) at /usr/src/sys/kern/vfs_syscalls.c:1028
#17 0xc07d8490 in open (td=0xc3c3e210, uap=0xdaae1cfc)
    at /usr/src/sys/kern/vfs_syscalls.c:995
#18 0xc0a49635 in syscall (frame=0xdaae1d38)
    at /usr/src/sys/i386/i386/trap.c:1035
#19 0xc0a2fc70 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
[<CODE>]

and from the other core dump
[<CODE>]
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
panic: vinvalbuf: dirty bufs
cpuid = 0
Uptime: 3h47m18s
Physical memory: 627 MB
Dumping 111 MB: 96 80 64 48 32 16

#0  doadump () at pcpu.h:195
195             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:195
#1  0xc0754457 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0754719 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc07cb2c6 in bufobj_invalbuf (bo=0xc370cc70, flags=1, td=0xc39b7000, 
    slpflag=0, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1075
#4  0xc07cb5f2 in vinvalbuf (vp=0xc370cbb0, flags=1, td=0xc39b7000, slpflag=0, 
    slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1143
#5  0xc07cb6bf in vgonel (vp=0xc370cbb0) at /usr/src/sys/kern/vfs_subr.c:2514
#6  0xc07cb9a7 in vgone (vp=0xc370cbb0) at /usr/src/sys/kern/vfs_subr.c:2472
#7  0xc06e7c05 in devfs_delete (dm=0xc36771c0, de=0xc3672000, vp_locked=0)
    at /usr/src/sys/fs/devfs/devfs_devs.c:265
#8  0xc06e80cd in devfs_populate_loop (dm=0xc36771c0, cleanup=0)
    at /usr/src/sys/fs/devfs/devfs_devs.c:384
#9  0xc06e840e in devfs_populate (dm=0xc36771c0)
    at /usr/src/sys/fs/devfs/devfs_devs.c:485
#10 0xc06ebe6e in devfs_lookup (ap=0xdaa809b8)
    at /usr/src/sys/fs/devfs/devfs_vnops.c:601
#11 0xc0a5f256 in VOP_LOOKUP_APV (vop=0xc0b71680, a=0xdaa809b8)
    at vnode_if.c:99
#12 0xc07c28f1 in lookup (ndp=0xdaa80b80) at vnode_if.h:57
#13 0xc07c35ff in namei (ndp=0xdaa80b80) at /usr/src/sys/kern/vfs_lookup.c:219
#14 0xc07d9d32 in vn_open_cred (ndp=0xdaa80b80, flagp=0xdaa80c78, cmode=420, 
    cred=0xc39b9a00, fp=0xc3936798) at /usr/src/sys/kern/vfs_vnops.c:130
#15 0xc07da2a3 in vn_open (ndp=0xdaa80b80, flagp=0xdaa80c78, cmode=420, 
    fp=0xc3936798) at /usr/src/sys/kern/vfs_vnops.c:94
#16 0xc07d7f27 in kern_open (td=0xc39b7000, 
    path=0x28202480 <Address 0x28202480 out of bounds>, pathseg=UIO_USERSPACE, 
    flags=1538, mode=438) at /usr/src/sys/kern/vfs_syscalls.c:1028
#17 0xc07d8490 in open (td=0xc39b7000, uap=0xdaa80cfc)
    at /usr/src/sys/kern/vfs_syscalls.c:995
#18 0xc0a49635 in syscall (frame=0xdaa80d38)
    at /usr/src/sys/i386/i386/trap.c:1035
---Type <return> to continue, or q <return> to quit---
#19 0xc0a2fc70 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

[</CODE>]

Note that I was running gmirror at this time. So this has nothing to do with gmirror.

I thought perhaps there was a specific problem with a specific folder. But after several tests
It seems quite random. I could not reproduce it consistently. yikes.

Salik.

_________________________________________________________________

http://clk.atdmt.com/UKM/go/msnnkmgl0010000009ukm/direct/01/