kern/135690: 7.1 p4 panic: panic: ufs_dirbad: /backuphd: bad dir ino 22259126 at offset 0: mangled entry

Thu Jun 18 07:00:07 UTC 2009

>Number:         135690
>Category:       kern
>Synopsis:       7.1 p4 panic: panic: ufs_dirbad: /backuphd: bad dir ino 22259126 at offset 0: mangled entry
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jun 18 07:00:06 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Derek Kulinski
>Release:        FreeBSD 7.1 p4
>Organization:
>Environment:
FreeBSD chinatsu.takeda.tk 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4 #0: Wed Mar 25 02:01:36 PDT 2009     root at chinatsu.takeda.tk:/usr/obj/usr/src/sys/CHINATSU  i386

>Description:
Ok... so this is a bit complicated...

Output from kgdb:
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
ad2: TIMEOUT - READ_DMA48 retrying (1 retry left) LBA=355843068
panic: ufs_dirbad: /backuphd: bad dir ino 22259126 at offset 0: mangled entry
KDB: stack backtrace:
db_trace_self_wrapper(c0809ee8,c0882620,c081def3,d139f930,d139f930,...) at db_trace_self_wrapper+0x26
panic(c081def3,c26c84dc,153a5b6,0,c081dfb1,...) at panic+0xf8
ufs_dirbad(c2d01900,0,c081dfb1,0,d139f9cc,...) at ufs_dirbad+0x73
ufs_lookup(d139f9f8,d139f9f8,d139fbcc,d139fbb8,c2b49200,...) at ufs_lookup+0x4bd
vfs_cache_lookup(d139fa84,c05f5271,2,c2d0fe04,d139faa4,...) at vfs_cache_lookup+0xf2
VOP_LOOKUP_APV(c085a940,d139fa84,d139fbcc,c080fb81,2a9,...) at VOP_LOOKUP_APV+0x3d
lookup(d139fba4,c24d5400,0,d139fbc0,c2d0fe8c,...) at lookup+0x50f
namei(d139fba4,d139fb44,60,0,c2ad98c0,...) at namei+0x3a8
kern_stat(c2ad98c0,bfbfd710,0,d139fc14,52,...) at kern_stat+0x3d
stat(c2ad98c0,d139fcf8,8,c05a18b5,c2ad98c0,...) at stat+0x2f
syscall(d139fd38) at syscall+0x208
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (188, FreeBSD ELF32, stat), eip = 0x206fa46b, esp = 0xbfbfc14c, ebp = 0xbfbfc168 ---
Uptime: 1h23m45s
Physical memory: 367 MB
Dumping 77 MB: 62 46 30 14

Reading symbols from /boot/kernel/mac_seeotheruids.ko...Reading symbols from /boot/kernel/mac_seeotheruids.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/mac_seeotheruids.ko
Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols from /boot/kernel/geom_journal.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_journal.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
#0  doadump () at pcpu.h:196
196             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc05824eb in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc058288c in panic (fmt=0xc081def3 "ufs_dirbad: %s: bad dir ino %lu at offset %ld: %s")
    at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc0768733 in ufs_dirbad (ip=Variable "ip" is not available.
) at /usr/src/sys/ufs/ufs/ufs_lookup.c:607
#4  0xc076992d in ufs_lookup (ap=0xd139f9f8) at /usr/src/sys/ufs/ufs/ufs_lookup.c:297
#5  0xc05f2b52 in vfs_cache_lookup (ap=0xd139fa84) at vnode_if.h:83
#6  0xc07e0bfd in VOP_LOOKUP_APV (vop=0xc085ae60, a=0xd139fa84) at vnode_if.c:99
#7  0xc05f970f in lookup (ndp=0xd139fba4) at vnode_if.h:57
#8  0xc05fa788 in namei (ndp=0xd139fba4) at /usr/src/sys/kern/vfs_lookup.c:219
#9  0xc0608b3d in kern_stat (td=0xc2ad98c0, path=0xbfbfd710 <Address 0xbfbfd710 out of bounds>, pathseg=UIO_USERSPACE,
    sbp=0xd139fc14) at /usr/src/sys/kern/vfs_syscalls.c:2113
#10 0xc0608d3f in stat (td=0xc2ad98c0, uap=0xd139fcf8) at /usr/src/sys/kern/vfs_syscalls.c:2097
#11 0xc07cbb38 in syscall (frame=0xd139fd38) at /usr/src/sys/i386/i386/trap.c:1090
#12 0xc07b50f0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
#13 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) up
#1  0xc05824eb in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
418                     doadump();
(kgdb) up
#2  0xc058288c in panic (fmt=0xc081def3 "ufs_dirbad: %s: bad dir ino %lu at offset %ld: %s")
    at /usr/src/sys/kern/kern_shutdown.c:574
574             boot(bootopt);
(kgdb) up
#3  0xc0768733 in ufs_dirbad (ip=Variable "ip" is not available.
) at /usr/src/sys/ufs/ufs/ufs_lookup.c:607
607                     panic("ufs_dirbad: %s: bad dir ino %lu at offset %ld: %s",
(kgdb) up
#4  0xc076992d in ufs_lookup (ap=0xd139f9f8) at /usr/src/sys/ufs/ufs/ufs_lookup.c:297
297                             ufs_dirbad(dp, dp->i_offset, "mangled entry");

I have two disks.
The disk where's an empty /backuphd directory is using UFS2 (no softupdates):
/dev/ad0s1a on / (ufs, NFS exported, local)

On top of that I have partition from another disk (/dev/ad2), I was experimenting and I set up journaling on it. The disk is mounted as /backuphd.

Now, perhaps this isn't really related to the issue, but for completness sake I'll mention it. I have one directory called "david" I just created to back up files from a windows machine. I also have samba3 setup for sharing my home directory. I didn't felt like creating separate share for this so I simply made a symlink to that directory.

Here's the layout
/dev/ad0s1a on / (ufs, NFS exported, local)
devfs on /dev (devfs, local)
/dev/ad0s1d on /var (ufs, local, with quotas, soft-updates)
/dev/ad0s1e on /tmp (ufs, local, soft-updates)
/dev/ad0s1f on /usr (ufs, NFS exported, local, with quotas, soft-updates)
devfs on /var/named/dev (devfs, local)
devfs on /var/db/dhcpd/dev (devfs, local)
mayumi:/usr/obj on /usr/obj (nfs)
/dev/ad2.journal on /backuphd (ufs, asynchronous, NFS exported, local, gjournal)

The drives seem to be clean:
[chinatsu]:/usr/obj/usr/src/sys/CHINATSU# fsck /
** /dev/ad0s1a (NO WRITE)
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
11727 files, 200752 used, 53063 free (2799 frags, 6283 blocks, 1.1% fragmentation)

[chinatsu]:/usr/obj/usr/src/sys/CHINATSU# fsck /backuphd
** /dev/ad2.journal (NO WRITE)
** Last Mounted on /backuphd
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
401506 files, 91325079 used, 144677463 free (16119 frags, 18082668 blocks, 0.0% fragmentation)

The filesystems are clean as you can see, I wasn't repairing them manually since last crash, all it run was the standard check that happens when filesystem wasn't properly dismounted.

>How-To-Repeat:
Don't know exact steps, it seems random.

This crash happened about 3 times within an hour (I just stopped accessing the drive after that, and system works for 11 hours already... it had uptime of 70 days or so before the crash).

It feels like the crash was caused by scanning contents of a directory, though that's just my assumption.

Basically the first crash happened after a night of copying files. In the morning it looked like it was done, so I went to Total Commander (windows file manager - remember it was through samba) to see the files and instantly I heard noises of computer restarting. Total commander refreshes the contents of directory listing whenever it is set to foreground.
>Fix:
No idea

>Release-Note:
>Audit-Trail:
>Unformatted: