7-BETA4/7-RC1 crashes randomly at shutdown

[Yoshihiro Ota]

Hello.

I have been following RELENG_7 and then RELENG_7_0.
I update /usr/src daily and install them.
I think the crash started around 12/18.
Since then, I started seen pancing at umount during shutdown.
I am using GENERIC kernel with SCHED_ULE.

This crash tends to happen after some amount of I/O performed.
I cannot really tell what kind of I/O will trigger
this umount failure, yet.

I got a core and 'where' command.

Does anyone have any ideas or suggestions?

Thanks,
Hiro

# kgdb /usr/obj/usr/src/sys/GENERIC/kernel.debug /var/crash/vmcore.3 
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
<118>/etc/rc.shutdown: WARNING: $samba_enable is not set properly - see rc.conf(5).
<118>Writing entropy file:
<118>.
<118>.
<118>Dec 26 01:11:56 XXXXXX syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Wait
iSnygn c(imnagx  dis60k s,s evcnoonddess)  rfeomra isynsitneg.m. .p3r ocess `syncer' to stop...3 3 1 1 1 1 0 0 0 0 done
All buffers synced.
GEOM_ELI: Detached da0.eli on last close.
acpi_ec0: warning: EC done before starting event wait
GEOM_ELI: Detached ad4s2f.eli on last close.


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x188
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc07479d4
stack pointer           = 0x28:0xe3c71b1c
frame pointer           = 0x28:0xe3c71b34
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1 (init)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 1h24m11s
Physical memory: 2025 MB
Dumping 155 MB: 140 124 108 92 76 60 44 28 12

#0  doadump () at pcpu.h:195
195     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) where             
#0  doadump () at pcpu.h:195
#1  0xc0753e87 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0754149 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc0a689ec in trap_fatal (frame=0xe3c71adc, eva=392)
    at /usr/src/sys/i386/i386/trap.c:899
#4  0xc0a68c70 in trap_pfault (frame=0xe3c71adc, usermode=0, eva=392)
    at /usr/src/sys/i386/i386/trap.c:812
#5  0xc0a6961c in trap (frame=0xe3c71adc) at /usr/src/sys/i386/i386/trap.c:490
#6  0xc0a4f5ab in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc07479d4 in _mtx_lock_sleep (m=0xc8abfa18, tid=3306133088, opts=0, 
    file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:335
#8  0xc07cc94f in vgone (vp=0xc8abf990) at /usr/src/sys/kern/vfs_subr.c:2471
#9  0xc5742a79 in ?? ()
#10 0xc8abf990 in ?? ()
#11 0xc55a9a70 in ?? ()
#12 0x00000000 in ?? ()
#13 0xc5743c20 in ?? ()
#14 0x0000016c in ?? ()
#15 0xc5746074 in ?? ()
#16 0x00003002 in ?? ()
#17 0xc8abf990 in ?? ()
#18 0xe3c71bf0 in ?? ()
#19 0x00000004 in ?? ()
---Type <return> to continue, or q <return> to quit---
#20 0xc50f9660 in ?? ()
#21 0xe3c71ba8 in ?? ()
#22 0xc57420b4 in ?? ()
#23 0xc55a9a70 in ?? ()
#24 0xc5746000 in ?? ()
#25 0x00001002 in ?? ()
#26 0xe3c71bf0 in ?? ()
#27 0xc50f9660 in ?? ()
#28 0xc569d550 in ?? ()
#29 0xe3c71c00 in ?? ()
#30 0xc07c6ac1 in dounmount (mp=0xc55a9a70, flags=-982228992, td=0x10)
    at /usr/src/sys/kern/vfs_mount.c:1270
Previous frame inner to this frame (corrupt stack?)
(kgdb)