Fatal Trap 12 in various processes always at address 0x3030313a

[Richard Mahlerwein]

From: Richard Mahlerwein <mahlerrd at yahoo.com>
Subject: Re: Fatal Trap 12 in various processes always at address 0x3030313a
To: "FreeBSD-Stable" <freebsd-stable at freebsd.org>
Date: Tuesday, September 1, 2009, 2:58 PM

> From: Gavin Atkinson <gavin at FreeBSD.org>
> Subject: Re: Fatal Trap 12 in various processes always at address 0x3030313a
> To: "Richard Mahlerwein" <mahlerrd at yahoo.com>
> Cc: "FreeBSD-Stable" <freebsd-stable at FreeBSD.org>
> Date: Sunday, August 30, 2009, 3:47 PM
> On Sat, 29 Aug 2009, Richard
> Mahlerwein wrote:
> 
>> I upgraded from 7.1-PRELEASE to -stable and all seemed fine
>> until I rebooted out of single user mode after doing make
>> installworld and mergemaster.  At that point, near the
>> end of the boot sequence I got a core dump, apparently
>> triggered by devd.
>> 
>> ****
>> Fatal trap 12: page fault while in kernel mode.
>> cpu id = 0; apic id = 00
>> fault virtual address = 0x3030313a
>> fault code = supervisor write, page not present
>> [snip]
>> current process = 355 (devd)
>> ****

>> Does anyone have a further recommendation on what to do,
>> try, test or change?
> 
> Firstly, please set up a dump partition by adding
> 'dumpdev="AUTO"' to your rc.conf.
> 
> Then, can you compile in the kernel debugger (options KGB /
> options DDB) and when this happens again, please obtain a
> backtrace from the debugger with the "bt" command. 
> Then, give the "show registers" command so that we can
> establish which register is pointing to the odd address.
> Finally, issue the "call" command to hopefully
> save a copy of the kernel dump for later analysis.
> 
> Thanks,
> 
> Gavin

Error at virtual address 0x3030313a
current process 352(sysctl) 

bt says (typing blind, but I think I got it):

Tracing pid 352 tid 100044 td 0xc3378480
sysctl_devctl_disable(c0c80ac0,0,0,d82fcba4,d82fcba4...) at sysctl_devctl_disable+0xaa
sysctl_root(d82fcba4,4,1,d82fcc60,0,...) at sysctl_root+0x187
userland_sysctl(c3378480,d82fcc14,3,0,0,...) at userland_sysctl+0x1c4
__sysctl(c3378480,d82fccfc,18,d82fcd38,d82fcd2c,...) at __sysctl_0x94
syscall(d82fcd38) at syscall_0x335
Xint0x80_syscall() at Xint0x80_syscall_0x20
--- syscall (202,FreeBSD ELF32, __sysctl, eip = 0x2815beaf, esp=0xbfbfe55c, epb = 0xbfbfe588 ---

show registers :

cs 0x20
ds 0xc1470028
es 0xc1470028
fs 0xc1460008
ss 0x28
eax 0xc0cea0cc devsoftc_0x4c
ecx 0
edx 0x30303132
ebx 0xc3181350
esp 0xd82fcb34
ebp 0xd82fcb54
esi 0
edi 0
eip 0xc08218da sysctl_devctl_disable+0xaa
efl 0x10202
sysctl_devctl_disable+0xaa: movl %eax,0x8(%edx)

I hope that tells someone on the list way more than it tells me.

Thanks again for all the help so far!

-Rich
I hope changing back to non-text (to get around some messed-up-ed-ness I've been having in my Yahoo account) won't mess
 this up too bad.  Please don't yell at me if it does - just a polite mention will do.  :)

I
changed devd_enable="NO" to devd_enable="YES" in rc.conf, and get THIS
error (which is back to being almost the same as the original one I got
on the first attempt at a new kernel)

vault virtual address 0x3030313a
fault code = supervisor write, page not present
instruction pointer = 0x20:0xc08216dc
stack pointer = 0x28: 0xd8311b70
frame pointer = 0x28:0xd8300b8c
code segment = base 0x0, limit 0xfffff, type 0x1b
=DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process -= 255 (devd)
[thread pid 355 tid 100045 ]
stopped at devread+0x12c: movl %eax,0x8(%edx)

bt says:
Tracing pid 355 tid 100045 tc 0xc3378240
devread(c30df400,d8300c60,0,c3434564,d8300bd8,...) at devread+0x12c
giant_read(c30df400,d8300c60,0,0,400,...) at giant_read+0x89
devfs_read_f(c33adb48,d8300c60,c3071300,0,c33adb48,...) at
 dofileread+0x96
kern_readv(c3378240,3,d8300c60,bfbfe9f7,400,...) at kern_readv+0x58
read(c3378240,d8300cfc,c,80000000,369e99,...) at read+0x4f
syscall(d8300d38) at syscall_0x335
Xint0x80_syscall() at Xint0x80_syscall_0x20
--- syscall (3, FreeBSD ELF32, read), eip - 0x8086c7f, esp = 0xbfbfe9bc, ebp = 0xbfbfee98 ---

And show registers:
cs=20,ds=d8300028, es=d8300028,fs= d8300008, ss=28, eax=c0cea0cc devsoftc_0x4c
ecx=0x4,,
edx=0x30303132, ebx=oxc3181350, esp=pxd8300b70, ebp=oxd8300b8c,
esi=0xc30df400, edi=0x6, eip=oxc08216dc devread_0x12c, efl=0x10202
devread+0x12c: movl %eax, 0x8(%edx)

I'm
still not quite sure what that's telling me, but is it significant that
it's the same virtual location with a different piece of code loaded
there because of a change in options?  No other change was made, just
changing devd_enable.  I'm going to check loader.conf, too, as someone
else suggested (but at this point, I didn't want to muddy the waters by
changing two things at once).