dtrace ustack kernel panic

[Andriy Gapon]

on 30/07/2011 00:27 maestro something said the following:
> Hi,
> 
> trying to do so I don't really find my way around. This is what I get when I run
> kgdb
> 
> On startup the assert frame is #7 and the probe frame is #8.
> 
[snip]
> kernel trap 12 with interrupts disabled
> 
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address    = 0x108
> fault code        = supervisor write, page not present
> instruction pointer    = 0x20:0xc11012d7
> stack pointer            = 0x28:0xcd3ed9f4
> frame pointer            = 0x28:0xcd3eda0c
> code segment        = base 0x0, limit 0xfffff, type 0x1b
>             = DPL 0, pres 1, def32 1, gran 1
> processor eflags    = resume, IOPL = 0
> current process        = 1132 (nc)
> trap number        = 12
> panic: page fault
> cpuid = 0
> KDB: stack backtrace:
> #0 0xc09036a7 at kdb_backtrace+0x47
> #1 0xc08d1a07 at panic+0x117
> #2 0xc0c158c3 at trap_fatal+0x323
> #3 0xc0c15bc0 at trap_pfault+0x2f0
> #4 0xc0c1612a at trap+0x48a
> #5 0xc0bfc97c at calltrap+0x6
> #6 0xc10e992b at dtrace_panic+0x1b
> #7 0xc10e995d at dtrace_assfail+0x2d
> #8 0xc10fb28c at dtrace_probe+0x135c
> #9 0xc1237f72 at systrace_probe+0x62
> #10 0xc090f63f at syscallenter+0x47f
> #11 0xc0c15c14 at syscall+0x34
> #12 0xc0bfca11 at Xint0x80_syscall+0x21
> Uptime: 3m0s
> Physical memory: 239 MB
> Dumping 79 MB: 64 48 32 16
[snip]
> #0  doadump () at pcpu.h:231
> 231        __asm("movl %%fs:0,%0" : "=r" (td));
> 
> once I'm in the kgdb console i type where and all of a sudden the stack trace
> has only 7 frames... that does not seem correct. Furthermore, the "Previous
> frame inner to this frame (corrupt stack?)" error message is not too encuraging
> either.
> 
> (kgdb) where
> #0  doadump () at pcpu.h:231
> #1  0xc08d17a3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
> #2  0xc08d1a40 in panic (fmt=Variable "fmt" is not available.
> ) at /usr/src/sys/kern/kern_shutdown.c:592
> #3  0xc0c158c3 in trap_fatal (frame=0xcd3ed9b4, eva=264) at
> /usr/src/sys/i386/i386/trap.c:946
> #4  0xc0c15bc0 in trap_pfault (frame=0xcd3ed9b4, usermode=0, eva=264) at
> /usr/src/sys/i386/i386/trap.c:859
> #5  0xc0c1612a in trap (frame=0xcd3ed9b4) at /usr/src/sys/i386/i386/trap.c:532
> #6  0xc0bfc97c in calltrap () at /usr/src/sys/i386/i386/exception.s:166
> #7  0xc11012d7 in dtrace_panic_trigger () from /boot/kernel/dtrace.ko
> Previous frame inner to this frame (corrupt stack?)
> (kgdb)
> 
> what am I doing wrong and what do I have to do to get the specific assert that
> fails?

I am not quite sure.  Apparently there is some issue with either what
information we store in a crash dump and how, or with how kgdb interprets that
information, or with a combination of both...
Perhaps this is a result of -O2 optimization and -O1 would improve the situation
- not sure again...
Meanwhile, there is something simple that you can do without much hassle:
(kgdb) list *dtrace_probe+0x135c
where the address is taken from the backtrace printed by panic(9).


> On Tue, Jul 26, 2011 at 4:05 AM, Andriy Gapon <avg at freebsd.org
> <mailto:avg at freebsd.org>> wrote:
> 
>     on 26/07/2011 04:20 maestro something said the following:
>     > Hi,
>     >
>     > when using the ustack action on the latest FreeBSD8.2 i386 the kernel
>     > panics.
>     >
>     > Here is the information I could gather:
>     >
>     > let me know if I can provide more information. (i.e., i have the full crash
>     > information 80+mbs handy)
> 
>     Use kgdb on the crash dump, go to the dtrace_probe frame and see which exactly
>     assert fails and why.
> 
>     > Here is the backtrace:
>     >
>     > Fatal trap 12: page fault while in kernel mode
>     > cpuid = 0; apic id = 00
>     > fault virtual address   = 0x108
>     > fault code              = supervisor write, page not present
>     > instruction pointer     = 0x20:0xc11012d7
>     > stack pointer           = 0x28:0xcd3ed9f4
>     > frame pointer           = 0x28:0xcd3eda0c
>     > code segment            = base 0x0, limit 0xfffff, type 0x1b
>     >                         = DPL 0, pres 1, def32 1, gran 1
>     > processor eflags        = resume, IOPL = 0
>     > current process         = 1132 (nc)
>     > trap number             = 12
>     > panic: page fault
>     > cpuid = 0
>     > KDB: stack backtrace:
>     > #0 0xc09036a7 at kdb_backtrace+0x47
>     > #1 0xc08d1a07 at panic+0x117
>     > #2 0xc0c158c3 at trap_fatal+0x323
>     > #3 0xc0c15bc0 at trap_pfault+0x2f0
>     > #4 0xc0c1612a at trap+0x48a
>     > #5 0xc0bfc97c at calltrap+0x6
>     > #6 0xc10e992b at dtrace_panic+0x1b
>     > #7 0xc10e995d at dtrace_assfail+0x2d
>     > #8 0xc10fb28c at dtrace_probe+0x135c
>     > #9 0xc1237f72 at systrace_probe+0x62
>     > #10 0xc090f63f at syscallenter+0x47f
>     > #11 0xc0c15c14 at syscall+0x34
>     > #12 0xc0bfca11 at Xint0x80_syscall+0x21
>     > Uptime: 3m0s
>     > Physical memory: 239 MB
>     > Dumping 79 MB: 64 48 32 16
>     >
>     >
>     > Steps To reproduce:
>     >
>     > the dtrace program (i.e., test.d) was:
>     >
>     > #!/usr/sbin/dtrace -s
>     >
>     > syscall::accept:return
>     > / execname == "nc" /
>     > {
>     >     printf("%s accept:return\n", execname);
>     >     ustack();
>     > }
>     >
>     > % dtrace -s test.d
>     >
>     > then running
>     > % nc -vl 8080
>     > on one shell and
>     > % nc localhost 8080
>     > on another would make the kernel panic


-- 
Andriy Gapon