How to get anything useful out of kgdb?

[Andriy Gapon]

On 15/05/2015 20:57, Ryan Stone wrote:
> *Sigh*,  kgdb isn't unwinding the trap frame properly.  You can try this to
> figure out where it was running:

I wonder, what is a reason for this?
Can that be fixed in kgdb itself?
It seems that usually kgdb handles trap frames just fine, but not always?

> That gives you the top of the callstack at the time that the core was
> taken.  To get the rest of it, try:
> 
> define trace_stack
>   set $frame_ptr=$arg0
>   set $iters=0
>   while $frame_ptr != 0 && $iters < $arg1
>     set $ret_addr=((char*)$frame_ptr) + sizeof(void*)
>     printf "frameptr=%p, ret_addr=%p\n", (void*)$frame_ptr, *(void**)$ret_addr
>     printf "    "
>     info line **(void***)$ret_addr
>     set $frame_ptr=*(void**)$frame_ptr
>     set $iters=$iters+1
>   end
> end
> 
> trace_stack frame->tf_rbp 20

Thank you for this script.
Here is an example from my practice.

(kgdb) bt
#0  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:291
#1  0xffffffff8063453f in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:359
#2  0xffffffff80634ba4 in vpanic (fmt=<value optimized out>, ap=<value optimized
out>) at /usr/src/sys/kern/kern_shutdown.c:635
#3  0xffffffff806348a3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:568
#4  0xffffffff8041bba7 in db_panic (addr=<value optimized out>, have_addr=false,
count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:473
#5  0xffffffff8041b67b in db_command (cmd_table=0x0) at
/usr/src/sys/ddb/db_command.c:440
#6  0xffffffff8041b524 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493
#7  0xffffffff8041de0b in db_trap (type=<value optimized out>, code=0) at
/usr/src/sys/ddb/db_main.c:251
#8  0xffffffff80669de8 in kdb_trap (type=19, code=0, tf=0xffffffff80f976d0) at
/usr/src/sys/kern/subr_kdb.c:653
#9  0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at
/usr/src/sys/amd64/amd64/trap.c:381
#10 0xffffffff80809623 in nmi_calltrap () at
/usr/src/sys/libkern/explicit_bzero.c:28
#11 0xffffffff80619e1f in __mtx_assert (c=<value optimized out>, what=<value
optimized out>, file=<value optimized out>, line=<value optimized out>) at
/usr/src/sys/kern/kern_mutex.c:842
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal

(kgdb) fr 9
#9  0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at
/usr/src/sys/amd64/amd64/trap.c:381
381                                             kdb_trap(type, 0, frame);

(kgdb) trace_stack frame->tf_rbp 20
frameptr=0xfffffe02b8356e90, ret_addr=0xffffffff807fef86
    Line 833 of "/usr/src/sys/vm/vm_reserv.c" starts at address
0xffffffff807fef86 <vm_reserv_free_page+38> and ends at 0xffffffff807fef90
<vm_reserv_free_page+48>.
frameptr=0xfffffe02b8356eb0, ret_addr=0xffffffff807f2b96
    Line 2432 of "/usr/src/sys/vm/vm_page.c" starts at address
0xffffffff807f2b96 <vm_page_free_toq+262> and ends at 0xffffffff807f2b9c
<vm_page_free_toq+268>.
frameptr=0xfffffe02b8356ed0, ret_addr=0xffffffff807f2e4d
    Line 963 of "/usr/src/sys/vm/vm_page.c" starts at address 0xffffffff807f2e4d
<vm_page_free+13> and ends at 0xffffffff807f2e50 <vm_page_free_zero>.
frameptr=0xfffffe02b8356ee0, ret_addr=0xffffffff821c28e2
    Line 268 of
"/usr/src/sys/modules/drm2/drm2/../../../dev/drm2/ttm/ttm_bo_vm.c" starts at
address 0xffffffff821c28e2 <ttm_bo_vm_fault+1010> and ends at 0xffffffff821c28ee
<ttm_bo_vm_fault+1022>.
frameptr=0xfffffe02b8356f50, ret_addr=0xffffffff807d4fd3
    Line 321 of "/usr/src/sys/vm/device_pager.c" starts at address
0xffffffff807d4fce <dev_pager_getpages+94> and ends at 0xffffffff807d4fdb
<dev_pager_getpages+107>.
frameptr=0xfffffe02b8356fa0, ret_addr=0xffffffff807f9d67
    Line 291 of "/usr/src/sys/vm/vm_pager.c" starts at address
0xffffffff807f9d58 <vm_pager_get_pages+40> and ends at 0xffffffff807f9d6a
<vm_pager_get_pages+58>.
frameptr=0xfffffe02b8356fd0, ret_addr=0xffffffff807e0d84
    Line 675 of "/usr/src/sys/vm/vm_fault.c" starts at address
0xffffffff807e0d84 <vm_fault_hold+1860> and ends at 0xffffffff807e0d8d
<vm_fault_hold+1869>.
frameptr=0xfffffe02b83578f0, ret_addr=0xffffffff807e05ee
    Line 277 of "/usr/src/sys/vm/vm_fault.c" starts at address
0xffffffff807e05d9 <vm_fault+121> and ends at 0xffffffff807e05f1 <vm_fault+145>.
frameptr=0xfffffe02b8357930, ret_addr=0xffffffff80821342
    Line 735 of "/usr/src/sys/amd64/amd64/trap.c" starts at address
0xffffffff80821342 <trap_pfault+290> and ends at 0xffffffff80821346
<trap_pfault+294>.
frameptr=0xfffffe02b83579c0, ret_addr=0xffffffff80820bda
    Line 326 of "/usr/src/sys/amd64/amd64/trap.c" starts at address
0xffffffff80820bc6 <trap+1366> and ends at 0xffffffff80820bdf <trap+1391>.
frameptr=0xfffffe02b8357bd0, ret_addr=0xffffffff8082154a
    Line 629 of "/usr/src/sys/amd64/amd64/trap.c" starts at address
0xffffffff8082154a <trap_check+42> and ends at 0xffffffff80821560
<dblfault_handler>.
frameptr=0xfffffe02b8357bf0, ret_addr=0xffffffff808091e3
    Line 28 of "/usr/src/sys/libkern/explicit_bzero.c" starts at address
0xffffffff806e74dd <explicit_bzero+29> and ends at 0xffffffff8088a2d0
<__explicit_bzero_hook>.
frameptr=0x7fffffffe8f0Cannot access memory at address 0x7fffffffe8f8

Output of trace_stack looks perfectly sane for me up to the next trap frame.

-- 
Andriy Gapon