Problems with bhyve's kgdb support and loadable modules

[Kurt Lidl]

On 5/3/14 11:01 PM, Peter Grehan wrote:
> Hi Kurt,
>
>> Is there any reasonable tutorial for using kgdb with the
>> bvmdebug kernel option?
>
>   Not really (https://wiki.freebsd.org/BHyVe/gdb)
>
>   In any event, 9.2 doesn't have bvmdebug, though it would be a simple
> backport. It's also not strictly required - you can use the serial port
> same as on h/w; see below.
>
>> A couple of folks I know have run into issues trying to
>> debug a FreeBSD stable/9 kernel from their bhyve
>> hosting machine (running stable/10).
>>
>> In particular, the loadable modules that are in use in
>> the stable/9 kernel are being "troublesome" to get to
>> the point where source-level debugging actually works.
>>
>> Even a pointer to a couple of "worked" examples might be
>> useful.
>>
>> I've read this:
>> http://people.freebsd.org/~jhb/papers/bsdcan/2008/article/node4.html
>> but not all the techniques in there appear to work properly.
>
>   I tried to repro this with some success from a host running CURRENT.
>
>   Firstly, I installed a 9.2 VM, with source. I edited GENERIC and added
> options DDB and GDB, and reinstalled the kernel.
>
>   The disk was then copied, and mdconfig'd/mounted on the host to
> provide access to the just-buit 9.2 kernel syms and sources.
>
>   com2 was set up as a debug port by dropping to the bhyveload prompt and
>
>     hint.uart.1.flags="0x80"
>
>   (this could also have been done in the guest's /boot/loader.conf)
>
>   com2 was then set up in the bhyve command line to point to an nmdm device
>
>    -l com2,/dev/nmdm0A
>
>   The guest probed uart1 as a debug port:
>
> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 flags 0x80 on acpi0
> ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 64
> uart1: fast interrupt
> uart1: debug port (9600,n,8,1)
>
>   After booting, I loaded the the tap device in the guest to provide a
> kld for kgdb to examine:
>
> root at fbsd9-2:~ # kldload if_tap
> root at fbsd9-2:~ # kldstat
> Id Refs Address            Size     Name
>   1    3 0xffffffff80200000 15f92d8  kernel
>   2    1 0xffffffff81a12000 59e9     if_tap.ko
>
>   Now time to try kgdb:
>
> root at fbsd9-2:~ # sysctl debug.kdb.enter=1
> debug.kdb.enter: 0KDB: enter: sysctl debug.kdb.enter
> [ thread pid 577 tid 100054 ]
> Stopped at      kdb_enter+0x3b: movq    $0,0xaf0362(%rip)
> db> gdb
> (ctrl-c will return control to ddb)
> Switching to gdb back-end
> Switching to gdb back-end
>
>   In another window, cd'd to the mounted copy of the 9.2 disk:
>
> kgdb -r /dev/nmdm0B kernel.debug
> ...
> This GDB was configured as "amd64-marcel-freebsd"...Switching to remote
> protocol
> kdb_enter (why=0xffffffff80fbf671 "sysctl", msg=0x80 <Address 0x80 out
> of bounds>) at ../../../kern/subr_kdb.c:441
> 441            kdb_why = KDB_WHY_UNSET;
>
> (kgdb)
>
>   After some experimentation, I found the way to get the correct symbols
> for the kld was to manually specify it:
>
> (kgdb) add-kld /mnt/boot/kernel/if_tap.ko
> add symbol table from file "/mnt/boot/kernel/if_tap.ko.symbols" at
>      .text_addr = 0xffffffff81a12000
>      .rodata.str1.8_addr = 0xffffffff81a13b50
>      .rodata.str1.1_addr = 0xffffffff81a13ddb
>      set_sysinit_set_addr = 0xffffffff81a13f68
>      set_modmetadata_set_addr = 0xffffffff81a13f80
>      set_sysctl_set_addr = 0xffffffff81a13f90
>      set_sysuninit_set_addr = 0xffffffff81a13fc0
>      .data_addr = 0xffffffff81a13fe0
>      .bss_addr = 0xffffffff81a14620
> (y or n) y
>
>   However, 'info sharedlibrary' didn't seem to reflect this:
>
> (kgdb) info sharedlibrary
> From                To                  Syms Read   Shared Object Library
> 0xffffffff81a12000  0xffffffff81a13c04  No /boot/kernel/if_tap.ko.symbols
>
>   Might have been a bug there, since I was able to successfully set
> breakpoints in if_tap routines and have them trigger.
>
>   I did have some trouble getting the source path set up correctly, but
> never fully investigated that: seems like gdb is quite rich in that area
> and it should be possible to get sorted.
>
> later,
>
> Peter.
>
>
>
Thanks for your help.

It showed me one or two things that I didn't know before, and
confirms some of the shortcomings of the current kgdb that others
have noticed in trying to get this work.  Namely, that the
'info sharedlibrary' command doesn't seem to work properly.

-Kurt