rumpkernel and bhyve: triple faults
Fabian Freyer
fabian.freyer at physik.tu-berlin.de
Tue Mar 6 06:52:23 UTC 2018
Hello lists,
I’m currently playing around with getting rump kernels built with rumpkernel(7) running on FreeBSD’s bhyve(4). I’m using a custom boot loader [1] which builds on some patches to bhyveload / user boot [2].
To test, I’m using a simple “helloer” unikernel from the tutorial [3]:
#include <stdio.h>
#include <unistd.h>
int
main()
{
printf("Hello, Rumprun ... I'm feeling tired\n");
sleep(2);
printf("much better!\n");
return 0;
}
I’m compiling this for the hw_virtio platform:
x86_64-rumprun-netbsd-gcc -o helloer-rumprun helloer.c
rumprun-bake hw_virtio helloer-rumprun.elf helloer-rumprun
Starting this kernel in bhyve gives me the following VM Exit:
From the Intel SDM 3D, Appendix C [4], VM Exit 2 is a triple fault.
To further debug this, I’m using the ktr(4) support introduced in FreeBSD r276098 [5].
I’ve appended a full KTR trace further down in my mail, but I’ll go along the trace here with some commentary in execution order, and attempt to annotate the source code lines from rumpkernel:
0x104000 is the entry address given in rump kernel’s multi boot header.
0000000000104000 <_start>:
632 0 350886885990 vm testing[0]: Setting nextrip to 0x104000
636 0 350886961656 vm testing[0]: Resume execution at 0x104000
This is the cpuid instruction in x86_cpuid which is called from hypervisor_detect:
rumprun/platform/hw/arch/x86/x86_subr.c: 91
0000000000102f90 <x86_cpuid>:
102f90: 49 89 d2 mov r10,rdx
102f93: 49 89 c9 mov r9,rcx
102f96: 53 push rbx
102f97: 89 f8 mov eax,edi
102f99: 0f a2 cpuid
rumprun/platform/hw/arch/x86/hypervisor.c: 38
637 0 350887061536 vm testing[0]: cpuid 0,0x100fd8
638 0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f99
639 0 350887079146 vm testing[0]: Resume execution at 0x102f9b
rumprun/platform/hw/arch/x86/hypervisor.c: 40
640 0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8
641 0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f99
642 0 350887121556 vm testing[0]: Resume execution at 0x102f9b
rumprun/platform/hw/arch/x86/hypervisor.c: 51
643 0 350887143374 vm testing[0]: cpuid 0x40000000,0x100fd8
644 0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f99
645 0 350887149146 vm testing[0]: Resume execution at 0x102f9b
The next vexit is in cons_init, directly after the call to hypervisor_detect.
0000000000102a50 <cons_init>:
102a50: 53 push rbx
102a51: e8 ea 0a 00 00 call 103540 <hypervisor_detect>
102a56: 66 83 3d a4 d5 ef ff cmp WORD PTR [rip-0x102a5c],0x0 # 2 <current_lwp+0x2>
Due to compiler optimisations, the check here isn’t the
(hypervisor == HYPERVISOR_XEN) check directly after the call to hypervisor_detect,
but the check (bios_crtc_base == 0) in
rumprun/platform/hw/arch/x86/cons.c:59:
649 0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56
Therefore, I’m assuming this is the origin of the fault.
Tracking down bios_crtc_base, I find that it’s loaded in
rumprun/platform/hw/arch/amd64/locore.S:70:
/* save BIOS data area values */
movw BIOS_COM1_BASE, %bx
movw %bx, bios_com1_base
movw BIOS_CRTC_BASE, %bx
movw %bx, bios_crtc_base
Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the bhyve
device node in /dev/vmm with xxd(1), I find the words at these addresses to be
Uninitialised:
00000400: 0000 ..
00000483: 0000 ..
I’m not sure where to go from here. Is this a bug in bhyve(4), should these
values be initialised somehow, or should I patch rumpkernel(7) to skip this check
when running on bhyve(4)?
Fabian
Full KTR trace:
index cpu timestamp trace
------ --- ---------------- -----
673 0 350887364560 vm testing[0]: vcpu state changed from frozen to idle
672 0 350887363766 vm testing[0]: retu 0/14
671 0 350887362932 vm testing[0]: All vcpus suspended
670 0 350887357212 vm testing[0]: vcpu state changed from running to frozen
669 0 350887350444 vm testing[0]: returning from vmx_run: exitcode 14
668 0 350887348806 vm testing: virtual machine successfully suspended 4
667 0 350887347712 vm testing[0]: triple fault: info1(0x80000b08), info2(0x80000b0e)
666 0 350887347590 vm testing[0]: Exception 14 delivered: 0x80000b0e
665 0 350887346312 vm testing[0]: handled exception vmexit at 0x102a56
664 0 350887346178 vm testing[0]: Exception 14 pending
663 0 350887346060 vm testing[0]: Setting intr_shadow to 0 succeeded
662 0 350887341876 vm testing[0]: Reflecting exception 14/0 into the guest
661 0 350887339570 vm testing[0]: vm_exit_intinfo: info1(0x80000b08)
660 0 350887315326 vm testing[0]: Resume execution at 0x102a56
659 0 350887311168 vm testing[0]: vm_entry_intinfo: info1(0x80000b0e), info2(0x80000b0e), retinfo(0x80000b08)
658 0 350887310988 vm testing[0]: Exception 14 delivered: 0x80000b0e
657 0 350887309700 vm testing[0]: handled exception vmexit at 0x102a56
656 0 350887309570 vm testing[0]: Exception 14 pending
655 0 350887309442 vm testing[0]: Setting intr_shadow to 0 succeeded
654 0 350887305126 vm testing[0]: Reflecting exception 14/0 into the guest
653 0 350887302436 vm testing[0]: vm_exit_intinfo: info1(0x80000b0e)
652 0 350887248280 vm testing[0]: Resume execution at 0x102a56
651 0 350887184160 vm testing[0]: vm_entry_intinfo: info1(0), info2(0x80000b0e), retinfo(0x80000b0e)
650 0 350887184040 vm testing[0]: Exception 14 delivered: 0x80000b0e
649 0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56
648 0 350887182374 vm testing[0]: Exception 14 pending
647 0 350887182240 vm testing[0]: Setting intr_shadow to 0 succeeded
646 0 350887176682 vm testing[0]: Reflecting exception 14/0 into the guest
645 0 350887149146 vm testing[0]: Resume execution at 0x102f9b
644 0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f99
643 0 350887143374 vm testing[0]: cpuid 0x40000000,0x100fd8
642 0 350887121556 vm testing[0]: Resume execution at 0x102f9b
641 0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f99
640 0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8
639 0 350887079146 vm testing[0]: Resume execution at 0x102f9b
638 0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f99
637 0 350887061536 vm testing[0]: cpuid 0,0x100fd8
636 0 350886961656 vm testing[0]: Resume execution at 0x104000
635 0 350886900428 vm testing[0]: vcpu state changed from frozen to running
634 0 350886895942 vm testing[0]: vcpu state changed from idle to frozen
633 0 350886886630 vm testing[0]: vcpu state changed from frozen to idle
632 0 350886885990 vm testing[0]: Setting nextrip to 0x104000
631 0 350886802596 vm testing[0]: vcpu state changed from idle to frozen
630 3 350886414966 vm testing[0]: vcpu state changed from frozen to idle
629 3 350886414184 vm testing[0]: activated
628 3 350886412730 vm testing[0]: vcpu state changed from idle to frozen
627 3 350886137831 vm testing[0]: vcpu state changed from frozen to idle
626 3 350885989045 vm testing[0]: vcpu state changed from idle to frozen
625 3 350885773367 vm testing: RTC time set to 0x5a9e154e
624 3 350885773017 vm testing: Updating RTC base uptime from 0x31c2762314 to 0x8bd6b051a2
623 3 350885772543 vm testing: Updating RTC secs from 0x5a9e14f4 to 0x5a9e154e
622 3 350885017699 vm testing: RTC nvram write 0 to offset 0x5d
621 3 350885015953 vm testing: RTC nvram write 0 to offset 0x5c
620 3 350885013951 vm testing: RTC nvram write 0 to offset 0x5b
619 3 350885010615 vm testing: RTC nvram write 0x1 to offset 0x35
618 3 350885008163 vm testing: RTC nvram write 0 to offset 0x34
[1] https://github.com/fubarnetes/bhyve-multiboot
[2] https://reviews.FreeBSD.org/D14473
[3] https://github.com/rumpkernel/wiki/wiki/Tutorial%3A-Building-Rumprun-Unikernels#2-building-applications
[4] https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3d-part-4-manual.pdf
[5] https://svnweb.freebsd.org/base?view=revision&revision=276098
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 882 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-virtualization/attachments/20180306/4ee15a25/attachment.sig>
More information about the freebsd-virtualization
mailing list