rumpkernel and bhyve: triple faults
Fabian Freyer
fabian.freyer at physik.tu-berlin.de
Tue Mar 6 16:22:08 UTC 2018
Hi Peter,
On 6 Mar 2018, at 16:15, Peter Grehan wrote:
> Exception 14 is a page fault (SDM Vol3 ch 6.15). The exception type is "fault" which means it is delivered at the address it was detected at.
>
> This cascaded very quickly into a triple-fault, so it looks like it could possibly be an issue with the stack. One debug tool you do have is to get a register dump on exit, with 'bhyvectl --get-all --vm=<your vn name>'.
>
> For a page-fault, the virtual address that resulted in the fault will be in the CR2 register.
I don’t see a CR2 register in the output of bhyvectl --get-all, I was looking for that too.
> From the code at the faulting address:
>
> > 0000000000102a50 <cons_init>:
> > 102a50: push rbx
> > 102a51: call 103540 <hypervisor_detect>
> > 102a56: cmp WORD PTR [rip-0x102a5c],0x0 # 2 <current_lwp+0x2>
>
> It's using RIP-relative addressing here, but objdump seems to think this may be an offset in the current_lwp structure - is it possible that may have an uninitialized value ?
I’m pretty sure it’s tooling that’s displaying something off, since hopper is showing me this as
0x0000000000102a56 cmp word [0x2], 0x0
Which is very similar to what r2 is giving me:
;-- cons_init:
0x00102a50 53 push rbx ; /arch/x86:43
0x00102a51 e8ea0a0000 call sym.hypervisor_detect ; /arch/x86:47
0x00102a56 66833da4d5ef. cmp word [0x00000002], 0 ; /arch/x86:62
> (I don't believe this has anything to do with VGA).
Maybe I’m off with my analysis of the actual fault here, but how I understand
the source (assuming compilers work as I would expect, which is not always true)
the values here are initialised from values in the bios data area (which is
zeroed out on bhyve):
#define BIOS_COM1_BASE 0x400
#define BIOS_CRTC_BASE 0x463
...
movw BIOS_COM1_BASE, %bx
movw %bx, bios_com1_base
movw BIOS_CRTC_BASE, %bx
movw %bx, bios_crtc_base
...
/*
* If the BIOS says no CRTC is present use the serial console if
* available.
*/
if (bios_crtc_base == 0)
prefer_serial = 1;
Here’s my full output from bhyvectl --get-all:
ID Length Name
0 128MB sysmem
Address Length Segment Offset Prot Flags
0 128MB sysmem 0 RWX
efer[0] 0x0000000000000500
cr0[0] 0x0000000080010031
cr3[0] 0x000000000010b000
cr4[0] 0x0000000000002620
dr7[0] 0x0000000000000400
rsp[0] 0x0000000000100ff0
rip[0] 0x0000000000102a56
rax[0] 0x0000000000000000
rbx[0] 0x00000000003eaa2b
rcx[0] 0x0000000068622065
rdx[0] 0x0000000020657679
rsi[0] 0x0000000000100fd0
rdi[0] 0x0000000040000000
rbp[0] 0x0000000000000000
r8[0] 0x0000000000100fdc
r9[0] 0x0000000000100fd8
r10[0] 0x0000000000100fd4
r11[0] 0x0000000000000000
r12[0] 0x0000000000000000
r13[0] 0x0000000000000000
r14[0] 0x0000000000000000
r15[0] 0x0000000000000000
rflags[0] 0x0000000000010006
ds desc[0] 0x0000000000000000/0xffffffff/0x0000c093
es desc[0] 0x0000000000000000/0xffffffff/0x0000c093
fs desc[0] 0x0000000000000000/0xffffffff/0x0001c001
gs desc[0] 0x0000000000000000/0xffffffff/0x0001c001
ss desc[0] 0x0000000000000000/0xffffffff/0x0000c093
cs desc[0] 0x0000000000000000/0xffffffff/0x0000a09b
tr desc[0] 0x0000000000000000/0x00000000/0x0000008b
ldtr desc[0] 0x0000000000000000/0x0000ffff/0x00000082
gdtr[0] 0x0000000000378040/0x0000002f
idtr[0] 0x0000000000000000/0x0000ffff
cs[0] 0x0008
ds[0] 0x0018
es[0] 0x0018
fs[0] 0x0000
gs[0] 0x0000
ss[0] 0x0018
tr[0] 0x0000
ldtr[0] 0x0000
cr0_mask[0] 0xffffffff60000020
cr0_shadow[0] 0x0000000000000021
cr4_mask[0] 0xffffffffffe8f800
cr4_shadow[0] 0x0000000000000000
cr3_target_count[0] 0x0000000000000000
cr3_target0[0] 0x0000000000000000
cr3_target1[0] 0x0000000000000000
cr3_target2[0] 0x0000000000000000
cr3_target3[0] 0x0000000000000000
pinbased_ctls[0] 0x000000000000003f
procbased_ctls[0] 0x00000000f51865f2
procbased_ctls2[0] 0x00000000000010a2
gla[0] 0xfffffe0000c41000
gpa[0] 0x0000000000000000
entry_interruption_info[0] 0x0000000000000000
tpr_threshold[0] 0x0000000000000000
instruction_error[0] 0x0000000000000000
exit_ctls[0] 0x000000000033efff
entry_ctls[0] 0x00000000000093ff
host_pat[0] 0x0001050600070406
host_cr0[0] 0x000000008005003b
host_cr3[0] 0x0000000038045054
host_cr4[0] 0x00000000001726e0
host_rip[0] 0xffffffff81435290
host_rsp[0] 0xfffffe003218d700
vmcs_pointer[0] 0xffffffffffffffff
vmcs_exit_interruption_info[0] 0x0000000080000b0e
vmcs_exit_interruption_error[0] 0x0000000000000000
vmcs_guest_interruptibility[0] 0x0000000000000000
vmcs_exit_inst_length[0] 0x00000003
vmcs_exit_qualification[0] 0x0000000000000080
x2apic_state[0] 0
eptp[0] 0x000000003817905e
exception_bitmap[0] 0xffffffff
io_bitmap_a[0] 0
io_bitmap_b[0] 0
tsc_offset[0] 0x0000000000000000
msr_bitmap[0] 0x1adbc000
MSR_TSC [0] R-
MSR_EFER [0] RW
MSR_STAR [0] RW
MSR_LSTAR [0] RW
MSR_CSTAR [0] RW
MSR_SF_MASK [0] RW
MSR_FSBASE [0] RW
MSR_GSBASE [0] RW
MSR_KGSBASE [0] RW
MSR_SYSENTER_CS_MSR [0] RW
MSR_SYSENTER_ESP_MSR[0] RW
MSR_SYSENTER_EIP_MSR[0] RW
vpid[0] 0x0011
guest_pat[0] 0x0000000000000000
guest_sysenter_cs[0] 0
guest_sysenter_sp[0] 0
guest_sysenter_ip[0] 0
exit_reason[0] 0
rtc nvram[000]: 0x34
rtc time 0x5a9ebfd2: Tue Mar 06 16:20:34 2018
Capability "hlt_exit" is set on vcpu 0
Capability "mtrap_exit" is not set on vcpu 0
Capability "pause_exit" is set on vcpu 0
Capability "unrestricted_guest" is set on vcpu 0
Capability "enable_invpcid" is set on vcpu 0
active cpus: 0
suspended cpus: 0
pending: n/a
current: n/a
vcpu0 stats:
number of times in/out was intercepted 0
number of times cpuid was intercepted 3
vm exits due to nested page fault 13
vm exits for instruction emulation 0
number of vm exits for unknown reason 0
number of times astpending at exit 0
number of times idle requested at exit 0
number of vm exits handled in userspace 14
number of times rendezvous pending at exit 0
number of vm exits due to exceptions 3
number of NMIs delivered to vcpu 0
number of ExtINTs delivered to vcpu 0
Resident memory 69632
Wired memory 0
vcpu total runtime 3112708
EOI without any in-service interrupt 0
error interrupts generated by vlapic 0
timer interrupts generated by vlapic 0
corrected machine check interrupts generated by vlapic 0
lvts triggered[0] 0
lvts triggered[1] 0
lvts triggered[2] 0
lvts triggered[3] 0
lvts triggered[4] 0
lvts triggered[5] 0
lvts triggered[6] 0
ipis sent to vcpu[0] 0
ipis sent to vcpu[1] 0
ipis sent to vcpu[2] 0
ipis sent to vcpu[3] 0
ipis sent to vcpu[4] 0
ipis sent to vcpu[5] 0
ipis sent to vcpu[6] 0
ipis sent to vcpu[7] 0
ipis sent to vcpu[8] 0
ipis sent to vcpu[9] 0
ipis sent to vcpu[10] 0
ipis sent to vcpu[11] 0
ipis sent to vcpu[12] 0
ipis sent to vcpu[13] 0
ipis sent to vcpu[14] 0
ipis sent to vcpu[15] 0
number of ticks vcpu was idle 0
vcpu migration across host cpus 1
total number of vm exits 19
vm exits due to external interrupt 0
Number of vpid invalidations saved 0
Number of vpid invalidations done 1
number of times hlt was intercepted 0
number of times %cr access was intercepted 0
number of times rdmsr was intercepted 0
number of times wrmsr was intercepted 0
number of monitor trap exits 0
number of times pause was intercepted 0
vm exits due to interrupt window opening 0
vm exits due to nmi window opening 0
Fabian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 882 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-virtualization/attachments/20180306/de3621d3/attachment.sig>
More information about the freebsd-virtualization
mailing list