bhyve: centos 7.1 with multiple virtual processors

[Andriy Gapon]

On 15/07/2015 10:02, Neel Natu wrote:
> 
> Could you update the host with the following patch?
> https://people.freebsd.org/~neel/patches/ktr_stray_nmi.patch
> 
> It enables KTR_GEN logging by default and turns it off when a VM-exit
> due to NMI is detected. If there is a pattern to the NMIs then it can
> be correlated with the guest activity around that time.

Neel,

sorry that it took so long to gather the additional data.
Yesterday I got another host NMI while running a VM with KTR turned on.
I panic-ed the system from the debugger prompt after the NMI.  Below is several
dozen of last messages taken from the crash dump.  It does not look like there
was a guest NMI or, at least, it didn't get a chance to be logged.  Please let
me know if the full log dump would be more useful.  Initially, the last reported
address 0x7f8c1185129b looked suspicious to me, but grepping the dump for
'extintr' I see many addresses like that.

$ ktrdump -N /boot/kernel.dn/kernel -M /var/crash/vmcore.21 | head -40
index  trace
------ -----
253313 vm centos7[0]: Resume execution at 0x7f8c1185129b
253312 vm centos7[0]: vmcb clean 0x3bf
253311 vm centos7[0]: handled extintr vmexit at 0x7f8c1185129b/0
253310 vm centos7[0]: Resume execution at 0xffffffff8105059c
253309 vm centos7[0]: vmcb clean 0x3bf
253308 vm centos7[0]: Guest interrupt blocking cleared due to rip change:
0xffffffff81050596/0xffffffff8105059c
253307 vm centos7[0]: vcpu state changed from frozen to running
253306 vm centos7[0]: nextrip updated to 0xffffffff8105059c after instruction
decoding
253305 vm centos7[0]: inst_emul fault accessing gpa 0xfed000f0
253304 vm centos7[0]: vcpu state changed from running to frozen
253303 vm centos7[0]: unhandled nptfault vmexit at 0xffffffff81050596/0
253302 vm centos7[0]: inst_emul fault for gpa 0xfed000f0/0x100000004 at rip
0xffffffff81050596
253301 vm centos7[0]: Resume execution at 0xffffffff8105058f
253300 vm centos7[0]: vmcb clean 0x3bf
253299 vm centos7[0]: Guest interrupt blocking cleared due to rip change:
0xffffffff8105058d/0xffffffff8105058f
253298 vm centos7[0]: vcpu state changed from frozen to running
253297 vm centos7[0]: nextrip updated to 0xffffffff8105058f after instruction
decoding
253296 vm centos7[0]: inst_emul fault accessing gpa 0xfed00108
253295 vm centos7[0]: vcpu state changed from running to frozen
253294 vm centos7[0]: unhandled nptfault vmexit at 0xffffffff8105058d/0
253293 vm centos7[0]: inst_emul fault for gpa 0xfed00108/0x100000006 at rip
0xffffffff8105058d
253292 vm centos7[0]: Resume execution at 0xffffffff8105057c
253291 vm centos7[0]: vmcb clean 0x3bf
253290 vm centos7[0]: Guest interrupt blocking cleared due to rip change:
0xffffffff81050576/0xffffffff8105057c
253289 vm centos7[0]: vcpu state changed from frozen to running
253288 vm centos7[0]: nextrip updated to 0xffffffff8105057c after instruction
decoding
253287 vm centos7[0]: inst_emul fault accessing gpa 0xfed000f0
253286 vm centos7[0]: vcpu state changed from running to frozen
253285 vm centos7[0]: unhandled nptfault vmexit at 0xffffffff81050576/0
253284 vm centos7[0]: inst_emul fault for gpa 0xfed000f0/0x100000004 at rip
0xffffffff81050576
253283 vm centos7[0]: Resume execution at 0xffffffff8104accc
253282 vm centos7[0]: vmcb clean 0x3b7
253281 vm centos7[0]: Clearing V_IRQ interrupt injection
253280 vm centos7[0]: Guest interrupt blocking cleared due to rip change:
0xffffffff8104acc6/0xffffffff8104accc
253279 vm centos7[0]: vcpu state changed from frozen to running
253278 vm centos7[0]: vlapic_update_ppr 0x00
253277 vm centos7[0]: vlapic_process_eoi isr7 0x00000000
253276 vm centos7[0]: vlapic_process_eoi isr6 0x00000000

And a snippet from the stack-trace:
...
#9  0xffffffff8081ed84 in trap (frame=0xfffffe01dc531f30) at
/usr/src/sys/amd64/amd64/trap.c:372
#10 0xffffffff808074e3 in nmi_calltrap () at
/usr/src/sys/libkern/explicit_bzero.c:28
#11 0xffffffff81c92327 in enable_gintr () at
/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895
#12 0xffffffff81c91c57 in svm_vmrun (arg=0xfffffe006d29c000, vcpu=0,
rip=-2130377316, pmap=0xfffff801e4771138, evinfo=0xfffffe02b8c45820) at
/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:2022
#13 0xffffffff81c7661e in vm_run (vm=0xfffffe0076277000,
vmrun=0xfffff8013fe43000) at /usr/src/sys/modules/vmm/../../amd64/vmm/vmm.c:1643
#14 0xffffffff81c79b78 in vmmdev_ioctl (cdev=<value optimized out>, cmd=<value
optimized out>, data=0xfffff8013fe43000 "", fflag=-1, td=0x0) at
/usr/src/sys/modules/vmm/../../amd64/vmm/vmm_dev.c:392
...

-- 
Andriy Gapon