[Bug 296872] bhyve guest hangs at start_init (100%+ vCPU spin in lock_delay) on AMD Ryzen (Zen2/SVM), 15.1-RELEASE host; identical guest boots on 14.3 Intel host

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296872

            Bug ID: 296872
           Summary: bhyve guest hangs at start_init (100%+ vCPU spin in
                    lock_delay) on AMD Ryzen (Zen2/SVM), 15.1-RELEASE
                    host; identical guest boots on 14.3 Intel host
           Product: Base System
           Version: 15.1-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: bhyve
          Assignee: virtualization@FreeBSD.org
          Reporter: ozgur@kazancci.com

Created attachment 272954
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=272954&action=edit
The zip file contains: 01-host-info.txt - 02-bhyve-cmdline.txt -
03-bhyvectl-getall-1 - 03-bhyvectl-getall-2 - 03-bhyvectl-getall-3 -
04-guest-console

Overview
--------
A FreeBSD 15.1-RELEASE amd64 guest under bhyve reaches userland (the kernel
prints
"start_init: trying /sbin/init") and then hangs permanently. The bhyve process
pins
~100% CPU per vCPU (~213% for 4 vCPUs, ~120% for 1 vCPU). The guest never
reaches
multi-user; no login prompt, no network. The identical guest disk image boots
normally on a different host (Intel Haswell, 14.3-RELEASE), isolating the
problem to
the 15.1 host's vmm/SVM path on AMD.

bhyvectl shows the guest spinning in lock_delay() with millions of intercepted
PAUSE
instructions and no interrupt-window activity — the classic signature of a
waited-for
timer/interrupt never being delivered to the guest.

Host environment
----------------
- Host: FreeBSD 15.1-RELEASE-p1 amd64 (GENERIC)
- CPU:  AMD Ryzen 7 3700X (Zen2), Family=0x17 Model=0x71, AMD
Features2=<...,SVM,...>
- vmm.ko loaded; managed via vm-bhyve 1.7.4 (also reproduced with 1.7.99.87)
- hw.vmm.svm.num_asids=32768, features=1293567, vmcb_clean=1023
- hw.vmm.amdvi.enable=0 (no PCI passthrough / IOMMU in use)

Guest
-----
- FreeBSD 15.1-RELEASE amd64 GENERIC (official 15.1 dist sets; also repro from
the
  stock 15.1 bootonly install ISO's own installer kernel).
- loader: bhyveload (NOT UEFI -> not an edk2 firmware issue).
- Reproduced with disk = nvme AND virtio-blk; backing = zvol AND raw file;
cpu=4 AND cpu=1.

bhyve invocation (verbatim, from vm-bhyve)
------------------------------------------
bhyve -c 4 -m 8G -AHPw -U <uuid> -u \
  -s 0,hostbridge -s 31,lpc \
  -s 0:4:0,virtio-blk,/zroot/vm/web1/disk0.img \
  -s 0:5:0,virtio-net,tap0,mac=... \
  -l com1,/dev/nmdm-web1.1A web1
(-H yield-on-HLT and -w ignore-unimplemented-MSR are both present.)

Steps to reproduce
------------------
1. On the AMD Ryzen 3700X / 15.1-RELEASE-p1 host, create a FreeBSD 15.1 amd64
guest
   (bhyveload loader, virtio-blk or nvme disk, virtio-net).
2. Boot the guest.

Actual result
-------------
Kernel loads, probes devices, mounts root, prints:
    Trying to mount root from ufs:/dev/ufs/rootfs []...
    start_init: trying /sbin/init
...then NO further output. bhyve spins ~100% CPU per vCPU indefinitely. No
/etc/rc
output, no login, no network. Guest is unresponsive (destroyable via bhyvectl
--destroy).

Expected result
---------------
Guest completes /etc/rc and reaches login (as it does on a 14.3 Intel host with
the
same image).

Diagnostic evidence (bhyvectl --get-all, twice ~4s apart while hung)
-------------------------------------------------------------------
  rip[0] 0xffffffff80bd77b2  (sample 1)
  rip[0] 0xffffffff80bd77b0  (sample 2)
    -> RIP oscillates within 2 bytes == tight PAUSE spin loop.
    -> nm on guest kernel: 0xffffffff80bd77a0 = lock_delay()  => spinning in
lock_delay().
  number of vm exits due to exceptions   : 0        ->  0        (rules out
MSR/#GP loop)
  vm exits due to external interrupt     : 11390    ->  13712    (~580/s)
  number of times pause was intercepted  : 9415553  ->  10025216 (~600k/s)
  vm exits due to interrupt window opening: 0        ->  0
  vm exits due to nested page fault      : 11657    ->  11657    (no growth)
Interpretation: ~600k PAUSE/s with RIP fixed in lock_delay() => guest stuck on
a
spinlock; exceptions=0 => not an MSR loop; interrupt-window=0 with a lock owner
waiting => a timer/callout interrupt is not being delivered; no forward
progress.

Cross-check / isolation (key data point)
----------------------------------------
The SAME 15.1 amd64 guest image, created with the identical procedure, boots
CLEANLY
to a login prompt on: Intel Core i7-4790 (Haswell, VT-x), FreeBSD
14.3-RELEASE-p15.
=> Guest image/config is correct; the fault is on the AMD/15.1 host side.

Additional observations
------------------------
- Reproduced with cpu=1 (rules out AP-launch/SMP-startup-only bugs; points at
the
  interrupt/timer delivery layer).
- Warm reboot AND cold power-cycle of the AMD host do not change behaviour
(expected
  for a deterministic software issue; argues against stuck hardware/firmware
state).
- Preceded by guest-side "nvme: Resetting controller due to a timeout" during
heavy
  fsync I/O — a SYMPTOM (I/O completion interrupts not arriving), not the
cause;
  reproduces afterwards on every boot including with virtio-blk (no nvme).

Workarounds attempted that did NOT help (guest still hangs in lock_delay)
-------------------------------------------------------------------------
guest hw.x2apic_enable=0; guest kern.eventtimer.timer=HPET; cpu=1; nvme &
virtio-blk;
zvol & raw file; vm-bhyve 1.7.4 & 1.7.99.87; vmm kldunload/kldload.
(hw.vmm.svm.num_asids=1 made the guest fail to run entirely — inconclusive.)

Probable root cause
-------------------
A regression in the 15.x vmm(4) AMD SVM path delivering a timer/APIC interrupt
(or in
event-injection / interrupt-window handling) to the guest. The guest reaches
early
userland, a thread blocks on a callout/timer, the interrupt is never injected,
the
lock is never released, another context spins in lock_delay() forever. The 15.x
vmm
refactor is known to have produced at least one AMD-relevant regression
(FreeBSD-EN-25:20.vmm, PCI-passthru IOMMU); this appears to be a separate
regression
in the same subsystem on the non-passthru interrupt path.

Willing to test patches / a rebuilt vmm.ko and provide additional bhyvectl
--get-all
dumps or a bhyve -G gdb-stub session on request.

-- 
You are receiving this mail because:
You are the assignee for the bug.