[Bug 300480] bhyve: guest sees unmasked AMD QoS CPUID leaf 0x80000020 while Intel RDT leaves are masked; Linux guests boot-loop (resctrl divide-by-zero in PID 1)

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=300480

            Bug ID: 300480
           Summary: bhyve: guest sees unmasked AMD QoS CPUID leaf
                    0x80000020 while Intel RDT leaves are masked; Linux
                    guests boot-loop (resctrl divide-by-zero in PID 1)
           Product: Base System
           Version: 15.1-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: bhyve
          Assignee: virtualization@FreeBSD.org
          Reporter: e@4779.name

Synopsis: bhyve: guest sees unmasked AMD QoS CPUID leaf 0x80000020 while
Intel RDT leaves are masked; Linux guests boot-loop (resctrl
divide-by-zero in PID 1) Product: Base System / Component: bhyve
Environment: FreeBSD 15.1-RELEASE-p6, AMD Ryzen 7 9700X (Zen 5, SVM);
stock /usr/sbin/bhyve; guest Debian 13, kernel 7.2.9+deb13-amd64, 1
vCPU.

Reproduction
------------

Stock bhyve on any Zen 4/Zen 5 host with QoS (CPUID 0x80000020 present,
EBX=0x7e on my 9700X), booting a current Debian 13 guest:

    bhyve -c 1 -m 4096M -H -s 0,hostbridge -s 31,lpc ... <guest>

The guest dies during early init and boot-loops, every time:

    [    0.436531] Oops: divide error: 0000 [#1] SMP NOPTI
    [    0.437625] RIP: 0010:rdt_get_l3_mon_config+0x130/0x220
    [    0.438069] RAX: 0000000000100000 ... RCX: 0000000000000000
    [    0.438069] Call Trace:
     resctrl_arch_late_init+0x3ce/0x5d0
     do_one_initcall+0x5c/0x300
     kernel_init_freeable+0x2a7/0x330
     kernel_init+0x1a/0x130
    "Attempted to kill init! exitcode=0xb"

Workaround: initcall_blacklist=resctrl_arch_late_init on the guest
cmdline (in production here since 2026-11-02).

Mechanism
---------

sys/amd64/vmm/x86.c masks Intel RDT leaves for guests — case
CPUID_0000_0010 zeroes everything, with the comment "Do not report any
Resource Director Technology capabilities. Exposing control of cache or
memory controller resource partitioning to the guest is not at all
sensible."  The AMD equivalent — the QoS leaf 0x80000020 (SMBA/BMEC/ABMC
feature bits in EBX, RMID/COS enumeration in subleaves 0-3) — has no
case and falls into the default passthrough:

    default:
default_leaf:
        atomic_add_long(&bhyve_xcpuids, 1);
        cpuid_count(func, param, regs);   /* raw host values */

Measured on this host (guest vs bare metal, identical values):

    guest 0x80000020.0: eax=0 ebx=0000007e ecx=0 edx=0  
(SMBA,BMEC,ABMC,SDCIAE)
    guest 0x80000020.3: eax=0 ebx=2 ecx=0000007f edx=0    (127 RMIDs)
    host  0x80000020.0/3: same

while in the same guest:

    guest 0x00000010.*: all zero     (explicitly masked)
    guest 0x0000000F.*: all zero
    guest 0x00000007:   EDX=0        (host: 0x10000110 — masked)

Linux 7.2 (Debian 13) maps the passed-through bits to resctrl features
(arch/x86/kernel/cpu/scattered.c):

    { X86_FEATURE_SMBA, CPUID_EBX, 2, 0x80000020, 0 },
    { X86_FEATURE_BMEC, CPUID_EBX, 3, 0x80000020, 0 },
    { X86_FEATURE_ABMC, CPUID_EBX, 5, 0x80000020, 0 },
    { X86_FEATURE_SDCIAE, CPUID_EBX, 6, 0x80000020, 0 },

and arch/x86/kernel/cpu/resctrl/core.c get_rdt_mon_resources() proceeds
on the AMD monitoring path into rdt_get_l3_mon_config(), whose divisors
are derived from the zeroed Intel-side enumeration (leaf 0xF/0x10
globals such as x86_cache_max_rmid / x86_cache_occ_scale). Result:
divide error, PID 1 dies, boot-loop.

The same hole is present on main (no CPUID_8000_0020 case in sys/amd64/
vmm/x86.c as of today).

Fix
---

Mirror the Intel-RDT policy for the AMD QoS leaf: report no
capabilities. Patch attached (zero all four registers for
CPUID_8000_0020, placed next to the other AMD extended-leaf cases).
Consistent with the existing CPUID_0000_0010 rationale: the QoS MSRs are
not emulated either.

Testing: releng/15.1 GENERIC with only this change (vmm.ko rebuilt): -
pre-fix, stock: guest 0x80000020.0 EBX=0x7e, deterministic boot-loop
  (divide error at 0.436531 s, PID 1, every boot);
- post-fix: same guest image, initcall_blacklist removed: full boot to
  login, zero oops; guest 0x80000020.* and 0x10.* both zero, as intended;
- 4-vCPU production VM (bcachefs, NFS server) running under the patched
  vmm.ko since the same day, no regressions.

Attachments: bhyve-amd-qos-cpuid.patch

Note on tooling: this report and the patch were prepared with
LLM-assisted tooling under my supervision; every measurement cited above
was executed on the host described and the raw dumps are kept.

-- 
You are receiving this mail because:
You are the assignee for the bug.