Fail to boot FreeBSD12 or 13 on POWER9 under PowerKVM

Fri Apr 5 04:08:07 UTC 2019

Hi!

I am trying a freebsd guest on a POWER9 (pvr=004e1201) host with
linux+kvm (5.1.0-rc2) and qemu (upstream, 4.0) and something goes wrong
- it crashes as (the full output is below):

=====
run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config
random: unblocking device.
panic: run_interrupt_driven_config_hooks: waited too long
cpuid = 0
time = 361
KDB: stack backtrace:
0xe000000000008660: at .kdb_backtrace+0x5c
0xe000000000008790: at .vpanic+0x1b4
0xe000000000008850: at .panic+0x38
0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194
0xe0000000000089e0: at .mi_startup+0x1f8
0xe000000000008a80: at btext+0xc4
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
db>
=====

The systems were installed from:
FreeBSD-12.0-RELEASE-powerpc-powerpc64-dvd1.iso
FreeBSD-13.0-CURRENT-powerpc-powerpc64-20190307-r344854-disc1.iso

Everything by default (vt100 terminal, etc), IBM vio-scsi disk and
cdrom, 8GB RAM, 16MB backing huge pages, the host is running in the hash
(HPT) mode.

However the exact same disk images + qemu/slof binary + qemu command
line + linux kernel do boot on POWER8E (pvr=004b0201) and POWER8NVL
(pvr=004c0100) to the login prompt.

I told QEMU to enforce XICS interrupt controller mode, POWER8
compatibility (although it does not make sense as FreeBSD does not do
"client-architecture-support" RTAS call), what else can I try?

While at it, FreeBSD is aware of 004b0201 and 004e1201 but it fails to
recognize 004c0100 (the FreeBSD guest still boots just fine):

cpu0: Unknown PowerPC CPU revision 0x0100, 3259.00 MHz
cpu0: Features c4000000<PPC32,PPC64,MMU>

but in fact architecturally it behaves exactly as IBMPOWER8 (004bxxxx or
004d0000).

build/qemu-aikrhel74alt-ppc64/ppc64-softmmu/qemu-system-ppc64 \
-nodefaults \
-chardev stdio,id=STDIO0,signal=off,mux=on \
-device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \
-mon id=MON0,chardev=STDIO0,mode=readline -nographic -vga none \
img/freebsd12-64G.qcow2 -enable-kvm \
-smp 1 -mem-prealloc -mem-path qemu_hp_16M_node0 -m 8G \
-machine \
pseries,cap-hpt-max-page-size=16M,cap-cfpc=broken,max-cpu-compat=power8,ic-mode=xics
\
-snapshot -bios ./slof.bin \
-L /home/aik/t/qemu-ppc64-bios/ \
-trace events=qemu_trace_events -d guest_errors \
-chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.8324 \
-mon chardev=SOCKET0,mode=control

SLOF **********************************************************************
QEMU Starting
 Build Date = Apr  5 2019 13:01:51
 FW Version = git-a5b428e1c1eae703
 Press "s" to enter Open Firmware.

Populating /vdevice methods
Populating /vdevice/nvram at 71000000
Populating /vdevice/v-scsi at 71000001
       SCSI: Looking for devices
          8000000000000000 DISK     : "QEMU     QEMU HARDDISK    2.5+"
Populating /vdevice/vty at 71000110
Populating /pci at 800000020000000
No NVRAM common partition, re-initializing...
Scanning USB
Using default console: /vdevice/vty at 71000110

  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Trying to load:  from: /vdevice/v-scsi at 71000001/disk at 8000000000000000 ...
  Successfully loaded

>> FreeBSD/powerpc Open Firmware boot block
   Boot path:   /vdevice/v-scsi at 71000001/disk at 8000000000000000
   Boot loader: /boot/loader
   Boot volume:   /vdevice/v-scsi at 71000001/disk at 8000000000000000:2
Consoles: Open Firmware console

FreeBSD/powerpc64 Open Firmware loader, Revision 0.1
Memory: 8388608KB
Booted from: /vdevice/v-scsi at 71000001/disk at 8000000000000000

Loading /boot/defaults/loader.conf
/boot/kernel/kernel data=0x136c550+0x4aa1f0 syms=[0x8+0x165c78+0x8+0x1643cb]
/boot/entropy size=0x1000

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...
Kernel entry at 0x1025d0 ...
---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RELEASE r341666 GENERIC powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
VT: init without driver.
cpu0: IBM POWER9 revision 2.1, 2250.00 MHz
cpu0: Features
dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE>
cpu0: Features2
eee00000<ARCH207,HTM,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN>
real memory  = 8544436224 (8148 MB)
avail memory = 8258940928 (7876 MB)
random: unblocking device.
random: entropy device external interface
kbd0 at kbdmux0
random: registering fast source PowerISA DARN random number generator
random: fast provider: "PowerISA DARN random number generator"
ofwbus0: <Open Firmware Device Tree> on nexus0
xicp0: <External Interrupt Presentation Controller> on ofwbus0
xicp0: Handling CPUs 0-7
vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0
uart0: <POWER Hypervisor Virtual Serial Port> irq 16781585 on vdevice0
vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781570 on vdevice0
vscsi0: Queue depth 22 commands
pcib0: <RTAS Host-PCI bridge> on ofwbus0
pci0: <POWER Hypervisor PCI bus> on pcib0
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
rtas0: <Run-Time Abstraction Services> on ofwbus0
rtas0: registered as a time-of-day clock, resolution 0.002000s
Timecounter "timebase" frequency 512000000 Hz quality 0
Event timer "decrementer" frequency 512000000 Hz quality 1000
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config
run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config
run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config
run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config
run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config
random: unblocking device.
panic: run_interrupt_driven_config_hooks: waited too long
cpuid = 0
time = 361
KDB: stack backtrace:
0xe000000000008660: at .kdb_backtrace+0x5c
0xe000000000008790: at .vpanic+0x1b4
0xe000000000008850: at .panic+0x38
0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194
0xe0000000000089e0: at .mi_startup+0x1f8
0xe000000000008a80: at btext+0xc4
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
db>

-- 
Alexey