Fail to boot FreeBSD12 or 13 on POWER9 under PowerKVM

Fri Apr 26 16:17:06 UTC 2019

Hi Alexey,

On Fri, 5 Apr 2019 15:07:58 +1100
Alexey Kardashevskiy <aik at ozlabs.ru> wrote:

> Hi!
> 
> I am trying a freebsd guest on a POWER9 (pvr=004e1201) host with
> linux+kvm (5.1.0-rc2) and qemu (upstream, 4.0) and something goes
> wrong
> - it crashes as (the full output is below):
> 
> =====
> run_interrupt_driven_hooks: still waiting after 300 seconds for
> xpt_config random: unblocking device.
> panic: run_interrupt_driven_config_hooks: waited too long
> cpuid = 0
> time = 361
> KDB: stack backtrace:
> 0xe000000000008660: at .kdb_backtrace+0x5c
> 0xe000000000008790: at .vpanic+0x1b4
> 0xe000000000008850: at .panic+0x38
> 0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194
> 0xe0000000000089e0: at .mi_startup+0x1f8
> 0xe000000000008a80: at btext+0xc4
> KDB: enter: panic
> [ thread pid 0 tid 100000 ]
> Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
> db>  
> =====
> 
> 
> The systems were installed from:
> FreeBSD-12.0-RELEASE-powerpc-powerpc64-dvd1.iso
> FreeBSD-13.0-CURRENT-powerpc-powerpc64-20190307-r344854-disc1.iso
> 
> Everything by default (vt100 terminal, etc), IBM vio-scsi disk and
> cdrom, 8GB RAM, 16MB backing huge pages, the host is running in the
> hash (HPT) mode.
> 
> However the exact same disk images + qemu/slof binary + qemu command
> line + linux kernel do boot on POWER8E (pvr=004b0201) and POWER8NVL
> (pvr=004c0100) to the login prompt.
> 
> 
> I told QEMU to enforce XICS interrupt controller mode, POWER8
> compatibility (although it does not make sense as FreeBSD does not do
> "client-architecture-support" RTAS call), what else can I try?

That seems pretty bizarre.  Is the peripheral list on the working and
non-working the same?

> 
> 
> While at it, FreeBSD is aware of 004b0201 and 004e1201 but it fails to
> recognize 004c0100 (the FreeBSD guest still boots just fine):
> 
> cpu0: Unknown PowerPC CPU revision 0x0100, 3259.00 MHz
> cpu0: Features c4000000<PPC32,PPC64,MMU>
> 
> but in fact architecturally it behaves exactly as IBMPOWER8 (004bxxxx
> or 004d0000).

This just needs another entry in the CPU table.  I can add that
tonight.  It's not entirely cosmetic, because we do make certain
decisions and set capabilities based on the CPU found.

> 
> 
> 
> build/qemu-aikrhel74alt-ppc64/ppc64-softmmu/qemu-system-ppc64 \
> -nodefaults \
> -chardev stdio,id=STDIO0,signal=off,mux=on \
> -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \
> -mon id=MON0,chardev=STDIO0,mode=readline -nographic -vga none \
> img/freebsd12-64G.qcow2 -enable-kvm \
> -smp 1 -mem-prealloc -mem-path qemu_hp_16M_node0 -m 8G \
> -machine \
> pseries,cap-hpt-max-page-size=16M,cap-cfpc=broken,max-cpu-compat=power8,ic-mode=xics
> \
> -snapshot -bios ./slof.bin \
> -L /home/aik/t/qemu-ppc64-bios/ \
> -trace events=qemu_trace_events -d guest_errors \
> -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.8324 \
> -mon chardev=SOCKET0,mode=control
> 
> 
> SLOF
> **********************************************************************
> QEMU Starting Build Date = Apr  5 2019 13:01:51
>  FW Version = git-a5b428e1c1eae703
>  Press "s" to enter Open Firmware.
> 
> Populating /vdevice methods
> Populating /vdevice/nvram at 71000000
> Populating /vdevice/v-scsi at 71000001
>        SCSI: Looking for devices
>           8000000000000000 DISK     : "QEMU     QEMU HARDDISK    2.5+"
> Populating /vdevice/vty at 71000110
> Populating /pci at 800000020000000
> No NVRAM common partition, re-initializing...
> Scanning USB
> Using default console: /vdevice/vty at 71000110
> 
>   Welcome to Open Firmware
> 
>   Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
>   This program and the accompanying materials are made available
>   under the terms of the BSD License available at
>   http://www.opensource.org/licenses/bsd-license.php
> 
> 
> Trying to load:
> from: /vdevice/v-scsi at 71000001/disk at 8000000000000000 ... Successfully
> loaded
> 
> >> FreeBSD/powerpc Open Firmware boot block  
>    Boot path:   /vdevice/v-scsi at 71000001/disk at 8000000000000000
>    Boot loader: /boot/loader
>    Boot volume:   /vdevice/v-scsi at 71000001/disk at 8000000000000000:2
> Consoles: Open Firmware console
> 
> FreeBSD/powerpc64 Open Firmware loader, Revision 0.1
> Memory: 8388608KB
> Booted from: /vdevice/v-scsi at 71000001/disk at 8000000000000000
> 
> Loading /boot/defaults/loader.conf
> /boot/kernel/kernel data=0x136c550+0x4aa1f0
> syms=[0x8+0x165c78+0x8+0x1643cb] /boot/entropy size=0x1000
> 
> Hit [Enter] to boot immediately, or any other key for command prompt.
> Booting [/boot/kernel/kernel]...
> Kernel entry at 0x1025d0 ...
> ---<<BOOT>>---
> Copyright (c) 1992-2018 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
> 1994 The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 12.0-RELEASE r341666 GENERIC powerpc
> gcc version 4.2.1 20070831 patched [FreeBSD]
> VT: init without driver.
> cpu0: IBM POWER9 revision 2.1, 2250.00 MHz
> cpu0: Features
> dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE>
> cpu0: Features2
> eee00000<ARCH207,HTM,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN>
> real memory  = 8544436224 (8148 MB)
> avail memory = 8258940928 (7876 MB)
> random: unblocking device.
> random: entropy device external interface
> kbd0 at kbdmux0
> random: registering fast source PowerISA DARN random number generator
> random: fast provider: "PowerISA DARN random number generator"
> ofwbus0: <Open Firmware Device Tree> on nexus0
> xicp0: <External Interrupt Presentation Controller> on ofwbus0
> xicp0: Handling CPUs 0-7
> vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0
> uart0: <POWER Hypervisor Virtual Serial Port> irq 16781585 on vdevice0
> vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781570 on vdevice0
> vscsi0: Queue depth 22 commands
> pcib0: <RTAS Host-PCI bridge> on ofwbus0
> pci0: <POWER Hypervisor PCI bus> on pcib0
> cpulist0: <Open Firmware CPU Group> on ofwbus0
> cpu0: <Open Firmware CPU> on cpulist0
> rtas0: <Run-Time Abstraction Services> on ofwbus0
> rtas0: registered as a time-of-day clock, resolution 0.002000s
> Timecounter "timebase" frequency 512000000 Hz quality 0
> Event timer "decrementer" frequency 512000000 Hz quality 1000
> Timecounters tick every 1.000 msec
> usb_needs_explore_all: no devclass
> run_interrupt_driven_hooks: still waiting after 60 seconds for
> xpt_config run_interrupt_driven_hooks: still waiting after 120
> seconds for xpt_config run_interrupt_driven_hooks: still waiting
> after 180 seconds for xpt_config run_interrupt_driven_hooks: still
> waiting after 240 seconds for xpt_config run_interrupt_driven_hooks:
> still waiting after 300 seconds for xpt_config random: unblocking
> device. panic: run_interrupt_driven_config_hooks: waited too long
> cpuid = 0
> time = 361
> KDB: stack backtrace:
> 0xe000000000008660: at .kdb_backtrace+0x5c
> 0xe000000000008790: at .vpanic+0x1b4
> 0xe000000000008850: at .panic+0x38
> 0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194
> 0xe0000000000089e0: at .mi_startup+0x1f8
> 0xe000000000008a80: at btext+0xc4
> KDB: enter: panic
> [ thread pid 0 tid 100000 ]
> Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
> db>  
> 
> 
> 

It's possible FreeBSD is making a decision based on certain assumptions
of the CPU type in the VM, but I'm not sure at all.  I hope someone
else might know more about the pseries FreeBSD code than I do.

- Justin