Fail to boot FreeBSD12 or 13 on POWER9 under PowerKVM

Mon Apr 29 07:24:19 UTC 2019

On 27/04/2019 02:16, Justin Hibbits wrote:
> Hi Alexey,
> 
> On Fri, 5 Apr 2019 15:07:58 +1100
> Alexey Kardashevskiy <aik at ozlabs.ru> wrote:
> 
>> Hi!
>>
>> I am trying a freebsd guest on a POWER9 (pvr=004e1201) host with
>> linux+kvm (5.1.0-rc2) and qemu (upstream, 4.0) and something goes
>> wrong
>> - it crashes as (the full output is below):
>>
>> =====
>> run_interrupt_driven_hooks: still waiting after 300 seconds for
>> xpt_config random: unblocking device.
>> panic: run_interrupt_driven_config_hooks: waited too long
>> cpuid = 0
>> time = 361
>> KDB: stack backtrace:
>> 0xe000000000008660: at .kdb_backtrace+0x5c
>> 0xe000000000008790: at .vpanic+0x1b4
>> 0xe000000000008850: at .panic+0x38
>> 0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194
>> 0xe0000000000089e0: at .mi_startup+0x1f8
>> 0xe000000000008a80: at btext+0xc4
>> KDB: enter: panic
>> [ thread pid 0 tid 100000 ]
>> Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
>> db>  
>> =====
>>
>>
>> The systems were installed from:
>> FreeBSD-12.0-RELEASE-powerpc-powerpc64-dvd1.iso
>> FreeBSD-13.0-CURRENT-powerpc-powerpc64-20190307-r344854-disc1.iso
>>
>> Everything by default (vt100 terminal, etc), IBM vio-scsi disk and
>> cdrom, 8GB RAM, 16MB backing huge pages, the host is running in the
>> hash (HPT) mode.
>>
>> However the exact same disk images + qemu/slof binary + qemu command
>> line + linux kernel do boot on POWER8E (pvr=004b0201) and POWER8NVL
>> (pvr=004c0100) to the login prompt.
>>
>>
>> I told QEMU to enforce XICS interrupt controller mode, POWER8
>> compatibility (although it does not make sense as FreeBSD does not do
>> "client-architecture-support" RTAS call), what else can I try?
> 
> That seems pretty bizarre.  Is the peripheral list on the working and
> non-working the same?

Yes.

>>
>>
>> While at it, FreeBSD is aware of 004b0201 and 004e1201 but it fails to
>> recognize 004c0100 (the FreeBSD guest still boots just fine):
>>
>> cpu0: Unknown PowerPC CPU revision 0x0100, 3259.00 MHz
>> cpu0: Features c4000000<PPC32,PPC64,MMU>
>>
>> but in fact architecturally it behaves exactly as IBMPOWER8 (004bxxxx
>> or 004d0000).
> 
> This just needs another entry in the CPU table.  I can add that
> tonight.  It's not entirely cosmetic, because we do make certain
> decisions and set capabilities based on the CPU found.
> 
>>
>>
>>
>> build/qemu-aikrhel74alt-ppc64/ppc64-softmmu/qemu-system-ppc64 \
>> -nodefaults \
>> -chardev stdio,id=STDIO0,signal=off,mux=on \
>> -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \
>> -mon id=MON0,chardev=STDIO0,mode=readline -nographic -vga none \
>> img/freebsd12-64G.qcow2 -enable-kvm \
>> -smp 1 -mem-prealloc -mem-path qemu_hp_16M_node0 -m 8G \
>> -machine \
>> pseries,cap-hpt-max-page-size=16M,cap-cfpc=broken,max-cpu-compat=power8,ic-mode=xics
>> \
>> -snapshot -bios ./slof.bin \
>> -L /home/aik/t/qemu-ppc64-bios/ \
>> -trace events=qemu_trace_events -d guest_errors \
>> -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.8324 \
>> -mon chardev=SOCKET0,mode=control
>>
>>
>> SLOF
>> **********************************************************************
>> QEMU Starting Build Date = Apr  5 2019 13:01:51
>>  FW Version = git-a5b428e1c1eae703
>>  Press "s" to enter Open Firmware.
>>
>> Populating /vdevice methods
>> Populating /vdevice/nvram at 71000000
>> Populating /vdevice/v-scsi at 71000001
>>        SCSI: Looking for devices
>>           8000000000000000 DISK     : "QEMU     QEMU HARDDISK    2.5+"
>> Populating /vdevice/vty at 71000110
>> Populating /pci at 800000020000000
>> No NVRAM common partition, re-initializing...
>> Scanning USB
>> Using default console: /vdevice/vty at 71000110
>>
>>   Welcome to Open Firmware
>>
>>   Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
>>   This program and the accompanying materials are made available
>>   under the terms of the BSD License available at
>>   http://www.opensource.org/licenses/bsd-license.php
>>
>>
>> Trying to load:
>> from: /vdevice/v-scsi at 71000001/disk at 8000000000000000 ... Successfully
>> loaded
>>
>>>> FreeBSD/powerpc Open Firmware boot block  
>>    Boot path:   /vdevice/v-scsi at 71000001/disk at 8000000000000000
>>    Boot loader: /boot/loader
>>    Boot volume:   /vdevice/v-scsi at 71000001/disk at 8000000000000000:2
>> Consoles: Open Firmware console
>>
>> FreeBSD/powerpc64 Open Firmware loader, Revision 0.1
>> Memory: 8388608KB
>> Booted from: /vdevice/v-scsi at 71000001/disk at 8000000000000000
>>
>> Loading /boot/defaults/loader.conf
>> /boot/kernel/kernel data=0x136c550+0x4aa1f0
>> syms=[0x8+0x165c78+0x8+0x1643cb] /boot/entropy size=0x1000
>>
>> Hit [Enter] to boot immediately, or any other key for command prompt.
>> Booting [/boot/kernel/kernel]...
>> Kernel entry at 0x1025d0 ...
>> ---<<BOOT>>---
>> Copyright (c) 1992-2018 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
>> 1994 The Regents of the University of California. All rights reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 12.0-RELEASE r341666 GENERIC powerpc
>> gcc version 4.2.1 20070831 patched [FreeBSD]
>> VT: init without driver.
>> cpu0: IBM POWER9 revision 2.1, 2250.00 MHz
>> cpu0: Features
>> dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE>
>> cpu0: Features2
>> eee00000<ARCH207,HTM,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN>
>> real memory  = 8544436224 (8148 MB)
>> avail memory = 8258940928 (7876 MB)
>> random: unblocking device.
>> random: entropy device external interface
>> kbd0 at kbdmux0
>> random: registering fast source PowerISA DARN random number generator
>> random: fast provider: "PowerISA DARN random number generator"
>> ofwbus0: <Open Firmware Device Tree> on nexus0
>> xicp0: <External Interrupt Presentation Controller> on ofwbus0
>> xicp0: Handling CPUs 0-7
>> vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0
>> uart0: <POWER Hypervisor Virtual Serial Port> irq 16781585 on vdevice0
>> vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781570 on vdevice0
>> vscsi0: Queue depth 22 commands
>> pcib0: <RTAS Host-PCI bridge> on ofwbus0
>> pci0: <POWER Hypervisor PCI bus> on pcib0
>> cpulist0: <Open Firmware CPU Group> on ofwbus0
>> cpu0: <Open Firmware CPU> on cpulist0
>> rtas0: <Run-Time Abstraction Services> on ofwbus0
>> rtas0: registered as a time-of-day clock, resolution 0.002000s
>> Timecounter "timebase" frequency 512000000 Hz quality 0
>> Event timer "decrementer" frequency 512000000 Hz quality 1000
>> Timecounters tick every 1.000 msec
>> usb_needs_explore_all: no devclass
>> run_interrupt_driven_hooks: still waiting after 60 seconds for
>> xpt_config run_interrupt_driven_hooks: still waiting after 120
>> seconds for xpt_config run_interrupt_driven_hooks: still waiting
>> after 180 seconds for xpt_config run_interrupt_driven_hooks: still
>> waiting after 240 seconds for xpt_config run_interrupt_driven_hooks:
>> still waiting after 300 seconds for xpt_config random: unblocking
>> device. panic: run_interrupt_driven_config_hooks: waited too long
>> cpuid = 0
>> time = 361
>> KDB: stack backtrace:
>> 0xe000000000008660: at .kdb_backtrace+0x5c
>> 0xe000000000008790: at .vpanic+0x1b4
>> 0xe000000000008850: at .panic+0x38
>> 0xe0000000000088e0: at .boot_run_interrupt_driven_config_hooks+0x194
>> 0xe0000000000089e0: at .mi_startup+0x1f8
>> 0xe000000000008a80: at btext+0xc4
>> KDB: enter: panic
>> [ thread pid 0 tid 100000 ]
>> Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
>> db>  
>>
>>
>>
> 
> It's possible FreeBSD is making a decision based on certain assumptions
> of the CPU type in the VM, but I'm not sure at all.  I hope someone
> else might know more about the pseries FreeBSD code than I do.

Well, I was kinda hoping that since
boot_run_interrupt_driven_config_hooks is in the common code, it might
look familiar to someone, may be. Or give some clues.

Can FreeBSD start from an arbitrary memory location? I can compile the
kernel, modify QEMU to load it into the guest RAM and start guest from
where the kernel is loaded, and this works for Linux-PPC64 (it detects
endianness and then relocates-readjusts itself) but will this work with
FreeBSD?

-- 
Alexey