unknown option "XENHVM" for Arm 32 ; missing code detected.

From: Mario Marietto <marietto2008_at_gmail.com>
Date: Wed, 10 Jan 2024 23:25:21 UTC
Hello.

I'm trying to boot FreeBSD for arm32 bit as DomU on my ARM Chromebook.
Basically,with the help of the xen developers I have patched the FreeBSD
kernel and I've been able to boot it as if it was a zImage format file.

I've enabled all the xen options inside the Linux kernel file used for the
domU and this is what happened when I tried to boot the FreeBSD kernel file
:


mario@devuan-bunsen:/mnt/zroot2/zroot2/OS/Chromebook/freebsd-xen/domU-freebsd#
./start-freebsd

Parsing config from freebsd.cfg

libxl: debug: libxl_create.c:2081:do_domain_create: ao 0x442780:
create: how=(nil) callback=(nil) poller=0x43cc50

libxl: detail: libxl_create.c:662:libxl__domain_make: passthrough: disabled
libxl: debug: libxl_arm.c:148:libxl__arch_domain_prepare_config:
Configure the domain
libxl: debug: libxl_arm.c:151:libxl__arch_domain_prepare_config: -
Allocate 0 SPIs

libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=unknown specification=xen

libxl: debug: libxl_device.c:452:libxl__device_disk_set_backend: Disk
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:1342:initiate_domain_create: Domain
1:running bootloader
libxl: debug: libxl_bootloader.c:417:libxl__bootloader_run: Domain
1:no bootloader configured, using user supplied kernel

libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch
w=0x43d8f0: deregister unregistered domainbuilder: detail:
xc_dom_allocate: cmdline="console=hvc0", features=""

domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x40008+0x966 at 0xb5972000
domainbuilder: detail: xc_dom_alloc_segment:   kernel       :
0x40008000 -> 0x4096e000  (pfn 0x40008 + 0x966 pages)
domainbuilder: detail: xc_dom_load_zimage_kernel: called
domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg
0x40008000-0x4096e000
domainbuilder: detail: xc_dom_load_zimage_kernel: copy 9851212 bytes
from blob 0xb62d8000 to dst 0xb5972000
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x48000+0x1 at 0xb6f0a000
domainbuilder: detail: xc_dom_alloc_segment:   devicetree   :
0x48000000 -> 0x48001000  (pfn 0x48000 + 0x1 pages)
domainbuilder: detail: alloc_magic_pages: called
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x48001000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: bootearly: doing nothing
domainbuilder: detail: start_info_arm: called
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 58 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 9620 kB
domainbuilder: detail:       domU mmap          : 9628 kB
domainbuilder: detail: vcpu_arm32: called
domainbuilder: detail: Initial state CPSR 0x400001d3 PC 0x40008000
domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[0] -> d0 0x39000
domainbuilder: detail: xc_dom_set_gnttab_entry: d1 gnt[1] -> d0 0x39001
domainbuilder: detail: xc_dom_release: called

libxl: debug: libxl_device.c:415:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=phy specification=xen

libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch
w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0:
register slotnum=3

libxl: debug: libxl_create.c:2120:do_domain_create: ao 0x442780:
inprogress: poller=0x43cc50, flags=i

libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x4410e4
wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: event
epath=/local/domain/0/backend/vbd/1/51712/state

libxl: debug: libxl_event.c:1054:devstate_callback: backend
/local/domain/0/backend/vbd/1/51712/state wanted state 2 still waiting
state 1

libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x4410e4
wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0: event
epath=/local/domain/0/backend/vbd/1/51712/state

libxl: debug: libxl_event.c:1051:devstate_callback: backend
/local/domain/0/backend/vbd/1/51712/state wanted state 2 ok

libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch
w=0x4410e4 wpath=/local/domain/0/backend/vbd/1/51712/state token=3/0:
deregister slotnum=3

libxl: debug: libxl_device.c:1150:device_backend_callback: Domain
1:calling device_backend_cleanup
libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch
w=0x4410e4: deregister unregistered
libxl: debug: libxl_linux.c:194:libxl__hotplug_disk: Domain 1:Args and
environment ready
libxl: debug: libxl_device.c:1251:device_hotplug: Domain 1:calling
hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_device.c:1252:device_hotplug: Domain 1:extra args:
libxl: debug: libxl_device.c:1260:device_hotplug: Domain 1:env:
libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: script:
/etc/xen/scripts/block
libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1: XENBUS_TYPE: vbd
libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1:
XENBUS_PATH: backend/vbd/1/51712
libxl: debug: libxl_device.c:1267:device_hotplug: Domain 1:
XENBUS_BASE_PATH: backend
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to
execute: /etc/xen/scripts/block add
libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch
w=0x441168: deregister unregistered
libxl: debug: libxl_linux.c:211:libxl__get_hotplug_script_info: Domain
1:num_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1236:device_hotplug: Domain 1:No hotplug
script to execute
libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch
w=0x441168: deregister unregistered

libxl: debug: libxl_event.c:812:libxl__ev_xswatch_register: watch
w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: register
slotnum=3

libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x43e7cc
wpath=/local/domain/1/console/tty token=3/1: event
epath=/local/domain/1/console/tty

libxl: debug: libxl_event.c:750:watchfd_callback: watch w=0x43e7cc
wpath=/local/domain/1/console/tty token=3/1: event
epath=/local/domain/1/console/tty

libxl: debug: libxl_event.c:2403:libxl__ao_progress_report: ao
0x442780: progress report: ignored

libxl: debug: libxl_event.c:849:libxl__ev_xswatch_deregister: watch
w=0x43e7cc wpath=/local/domain/1/console/tty token=3/1: deregister
slotnum=3

libxl: debug: libxl_event.c:863:libxl__ev_xswatch_deregister: watch
w=0x43e7cc: deregister unregistered
libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete, rc=0
libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy

libxl: debug: libxl_domain.c:704:libxl_domain_unpause: Domain 1:ao
0x442780: create: how=(nil) callback=(nil) poller=0x43cc50

libxl: debug: libxl_event.c:2067:libxl__ao_complete: ao 0x442780: complete, rc=0

libxl: debug: libxl_domain.c:712:libxl_domain_unpause: Domain 1:ao
0x442780: inprogress: poller=0x43cc50, flags=ic

libxl: debug: libxl_event.c:2036:libxl__ao__destroy: ao 0x442780: destroy

xencall:buffer: debug: total allocations:98 total releases:98
xencall:buffer: debug: current allocations:0 maximum allocations:3
xencall:buffer: debug: cache current size:3
xencall:buffer: debug: cache hits:86 misses:3 toobig:9
xencall:buffer: debug: total allocations:0 total releases:0
xencall:buffer: debug: current allocations:0 maximum allocations:0
xencall:buffer: debug: cache current size:0
xencall:buffer: debug: cache hits:0 misses:0 toobig:0

FROZEN.


I see these two processes enabled...but I don't know how to use FreeBSD...

# ps ax
2606 ?        Ssl    0:00 xl -vvvv create freebsd.cfg
2607 pts/0    Sl+    0:00 /usr/lib/xen-4.17/bin/xenconsole 1 --num 0 --type pv


now the fun begins...because,based on what say the xen developers :

... the ps output, it seems that ``xl create`` completed and you have the
console open. So the freeze you mention is just because your FreeBSD guest
is not outputting anything. As mentioned earlier, I don't think a lot of
testing has been done for 32-bit Arm FreeBSD. So it is quite possible that
there are some pieces of code missing. The first step is to check if the
FreeBSD kernel was built with Xen options. If they are, then you will need
to find out where FreeBSD is stuck (or why the console is not enabled).
Unfortunately, we don't have any support to use GDB on the guest kernel. So
you will have to modify FreeBSD a bit to check where it freezes. Assuming
you have a debug build of the hypervisor, then you can sprinkle the FreeBSD
boot code with the assembly instruction 'hvc 0xfffc'. When this is reached,
this will issue an hypercall that will print on the Xen console that the
given instruction has reached (the PC will be printed). There are other
useful 'hvc' calls implemented by Xen for low level debugging. You can look
at do_debug_trap() in Xen code.


I've added this to GENERIC kernel of FreeBSD guest :

# Xen HVM Guest Optimizations
# NOTE: XENHVM depends on xenpci.  They must be added or removed together.

options     XENHVM            # Xen HVM kernel infrastructure
device      xenpci            # Xen HVM Hypervisor services driver

Unfortunately,it is not accepted :

unknown option "XENHVM"


I've found these parameters ONLY on the file /usr/src/sys/conf/files.amd64 :

x86/xen/pv.c            optional    xenhvm
x86/xen/pvcpu_enum.c    optional    xenhvm
x86/xen/xen_pci_bus.c   optional    xenhvm


Look at the prefix x86. It is not for arm. It seems there is a piece of
code that hasn't been written. Is there a chance to add it ? Does anyone
want to help ?

No arch/arm64 does not contain Xen in KERNCONF either.

Surprisingly files.x86 invokes a different set of files. I don't know what
build it is.

x86/xen/hvm.c            optional    xenhvm
x86/xen/xen_intr.c       optional    xenhvm
x86/xen/xen_apic.c       optional    xenhvm
x86/xen/xenpv.c          optional    xenhvm
x86/xen/xen_msi.c        optional    xenhvm
x86/xen/xen_nexus.c      optional    xenhvm


-- 
Mario.