Re: vmm (bhyve) on GICv2

From: Mark Millard <marklmi_at_yahoo.com>
Date: Wed, 22 May 2024 16:50:23 UTC
On May 22, 2024, at 08:43, Ronald Klop <ronald-lists@klop.ws> wrote:


>> Van: Mark Millard <marklmi@yahoo.com>
>> Datum: woensdag, 22 mei 2024 17:34
>> Aan: Lexi Winter <lexi@le-fay.org>
>> CC: Mario Marietto <marietto2008@gmail.com>, arm@freebsd.org
>> Onderwerp: Re: vmm (bhyve) on GICv2
>> On May 22, 2024, at 00:25, Lexi Winter <lexi@le-fay.org> wrote:
>> 
>> > Mario Marietto:
>> >> On Wed, May 22, 2024 at 9:03AM Lexi Winter <lexi@le-fay.org> wrote:
>> >>> i was pleased to see ARM64 vmm has been merged to main recently.
>> >>>
>> >>> i have several use-cases to run bhyve on RPi4, but unfortunately this
>> >>> hardware only supports GICv2 while vmm requires GICv3.
>> >>>
>> >>> does anyone have an idea if GICv2 is likely to be supported in the
>> >>> future?
>> >>
>> >> Does RPi5 support GICv3 ? I would like to know which kind of relatively
>> >> cheap arm hardware can run bhyve.
>> >
>> > i don't know off hand, but i do know FreeBSD doesn't support RPi5 at
>> > all right now, so this is not an immediate solution :-)
>> 
>> I boot and run a RPi5 via EDK2 in the microsd card slot and a USB3
>> UFS system media (that has lots of different swap partition sizes
>> for use on various machines with widely varying amounts of RAM):
>> 
>> # gpart show -pl
>> =>       40  249737136    mmcsd0  GPT  (119G)
>>          40       2008            - free -  (1.0M)
>>        2048  249733120  mmcsd0p1  RPi5-edk2  (119G)
>>   249735168       2008            - free -  (1.0M)
>> 
>> =>        34  1875384941     da0  GPT  (894G)
>>           34       32734          - free -  (16M)
>>        32768      501760   da0p1  PkgBaseEFI  (245M)
>>       534528    20971520   da0p2  PkgBaseSwp10  (10G)
>>     21506048    29360128   da0p3  PkgBaseSwp14  (14G)
>>     50866176    33554432   da0p4  PkgBaseSwp16  (16G)
>>     84420608    67108864   da0p5  PkgBaseSwp32  (32G)
>>    151529472    96468992   da0p6  PkgBaseSwp46  (46G)
>>    247998464   268435456   da0p7  PkgBaseSwp128  (128G)
>>    516433920     7340032   da0p8  PkgBaseSwp3p5  (3.5G)
>>    523773952    13631488  da0p10  PkgBaseSwp6p5  (6.5G)
>>    537405440  1337979528   da0p9  PkgBaseUFS  (638G)
>>   1875384968           7          - free -  (3.5K)
>> 
>> See: https://github.com/worproject/rpi5-uefi
>> 
>> # uname -apKU
>> FreeBSD aarch64-main-pkgs 15.0-CURRENT FreeBSD 15.0-CURRENT #5 main-n269589-9dcf39575efb-dirty: Sun Apr 21 01:42:00 PDT 2024     root@aarch64-main-pbase:/usr/obj/BUILDs/main-CA76-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA76 arm64 aarch64 1500018 1500018
>> 
>> That kernel is at: /boot/kernel.CA76-NODBG/kernel in my context.
>> 
>> But I also have kernels that I got from:
>> 
>> https://pkg.freebsd.org/FreeBSD:15:aarch64/base_latest/
>> 
>> that boot it as well:
>> 
>> # strings /boot/kernel.GENERIC-NODEBUG/kernel | grep 15.0-
>> @(#)FreeBSD 15.0-CURRENT main-n269581-dfa39133b333 GENERIC-NODEBUG
>> FreeBSD 15.0-CURRENT main-n269581-dfa39133b333 GENERIC-NODEBUG
>> 15.0-CURRENT
>> 
>> And the debug version (witness and such):
>> 
>> # strings /boot/kernel/kernel | grep 15.0-
>> @(#)FreeBSD 15.0-CURRENT main-n269581-dfa39133b333 GENERIC
>> FreeBSD 15.0-CURRENT main-n269581-dfa39133b333 GENERIC
>> 15.0-CURRENT
>> 
>> (It has beem a while since I updated.)
>> 
>> All these kernels boot a world that I got from the same:
>> 
>> https://pkg.freebsd.org/FreeBSD:15:aarch64/base_latest/
>> 
>> I do have another world in a directory tree that I built that I
>> can chroot to.
>> 
>> ===
>> Mark Millard
>> marklmi at yahoo.com
>> 
>   
> 
> Hi,
> 
> This looks nice. Are there any FreeBSD patches needed to boot a RPI5? Or only an up-to-date EDK2?

The kernels and world installed via use of:

https://pkg.freebsd.org/FreeBSD:15:aarch64/base_latest/

that I referenced were/are used unmodified. Those materials are
as built by the FreeBSD project, not built by me.

I build my other kernels and the world that I can chroot to
in order to experiment with builds that explicitly target
Cortext-A76 instruction set compatible aarch64 contexts. Such
is not required. The same Cortex-A76 kernels and world can be
used on the Windows DevKit 2023's combination of Cortex-X1C's
and Cortex-A78C's.

I have only tried EDK2 as distributed via:

https://github.com/worproject/rpi5-uefi/

I have not built it myself.

> And if it works well can we collaborate in documenting the RPI5 on https://wiki.freebsd.org/arm/Raspberry%20Pi?

I'll note that my normal use is headless: serial console and
ssh-over-ethernet. The EtherNet use is via a USB3 dongle: the
built-in EtherNet is not supported via EDK2.

Also, I leave the microsd card that has EDK2 on it in the
microsd card slot and do not normally have it mounted. But
the /etc/fstab that I use has a line:

/dev/gpt/RPi5-edk2              /RPi5-edk2      msdosfs rw,noatime,noauto       0       0

to mount it when desired. Presuming the /RPi5-edk2/ was
created previously:

# mount /RPi5-edk2

# ls -loaT /RPi5-edk2
total 2180
drwxr-xr-x   1 root wheel -       32768 Dec 31 23:00:00 1979 .
drwxr-xr-x  26 root wheel -         512 May 22 09:29:16 2024 ..
-rwxr-xr-x   1 root wheel uarch 2031616 Mar 16 03:43:48 2024 RPI_EFI.fd
-rwxr-xr-x   1 root wheel -       76038 Mar 15 20:46:12 2024 bcm2712-rpi-5-b.dtb
-rwxr-xr-x   1 root wheel uarch     474 Apr 19 18:38:24 2024 config.txt
-rwxr-xr-x   1 root wheel uarch     474 Apr 19 18:38:24 2024 config.txt.m_m_edk2_rpi5

# cat /RPi5-edk2/config.txt
armstub=RPI_EFI.fd
device_tree_address=0x1f0000
device_tree_end=0x210000

# Force 32 bpp framebuffer allocation.
framebuffer_depth=32

# Disable compensation for displays with overscan.
disable_overscan=1

# Force maximum USB power regardless of the power supply.
usb_max_current_enable=1

# Force maximum CPU speed.
force_turbo=1

#
# Local additions:
enable_uart=1
uart_2ndstage=1
dtdebug=1
disable_commandline_tags=1
#
[pi5]
over_voltage_delta=100000
arm_freq=2600
[all]

The local additions are not required.

I'll note that:

https://github.com/worproject/rpi5-uefi/blob/master/README.md

documents ACPI mode as supporting (extracted from a table):

OS:                        FreeBSD
Version:                   13.2
Tested/supported hardware: Display, UART, USB, SD, PCIe
Notes:                     * SD is limited to HS.

DeviceTree mode is documented with just:

QUOTE
The included DTB is meant for the RPi downstream 6.1.y kernel.
END QUOTE

And, so, FreeBSD is likely not a useful match to DeviceTree
mode at this time.

===
Mark Millard
marklmi at yahoo.com