FreeBSD under VMware ESXi ARM Fling

[Robert Crowston]

My initial hypothesis was wrong.

I had meant, can you provide a text copy of the dmesg (collected from a serial port!)!

But, from the video, it seems that as soon as we get to the Release APs phase, there is some race condition that is causing text to be concurrently written to the screen by multiple processors--you can see it becomes garbled. KDB is entered shortly before the "Release APs..." is printed to the screen, but text continues to be written to the console. [We don't really know in what order these events happen, though.] A panic() is triggered within a few moments. At this point, we enter termcn_cnputc(), and violate a KASSERT in __mtx_lock_spin_flags. This causes a recursive panic.

For me the next logical step would be to attach a debugger to the guest process. Any idea if ESXi ARM allows debug or single stepping of guests?

    -- RHC.

On Mon, Oct 19, 2020 at 20:19, Vincent Milum Jr <freebsd-arm at darkain.com> wrote:

> I'm not too familiar with the FreeBSD kernel itself. Here is a recording of booting the VM from the 13.0 2020-10-15 Snapshot ISO. There are a few different outcomes that seem to randomly happen when booting.
>
> https://www.youtube.com/watch?v=Ls9gkQ14Z-A
>
> On Mon, Oct 19, 2020 at 11:49 AM Robert Crowston <crowston at protonmail.com> wrote:
>
> > > The largest issue currently is that SMP will cause the FreeBSD kernel to
> > panic during boot.
> >
> > Could you send the panic?
> >
> > I’m guessing lack of spintable support.
> >
> >     — RHC.
> >
> > On Mon, Oct 19, 2020 at 07:19, Vincent Milum Jr <freebsd-arm at darkain.com> wrote:
> >
> > > Recently, VMware released what they're calling "ESXi ARM Fling" - an
> > > ARM64/Aarch64 version of VMware ESXi server.
> > >
> > > Details of the release along with the download can be found here:
> > > https://flings.vmware.com/esxi-arm-edition
> > >
> > > They support a number of ARM platforms already, most notably is the
> > > Raspberry Pi
> > > 4 (only 4GB and 8GB models). They have instructions on their site on how to
> > > install and configure the UEFI firmware on the Pi for getting the ESXi
> > > installer
> > > and then ESXi itself up and running.
> > >
> > > UEFI builds of FreeBSD ARM64/Aarch64 mostly work "out of the box" under ESXi
> > > ARM, whereas ARM 32-bit kernels are not supported by any guest OS under ESXi
> > > ARM.
> > >
> > > There are, however, some areas that could use some improvement, or need some
> > > massive work to get working.
> > >
> > > The largest issue currently is that SMP will cause the FreeBSD kernel to
> > > panic
> > > during boot. A VM with only one single vCPU however will boot and run just
> > > fine.
> > >
> > > Prior to the 13.0-CURRENT snapshot 2020-10-15, the CD-Rom driver was not
> > > included with the ARM64 builds, so the ISO would fail to mount itself on
> > > boot. This
> > > means the easiest way to try earlier versions, such as 12.1-RELEASE, you'll
> > > need
> > > to either convert the pre-built vmdk file from Workstation to ESXi format,
> > > or
> > > convert the raw disk image to ESXi vmdk format. This is how I initially got
> > > FreeBSD up and running prior to the 2020-10-15 snapshot.
> > >
> > > Every version of FreeBSD I've tested on ARM is lacking several drivers by
> > > default.
> > >
> > > uhci for USB 2.0 is not enabled by default. Ensure your VM is initially set
> > > up
> > > for USB 3.1 so the virtual keyboard and mouse will function. This is easy
> > > enough
> > > to fix post install with: uhci_load="YES"
> > >
> > > ums is the same as above. ESXi ARM uses USB instead of PS2 for the virtual
> > > mouse, so add this post install if you want mouse support: ums_load="YES"
> > >
> > > vmxnet and pvscsi are not pre-compiled. However, they compile and operate
> > > perfectly as expected without any modification on ARM.
> > >
> > > vmci has some x86 assembly that I've not dug into yet, so that is currently
> > > not
> > > working.
> > >
> > > I've documented in detail, and continue to update as I test more drivers
> > > over on
> > > my blog:
> > > https://vincerants.com/freebsd-on-esxi-arm-fling-fixing-virtual-hardware/
> > >
> > > The ESXi UI has a known bug in that it'll list some virtual hardware that
> > > doesn't actually exist inside of the VM. These are all legacy devices, and
> > > should be easy enough to figure out. There are no virtual IDE/ATA or PS2
> > > ports.
> > > Keyboard and mouse now use the virtual USB controller instead.
> > >
> > > open-vm-tools has required a bit of work to make functional on FreeBSD under
> > > ARM. There is currently a patch open with VMware with my changes to the
> > > codebase
> > > to make it functional. To make things easy, I've uploaded some pre-compiled
> > > binaries to my blog, as well as included steps to compile this customized
> > > package yourself. The blog will also be updated as development progresses
> > > upstream with VMware.
> > > https://vincerants.com/open-vm-tools-on-freebsd-under-vmware-esxi-arm-fling/
> > >
> > > Xorg desktop runs, and is mostly functional. I've yet to get the
> > > xf86-input-vmmouse or xf86-video-vmware working. Xorg can use sysmouse and
> > > the
> > > console framebuffer for rendering, and this is working okay for basic
> > > testing
> > > right now.
> > >
> > > I also have an early dmesg, but this was from before getting most drivers
> > > working.
> > > https://dmesgd.nycbug.org/index.cgi?do=view&id=5693
> > >
> > > I welcome any additional feedback on getting FreeBSD to 100% under ESXi ARM!
> > >
> > > - Vince (darkain)
> > > _______________________________________________
> > > freebsd-arm at freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-arm
> > > To unsubscribe, send any mail to "freebsd-arm-unsubscribe at freebsd.org"