[RFC] Add support for Xen ARM guest on FreeBSD

[Warner Losh]

On Jan 14, 2014, at 2:01 PM, Julien Grall wrote:
> This new support brings 2 open questions (for both Xen and FreeBSD community).
> When a new guest is created, the toolstack will generated a device tree which
> will contains:
> 	- The amount of memory
> 	- The description of the platform (gic, timer, hypervisor node)
> 	- PSCI node for SMP bringup
> 
> Until now, Xen on ARM supported only Linux-based OS. When the support of
> device tree was added in Xen for guest, we chose to use Linux device tree
> bindings (gic, timer,...). It seems that FreeBSD choose a different way to
> implement device tree:
> 	- strictly respect ePAR (for interrupt-parent property)
> 	- gic bindings different (only 1 interrupt cell)
> 
> I would like to come with a common device tree specification (bindings, ...)
> across every operating system. I know the Linux community is working on having
> a device tree bindings out of the kernel tree. Does FreeBSD community plan to
> work with Linux community for this purpose?

We generally try to follow the common definitions for the FDT stuff. There are a few cases where we either lack the feature set of Linux, or where the Linux folks are moving quickly and changing the underlying definitions where we wait for the standards to mature before we implement. In some cases, where maturity hasn't happened, or where the bindings are overly Linux centric (which in theory they aren't supposed to be, but sometimes wind up that way). where we've not implemented things.

> As specified early, the device tree can vary accross Xen version and user input
> (eg the memory). I have noticed few places (mainly the timer) where the IRQs
> number are harcoded.

These cases should be viewed as bugs in FreeBSD, I believe. One of the goals that has wide support, at leas tin theory, is that we can boot with an unaltered Linux FDT. This goal is some time in the future.

> In the long-term, it would be nice to see FreeBSD booting out-of-box (eg the
> device tree is directly provided by the board firmware). I plan to add support
> for Device Tree loading via Linux Boot ABI, it the way that Xen use to boot a
> new guest.

That would be most welcome.

> The second question is related to memory attribute for page table. The early
> page table setup by FreeBSD are using Write-Through memory attribute which
> result to a likely random (not every time the same place) crash before the
> real page table are initialized.
> Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even today,
> I have no idea if it's a requirement from Xen or a bug (either in Xen or
> FreeBSD).

There were some problems with pages being setup improperly for the mutexes to operate properly. Have you confirmed this on a recent version of FreeBSD?

> The code is taking its first faltering steps, therefore the TODO is quite big:
> 	- Try the code on x86 architecture. I did lots of rework for the event
> 	channels code and didn't even try to compile
> 	- Clean up event channel code
> 	- Clean up xen control code
> 	- Add support for Device Tree loading via Linux Boot ABI
> 	- Fixing crashing on userspace. Could be related to the patch
> 	series "arm SMP on Cortex-A15"
> 	- Add guest SMP support
> 	- DOM0 support?
> 
> Any help, comments, questions are welcomed.

I think this is great! We'd love to work with you to make this happen.

Warner

> Sincerely yours,
> 
> ============= Instruction to test FreeBSD on Xen on ARM ===========
[trimmed]