Linux domU only works with xen_platform_pci=0 ?

[Roger Pau Monné]

On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
> Hello,
> 
> I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
> Following a combination of handbook [1] and wiki [2], I was able to get
> get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
> including installation to disk and networking.
> 
> It seemed to me as if the switch 'xen_platform_pci' doesn't have an
> effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
> matter if I set it to 1 or 0.
> 
> Do I understand it correctly, that this switch makes the difference
> between HVM and PVHVM mode?
> According to xl.cfg(5), it 'enables a guest Operating System [...] to
> make use of paravirtualization features such as disk and network devices'.
> 
> 
> Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
> Linux only detected the harddisk with xen_platform_pci=0.
> 
> For Alpine Linux, with xen_platform_pci=1, I get the following messages
> on the console:
> 
> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632

That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
backend.

> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> xenbus_dev_probe on device/vbd/5632)
> 
> After waiting for a couple minutes it boots, but doesn't detect the disk.
> 
> Looking in /var/log/xen/qemu-dm-alpine-hvm.log, I see the following
> messages:
> 
> xen be core: can't open gnttab device
> xen be core: can't open gnttab device
> xen be: vkbd-0: initalize() failed
> xen be: vkbd-0: initalize() failed
> xen be: vkbd-0: initalize() failed
> 
> 
> 
> Googling around yielded [3], where someone apparently ran into the same
> problem, and used xen_platform_pci=0 as workaround.
> 
> 
> 
> So my question is:
> Is xen_platform_pci=1 required for PVHVM mode?

No, it shouldn't be.

> If yes, is PVHVM mode only available for FreeBSD domU's?
> If no, how can I enable it for Linux guests?

If you boot FreeBSD Dom0 kernel with boot_verbose=YES (in
/boot/loader.conf), can you paste the messages you get when trying to
boot the PVHVM guest with xen_platform_pci=1?

Also, can you paste the output of `xenstore-ls -fp` executed on Dom0
after the PVHVM guest has failed to boot but is still running?

> Any response would be appreciated, thanks!
> 
> (Unrelated: I also couldn't get PVH guests to boot, neither FreeBSD nor
> Linux. The console stayed blank and CPU went to 100%. But that's a whole
> other story I guess.)

FreeBSD Dom0 runs in PVH mode, so you are indeed able to boot at least
one PVH guest :).

Let's look into the PVHVM issue first.

> My system:
> 
> OS: FreeBSD-11.1-RELEASE amd64
> xen installed via pkg
> Processor: E3-1240 v2, (which supports VT-x, VT-d and EPT)
> 
> 
> My alpine.cfg:
> 
> builder="hvm"
> name="alpine-hvm"
> memory=1024
> vcpus=1
> vif = [ 'bridge=bridge0' ]
> disk = [ '/dev/zvol/zbulk/vm/alpine-hvm_disk0,raw,hda,rw',
>  '/zbulk/mediashare/isos/linux/apline-virt-3.7.0-x86_64.iso,raw,hdb:cdrom,r']
> 
> xen_platform_pci=1 # or 0, see above
> vnc = 1
> vnclisten = "0.0.0.0"
> serial = "pty"
> usbdevice = "tablet"

Your domain config file seems OK to me.

Roger.