Linux domU only works with xen_platform_pci=0 ?

[Roger Pau Monné]

On Sun, May 13, 2018 at 07:33:03PM +0200, Kai Otto wrote:
> On 13/05/18 17:16, Roger Pau Monné wrote:
> > On Sun, May 13, 2018 at 03:51:36PM +0200, Kai Otto wrote:
> >> Hello,
> >>
> >> I'm trying to set up a FreeBSD 11.1 system as Xen virtualization host.
> >> Following a combination of handbook [1] and wiki [2], I was able to get
> >> get a FreeBSD dom0 in PVH mode, and FreeBSD domU in HVM mode running,
> >> including installation to disk and networking.
> >>
> >> It seemed to me as if the switch 'xen_platform_pci' doesn't have an
> >> effect on FreeBSD domU's, as I see e.g. a xenpci0 and xbd0 in dmesg no
> >> matter if I set it to 1 or 0.
> >>
> >> Do I understand it correctly, that this switch makes the difference
> >> between HVM and PVHVM mode?
> >> According to xl.cfg(5), it 'enables a guest Operating System [...] to
> >> make use of paravirtualization features such as disk and network devices'.
> >>
> >>
> >> Afterwards, I tried to create a Linux domU. Both Centos 7 and Alpine
> >> Linux only detected the harddisk with xen_platform_pci=0.
> >>
> >> For Alpine Linux, with xen_platform_pci=1, I get the following messages
> >> on the console:
> >>
> >> vbd vbd-5632: 19 xenbus_dev_probe on device/vbd/5632
> > That's ENODEV IIRC, I think there's something wrong with FreeBSD disk
> > backend.
> >
> >> vbd vbd-5632: failed to write error node for device/vbd/5632 (19
> >> xenbus_dev_probe on device/vbd/5632)
> >>
> >> After waiting for a couple minutes it boots, but doesn't detect the disk.
> >>
> >> Looking in /var/log/xen/qemu-dm-alpine-hvm.log, I see the following
> >> messages:
> >>
> >> xen be core: can't open gnttab device
> >> xen be core: can't open gnttab device
> >> xen be: vkbd-0: initalize() failed
> >> xen be: vkbd-0: initalize() failed
> >> xen be: vkbd-0: initalize() failed
> >>
> >>
> >>
> >> Googling around yielded [3], where someone apparently ran into the same
> >> problem, and used xen_platform_pci=0 as workaround.
> >>
> >>
> >>
> >> So my question is:
> >> Is xen_platform_pci=1 required for PVHVM mode?
> > No, it shouldn't be.
> >
> >> If yes, is PVHVM mode only available for FreeBSD domU's?
> >> If no, how can I enable it for Linux guests?
> > If you boot FreeBSD Dom0 kernel with boot_verbose=YES (in
> > /boot/loader.conf), can you paste the messages you get when trying to
> > boot the PVHVM guest with xen_platform_pci=1?
> >
> > Also, can you paste the output of `xenstore-ls -fp` executed on Dom0
> > after the PVHVM guest has failed to boot but is still running?
> >
> Hello again,
> thanks for your response Roger!
> 
> I attached both. Because I don't know how the mailinglist handles
> attachments, here is a pastebin of xenstore-ls:
> https://pastebin.com/s7wa1ee7
> And a direct paste of dmesg messages:
> 
> random: harvesting attach, 8 bytes (4 bits) from xbbd0
> tap0: bpf attached
> tap0: Ethernet address: 00:bd:ee:99:ff:00
> tap0: link state changed to UP
> tap0: changing name to 'xnb2.0-emu'
> xnb(xnb_probe:1127): Claiming device 0, xnb
> xnb2.0: bpf attached
> xnb(xnb_attach:1271): Attaching to backend/vif/2/0
> random: harvesting attach, 8 bytes (4 bits) from xnb0
> xnb(xnb_frontend_changed:1395): frontend_state=Initialising,
> xnb_state=InitWait
> xnb2.0: link state changed to DOWN
> xnb2.0: link state changed to UP
> xnb2.0: link state changed to DOWN
> xnb2.0: promiscuous mode enabled
> xnb2.0: link state changed to UP
> xnb2.0-emu: promiscuous mode enabled
> nd6_dad_timer: cancel DAD on xnb2.0 because of ND6_IFF_IFDISABLED.
> xnb(xnb_frontend_changed:1395): frontend_state=Connected, xnb_state=InitWait
> xnb(xnb_connect_comms:788): rings connected!

Can you please give a try to the following patch? I think it should
apply cleanly to stable/11 (11.1-RELEASE). You will have to rebuild
and install the kernel.


Thanks, Roger.
---8<---
diff --git a/sys/dev/xen/blkback/blkback.c b/sys/dev/xen/blkback/blkback.c
index 9744c8da73cb..e9c7afe7d18c 100644
--- a/sys/dev/xen/blkback/blkback.c
+++ b/sys/dev/xen/blkback/blkback.c
@@ -806,6 +806,9 @@ struct xbb_softc {
 
 	/** Watch to wait for hotplug script execution */
 	struct xs_watch		  hotplug_watch;
+
+	/** Got the needed data from hotplug scripts? */
+	bool			  hotplug_done;
 };
 
 /*---------------------------- Request Processing ----------------------------*/
@@ -3310,10 +3313,9 @@ xbb_connect(struct xbb_softc *xbb)
 {
 	int error;
 
-	if (xenbus_get_state(xbb->dev) != XenbusStateInitialised)
-		return;
-
-	if (xbb_collect_frontend_info(xbb) != 0)
+	if (!xbb->hotplug_done ||
+	    (xenbus_get_state(xbb->dev) != XenbusStateInitWait) ||
+	    (xbb_collect_frontend_info(xbb) != 0))
 		return;
 
 	xbb->flags &= ~XBBF_SHUTDOWN;
@@ -3412,6 +3414,7 @@ xbb_shutdown(struct xbb_softc *xbb)
 		free(xbb->hotplug_watch.node, M_XENBLOCKBACK);
 		xbb->hotplug_watch.node = NULL;
 	}
+	xbb->hotplug_done = false;
 
 	if (xenbus_get_state(xbb->dev) < XenbusStateClosing)
 		xenbus_set_state(xbb->dev, XenbusStateClosing);
@@ -3692,8 +3695,7 @@ xbb_attach_disk(struct xs_watch *watch, const char **vec, unsigned int len)
 		return;
 	}
 
-	/* Tell the front end that we are ready to connect. */
-	xenbus_set_state(dev, XenbusStateInitialised);
+	xbb->hotplug_done = true;
 }
 
 /**
@@ -3757,6 +3759,7 @@ xbb_attach(device_t dev)
 	 * We need to wait for hotplug script execution before
 	 * moving forward.
 	 */
+	KASSERT(!xbb->hotplug_done, ("Hotplug scripts already executed"));
 	watch_path = xs_join(xenbus_get_node(xbb->dev), "physical-device-path");
 	xbb->hotplug_watch.callback_data = (uintptr_t)dev;
 	xbb->hotplug_watch.callback = xbb_attach_disk;