11-RELEASE acting as vbd backend

[Roger Pau Monné]

On Sun, Dec 18, 2016 at 12:36:32PM -0700, Nathan Friess wrote:
> On 12/15/2016 10:34 AM, Roger Pau Monné wrote:
> > OK, I finally have this ready, you will need the following patches for FreeBSD,
> > which I pushed to my git repo on top of current HEAD:
> > 
> > git://xenbits.xen.org/people/royger/freebsd.git add_watches_xenstore
> > 
> > You can either compile a kernel from this branch or pickup the 4 top patches
> > and apply them to your local tree (I don't think you are going to find any
> > conflicts even if the tree is stable/11 or 11.0-RELEASE).
> > 
> > You should also update your xen-tools package to the latest version, I've just
> > updated it to contain the xenstore device path fix:
> > 
> > https://svnweb.freebsd.org/ports?view=revision&revision=428628
> > 
> > After this is done, it should just be a matter of launching xendriverdomain at
> > system boot and it should work out of the box. Let me know how this goes.
> 
> Hi Roger,
> 
> Thank you!  With your patches I've made some progress but not quite there
> yet.  I've been trying to trace through xl devd but I haven't been able to
> pinpoint the underlying issue.  Here is where I'm at.
> 
> I applied the 4 patches from your add_watches_xenstore branch to the
> 11.0-RELEASE kernel and built xen-tools from svn.
> 
> The xendriverdomain rc script now starts xl devd successfully.  However,
> when the other domU starts and the backend should be set up in FreeBSD, xl
> devd attempts to run /etc/xen/scripts/block, which is in /usr/local instead.
> My guess is that dom0 is generating that path and because dom0 is Linux the
> paths don't match.  (I'm not specifying the script in the domU config
> directly.)  Adding a symlink in /etc to /usr/local/etc/xen gets around this
> for now.

That's expected, this path is set by the Dom0, and it uses the Linux default
value. It might make more sense to simply don't set it when the backend is a
driver domain, but in any case you can easily solve it by using the "script"
disk option[0]. You should add script=/usr/local/etc/xen/scripts/block to your
disk configuration line.

> Secondly, when /etc/xen/scripts/block runs, it doesn't have /usr/local/bin
> or sbin in its PATH, so it cannot execute xenstore-read or write.  That was
> also easy enough to fix.

Hm, I guess I haven't seen this because I was running xl devd manually from the
command line. AFAICT the patch I'm attaching at the bottom of this email should
solve it.

> So, now the block script runs but after that nothing seems to happen. This
> is where I started to trace the code in libxl.  I think what needs to happen
> is that xl devd (or the block script?) needs to set
> local/domain/(freebsd)/backend/vbd/(linuxpv)/51713/sectors, info, and
> sector-size.  The .../state variable is also still set to 3 after the block
> script runs.

Backend in state 3 means it's ready and waiting for the frontend to connect.
That's quite weird because I've just tested a FreeBSD driver domain from a
Linux Dom0 and it seems to work fine, other guests can properly see and attach
the devices served by the FreeBSD driver domain.

Do you have driver_domain=1 set in your FreeBSD xl config file?

> It looks like the Linux hotplug scripts don't set those variables directly
> either, so I'm not sure where that should be happening.  (Aside from that, I
> did see that Linux sets .../physical-device to a major:minor and some
> feature- variables that FreeBSD does not set.)  The xldevd.log doesn't show
> anything interesting and running with -vvv only shows some cleanup being
> done after the script has finished.

This should be set by blkback itself IIRC.

Roger.

[0] https://xenbits.xen.org/docs/unstable/misc/xl-disk-configuration.txt

---8<---
>From 0e2e6b6f3892d83548dd1d5c3644972ee5fe9847 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <roger.pau at citrix.com>
Date: Mon, 19 Dec 2016 12:16:33 +0000
Subject: [PATCH] init/FreeBSD: set correct PATH for xl devd
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

FreeBSD init scripts don't have /usr/local/{bin/sbin} in it's PATH, which
prevents `xl devd` from working properly since hotplug scripts require the set
of xenstore cli tools to be in PATH.

While there also fix the usage of --pidfile, which according to the xl help
doesn't use "=", and add braces around XLDEVD_PIDFILE.

Signed-off-by: Roger Pau Monné <roger.pau at citrix.com>
---
Cc: Ian Jackson <ian.jackson at eu.citrix.com>
Cc: Wei Liu <wei.liu2 at citrix.com>
---
 tools/hotplug/FreeBSD/rc.d/xendriverdomain.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/hotplug/FreeBSD/rc.d/xendriverdomain.in b/tools/hotplug/FreeBSD/rc.d/xendriverdomain.in
index 8ece7c3..3917de2 100644
--- a/tools/hotplug/FreeBSD/rc.d/xendriverdomain.in
+++ b/tools/hotplug/FreeBSD/rc.d/xendriverdomain.in
@@ -29,7 +29,7 @@ xendriverdomain_startcmd()
 {
 	printf "Starting xenservices: xl devd."
 
-	${sbindir}/xl devd --pidfile=$XLDEVD_PIDFILE ${XLDEVD_ARGS}
+	PATH="${bindir}:${sbindir}:$PATH" ${sbindir}/xl devd --pidfile ${XLDEVD_PIDFILE} ${XLDEVD_ARGS}
 
 	printf "\n"
 }
-- 
2.10.1 (Apple Git-78)