11-RELEASE acting as vbd backend

[Nathan Friess]

On 12/19/2016 5:27 AM, Roger Pau Monné wrote:
>> Secondly, when /etc/xen/scripts/block runs, it doesn't have /usr/local/bin
>> or sbin in its PATH, so it cannot execute xenstore-read or write.  That was
>> also easy enough to fix.
>
> Hm, I guess I haven't seen this because I was running xl devd manually from the
> command line. AFAICT the patch I'm attaching at the bottom of this email should
> solve it.
>

I should have mentioned earlier that also in the rc script there is a 
typo in the stop function.  rc_pid is set to the pid of xl devd but the 
next lines refer to rc_pids.  Not that I ever want to stop xl devd 
anyway. :)


>> So, now the block script runs but after that nothing seems to happen. This
>> is where I started to trace the code in libxl.  I think what needs to happen
>> is that xl devd (or the block script?) needs to set
>> local/domain/(freebsd)/backend/vbd/(linuxpv)/51713/sectors, info, and
>> sector-size.  The .../state variable is also still set to 3 after the block
>> script runs.
>
> Backend in state 3 means it's ready and waiting for the frontend to connect.
> That's quite weird because I've just tested a FreeBSD driver domain from a
> Linux Dom0 and it seems to work fine, other guests can properly see and attach
> the devices served by the FreeBSD driver domain.
>
> Do you have driver_domain=1 set in your FreeBSD xl config file?

I did not have that before but I added it now.


>> It looks like the Linux hotplug scripts don't set those variables directly
>> either, so I'm not sure where that should be happening.  (Aside from that, I
>> did see that Linux sets .../physical-device to a major:minor and some
>> feature- variables that FreeBSD does not set.)  The xldevd.log doesn't show
>> anything interesting and running with -vvv only shows some cleanup being
>> done after the script has finished.
>
> This should be set by blkback itself IIRC.


Yes, now I see the missing connection.  What is happening is that 
blkback is setting the state to XenbusStateInitWait (2) in xbb_attach(). 
  Once the block script runs the state is set to  XenbusStateInitialised 
in xbb_attach_disk().  Now it waits for the frontend to update some 
variables.

In the Linux kernel (4.4 that I'm using, and 4.9 from what I see), 
drivers/block/xen-blkfront.c, the blkback_changed() function waits for 
state XenbusStateInitWait in the backend after which it will set the 
variables that the backend is waiting for.  It ignores state 
XenbusStateInitialised entirely.


I think what is happening in my case is that FreeBSD is too fast and is 
setting the state twice before Linux notices, so the frontend isn't 
continuing on as it should.  If I manually set the state of the backend 
backwards to 2 then the frontend connects up and the block device seems 
to work.  Now both the frontend and backend transition to the connected 
state as they should.

I don't know what the protocol should be, but it seems that the two ends 
are not agreeing on those states.  Maybe on your systems the timing is 
different so Linux sees the InitWait state before FreeBSD moves to 
Initalised and everything works?

(Aside from that, I wonder why Linux's watch on the state isn't firing 
once for each update?)


Nathan