Xen on FreeBSD 11: Cannot boot from QCOW2 properly (waiting for XENBUS, xenbusb_nop_confighook_cb)

Tue Jan 24 11:45:12 UTC 2017

On Mon, Jan 23, 2017 at 06:25:23PM +0300, Alexander Nusov wrote:
> Hello,
> Sorry for cross-posting, since it's related to Xen hypervisor I'm forwarding this message to the freebsd-xen mailing list.
> 
> I'm trying to launch a HVM DomU guest from QCOW2 image by using PV driver on FreeBSD 11 Dom0.
> The issue is that guest cannot connect to the device/vbd and requires to wait for 4 minutes to proceed, it goes through the countdown and starts fine (disk, networking)
> 
> [    6.684115] XENBUS: Waiting for devices to initialise: 25s...20s...15s...10s...5s...0s...235s...230s...225s...220s...215s...210s...205s...200s...195s...190s...185s...180s...175s...170s...165s...160s...155s...150s...145s...140s...135s...130s...125s...120s...115s...110s...105s...100s...95s...90s...85s...80s...75s...70s...65s...60s...55s...50s...45s...40s...35s...30s...25s...20s...15s...10s...5s...0s...
> [  271.591403] XENBUS: Timeout connecting to device: device/vbd/51712 (local state 3, remote state 1)
> [  271.599963] XENBUS: Device with no driver: device/vkbd/0
> [  271.604249]   Magic number: 1:453:334
> ...
> login: 
> 
> 
> Unlike Linux It's impossible to boot FreeBSD 11 guests from QCOW2 (xenbusb_nop_confighook_cb timeout)
> 
> Steps to reproduce:
> 1. Download qcow2 cirros image (small linux) 
> http://secure-web.cisco.com/1RtWXk7FS6UyRxe5A9ejZVfU2VK9512Yeds80mlP_0LoLwQJLjbPBoILL2e4QyF2OaMyO8fbrQrYVmfOybSYvHgWH1sRz1gK4Zi5XDAzBDUluwhlU4NxVQ7S17tH6vSTrIJmBJ_NO-sdA1Ph_KfSOsNvMkw-swwuKHD9ophVFfqEe6Lnt_PIP4H-LhZfp-5_aCuqbPYciXtMyxWbF1Od65jiFe-_LfmPRt53aCscOk7cBRI_-o603sb7htpDHH06Y8-M4oG5Lt4s1jr1XcKTrIWhnD-Lhz55blWyc2FnCgvk/http%3A%2F%2Fdownload.cirros-cloud.net%2F0.3.4%2Fcirros-0.3.4-x86_64-disk.img <http://secure-web.cisco.com/1RtWXk7FS6UyRxe5A9ejZVfU2VK9512Yeds80mlP_0LoLwQJLjbPBoILL2e4QyF2OaMyO8fbrQrYVmfOybSYvHgWH1sRz1gK4Zi5XDAzBDUluwhlU4NxVQ7S17tH6vSTrIJmBJ_NO-sdA1Ph_KfSOsNvMkw-swwuKHD9ophVFfqEe6Lnt_PIP4H-LhZfp-5_aCuqbPYciXtMyxWbF1Od65jiFe-_LfmPRt53aCscOk7cBRI_-o603sb7htpDHH06Y8-M4oG5Lt4s1jr1XcKTrIWhnD-Lhz55blWyc2FnCgvk/http%3A%2F%2Fdownload.cirros-cloud.net%2F0.3.4%2Fcirros-0.3.4-x86_64-disk.img> <http://secure-web.cisco.com/1h_h-Ne-wDq4bbwU9ZrC1722aqj6tnLpUb48xOrWDDI-W6cCX481vFPY87v33Z6en8zvIqxlgFkymKM5mDMhPoCbZSXzBlp7hYWr2FKVaEAU93FEW0HTQZ8ef6ygHIyGE6XTUgsLjgPZQfhl4_zNjWcIYzp6HWWVVZ3VrCNN-GEuMZ0JeedNQaKXavQhUK_dbVxh_4dLA1x9nKvutnTw5DDB-gECJjPjG07SrD4nh15HBLZi7Ua4X0PDRDbt3SGDcsqKO6oHPA2ajZjzWSLVp0l8PKgvNLjckBLRrWCFs8tQ/http%3A%2F%2Fdownload.cirros-cloud.net%2F0.3.4%2Fcirros-0.3.4-x86_64-disk.img <http://secure-web.cisco.com/1h_h-Ne-wDq4bbwU9ZrC1722aqj6tnLpUb48xOrWDDI-W6cCX481vFPY87v33Z6en8zvIqxlgFkymKM5mDMhPoCbZSXzBlp7hYWr2FKVaEAU93FEW0HTQZ8ef6ygHIyGE6XTUgsLjgPZQfhl4_zNjWcIYzp6HWWVVZ3VrCNN-GEuMZ0JeedNQaKXavQhUK_dbVxh_4dLA1x9nKvutnTw5DDB-gECJjPjG07SrD4nh15HBLZi7Ua4X0PDRDbt3SGDcsqKO6oHPA2ajZjzWSLVp0l8PKgvNLjckBLRrWCFs8tQ/http%3A%2F%2Fdownload.cirros-cloud.net%2F0.3.4%2Fcirros-0.3.4-x86_64-disk.img>>
> # file cirros-0.3.4-x86_64-disk.img 
> cirros-0.3.4-x86_64-disk.img: QEMU QCOW Image (v2), 41126400 bytes
> 2. create DomU from config bellow xl create -c config.cfg
> 
> builder = "hvm"
> memory = 512
> vcpus = 2
> name = "cirros"
> disk = [ 'file:qcow2:/root/cirros-0.3.4-x86_64-disk.img,xvda,w' ]
> boot = "c" 
> vnc = 1
> vnclisten = "0.0.0.0"
> usbdevice = 'tablet'
> on_poweroff = 'destroy'
> on_reboot = 'restart'
> on_crash = 'restart'
> acpi = 1
> serial = 'pty'
> 
> I've also tried multiple configurations like tap:qcow2:. tap2:qcow2:, aio:, switching from xen bus to ide. didn't work.
> The only driver that had no issues was PHY but it supports only RAW images.
> 
> Is that a bug or I'm missing something?
> 
> tested both STABLE snapshot and 11.0-RELEASE
> 
> # uname -a
> FreeBSD xen 11.0-STABLE FreeBSD 11.0-STABLE #0 r311441: Thu Jan  5 22:45:20 UTC 2017
> 
> # pkg info | grep xen
> xen-4.7.0_2                    Xen Hypervisor meta port
> xen-kernel-4.7.1_3             Hypervisor using a microkernel design
> xen-tools-4.7.1_1              Xen management tool, based on LibXenlight

So just that I understand this correctly, this is a FreeBSD 11.0-STABLE Dom0,
plus the Xen packages from pkg?

If that's the case, it's not going to work, FreeBSD 11.0 Dom0 doesn't yet
support qcow image format for HVM/PV guests, you will have to use FreeBSD 12
(HEAD) as your Dom0, or backport r308128 into STABLE (should be self contained,
so I don't expect any conflicts). You will also have to apply the following
patch to the xen-tools package and recompile:

https://lists.freebsd.org/pipermail/freebsd-xen/2016-August/002819.html

IIRC the right syntax to specify the disk device is:

'format=qcow2,vdev=xvda,access=rw,backendtype=qdisk,target=/root/cirros-0.3.4-x86_64-disk.img'

I'm also adding Akshay to the conversation, who did the gntdev implementation.

Roger.