Issues with XEN and ZFS

Fri Feb 8 19:35:28 UTC 2019

Hi.

Brief abstract: I'm having ZFS/Xen interaction issues with the disks being 
declared unusable by the dom0.

The longer bit:

I'm new to FreeBSD, so my apologies for all the stupid questions. I'm trying to 
migrate from Linux as my virtual platform host (very bad experiences with 
stability, let's leave it at that). I'm hosting mostly Solaris VMs (that being 
my choice of OS, but again, Betamax/VHS, need I say more), as well as a Windows 
VM (because I have to) and a Linux VM (as a future desktop via thin clients as 
and when I have to retire my SunRay solution which also runs on a VM for lack of 
functionality).

So, I got xen working on FreeBSD now after my newbie mistake was pointed out to me.

However, I seem to be stuck again:

I have, in this initial test server, only two disks. They are SATA hanging off 
the on-board SATA controller. The system is one of those Shuttle XPC cubes, an 
older one I had hanging around with 16GB memory and I think 4 cores.

I've given the dom0 2GB of memory and 2 core to start with.

The root filesystem is zfs with a mirror between the two disks.

The entire thing is dead easy to blow away and re-install as I was very 
impressed how easy the FreeBSD automatic installer was to understand and pick 
up, so I have it all scripted. If I need to blow stuff away to test, no problem 
and I can always get back to a known configuration.

As I only have two disks, I have created a zfs volume for the Xen domU thus:

zfs create -V40G -o volmode=dev zroot/nereid0

The domU nereid is defined thus:

cat - << EOI > /export/vm/nereid.cfg
builder = "hvm"
name = "nereid"
memory = 2048
vcpus = 1
vif = [ 'mac=00:16:3E:11:11:51,bridge=bridge0',
         'mac=00:16:3E:11:11:52,bridge=bridge1',
         'mac=00:16:3E:11:11:53,bridge=bridge2' ]
disk = [ '/dev/zvol/zroot/nereid0,raw,hda,rw' ]
vnc = 1
vnclisten = "0.0.0.0"
serial = "pty"
EOI

nereid itself also auto-installs, it's a Solaris 11.3 instance.

As it tries to install, I get this in the dom0:

Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): 
WRITE_FPDMA_QUEUED. ACB: 61 18 a0 ef 88 40 46 00 00 00 00 00
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): CAM status: 
CCB request was invalid
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): Error 22, 
Unretryable error
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada0:ahcich0:0:0:0): 
WRITE_FPDMA_QUEUED. ACB: 61 10 58 9d 86 40 3b 00 00 00 00 00
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada0:ahcich0:0:0:0): CAM status: 
CCB request was invalid
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada0:ahcich0:0:0:0): Error 22, 
Unretryable error
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk devd[555]: Processing event '!system=ZFS 
subsystem=ZFS type=ereport.fs.zfs.io class=ereport.fs.zfs.io 
ena=8237899368669119489 pool=zroot pool_guid=18382324475986588472 pool_context=0 
pool_failmode=wait vdev_guid=18239716500353510442 vdev_type=disk 
vdev_path=/dev/ada1p3 parent_guid=246995617967232936 parent_type=mirror 
zio_err=22 zio_offset=509174525952 zio_size=8192 zio_objset=236 zio_object=1 
zio_level=0 zio_blkid=1235478'
Feb  8 18:57:16 bianca.swangage.co.uk syslogd: exiting on signal 15
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada0:ahcich0:0:0:0): 
WRITE_FPDMA_QUEUED. ACB: 61 10 58 9d 86 40 3b 00 00 00 00 00
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada0:ahcich0:0:0:0): CAM status: 
CCB request was invalid
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada0:ahcich0:0:0:0): Error 22, 
Unretryable error
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): 
WRITE_FPDMA_QUEUED. ACB: 61 10 58 9d 86 40 3b 00 00 00 00 00
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): CAM status: 
CCB request was invalid
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times
Feb  8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): Error 22, 
Unretryable error
Feb  8 18:57:16 bianca.swangage.co.uk last message repeated 4 times

I get thousands or tens of thousands of these.

That has now lead to the system becoming unusable.

I didn't manage to complete the installation this time, but I did manage once 
for a different Solaris 11.3 domU (which also caused the same errors in the dom0 
during install but did complete) and once it runs, it still does happen, but not 
quite as often.

The issue does appear to be disk activity, in this particular instance: writes.

I'm wondering if I neglected a particular setting to make ZFS behave properly or 
perhaps the VM?

Any pointers would be greatly appreciated.

Thanks.

Eric

-- 

       ____
      /          .                           Eric A. Bautsch
     /--   __       ___                ______________________________________
    /     /    /   /                  /
   (_____/____(___(__________________/       email: eric.bautsch at pobox.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4127 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-xen/attachments/20190208/c8e6dcee/attachment.bin>