Booting r298488 as Xen Dom0 may break ZFS pool?

[Roger Pau Monné]

On Sat, Apr 23, 2016 at 10:40:20PM +0000, Marcin Cieslak wrote:
> On a freshly installed (via upgrade from 10.3 from source)
> -CURRENT on this machine:
> 
> FreeBSD 11.0-CURRENT #0 r298488: Sat Apr 23 11:10:01 UTC 2016
>     root at o.saper.info:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
> WARNING: WITNESS option enabled, expect reduced performance.
> VT(vga): resolution 640x480
> CPU: Intel(R) Xeon(R) CPU E31245 @ 3.30GHz (3300.09-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
>   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>   Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
>   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>   AMD Features2=0x1<LAHF>
>   XSAVE Features=0x1<XSAVEOPT>
>   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>   TSC: P-state invariant, performance statistics
> real memory  = 17179869184 (16384 MB)
> avail memory = 16475140096 (15711 MB)
> Event timer "LAPIC" quality 600
> ACPI APIC Table: <ALASKA A M I>
> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
> FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
> random: unblocking device.
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> 
> ... i am trying to boot the system as Dom0 under Xen (4.5.2_2
> installed via pkg install).
> 
> The kernel boots in one of four cases, mostly
> though I don't get a block cursor after Xen messages
> and the machine sits and waits.
> 
> 
> After trying Xen suddenly the system no longer
> boots again:
> 
> Booting from local disk...
> PXE-M0F: Existing Intel Boot Agent.
> 
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> 
> Can't find /boot/zfsloader
> 
> FreeBSD/x86 boot
> Default: zroot:/boot/kernel/kernel
> boot:
> ZFS: i/o error - all block copies unavailable

I'm currently running a r298464 world+kernel with ZFS on root and have seen 
no issues so far.
 
> (...)
> 
> The zpool imports without problems when
> booting from the rescue mfsbsd (10.3):
> 
>   pool: zroot
>  state: ONLINE
>   scan: none requested
> config:
> 
> 	NAME        STATE     READ WRITE CKSUM
> 	zroot       ONLINE       0     0     0
> 	  mirror-0  ONLINE       0     0     0
> 	    ada0p3  ONLINE       0     0     0
> 	    ada1p3  ONLINE       0     0     0
> 
> errors: No known data errors
> 
> gpart layout:
> 
> =>        34  5860533101  ada0  GPT  (2.7T)
>           34        1024     1  freebsd-boot  (512K)
>         1058     4194304     2  freebsd-swap  (2.0G)
>      4195362  5856337773     3  freebsd-zfs  (2.7T)
> 
> =>        34  5860533101  ada1  GPT  (2.7T)
>           34        1024     1  freebsd-boot  (512K)
>         1058     4194304     2  freebsd-swap  (2.0G)
>      4195362  5856337773     3  freebsd-zfs  (2.7T)
> 
> I have managed to make zpool boot again by doing voodoo
> similar to this one:
> 
> [root at rescue ~]# zpool import -R /mnt zroot
> [root at rescue ~]# mount -t devfs devfs /mnt/dev
> [root at rescue ~]# chroot /mnt /bin/tcsh
> 
> (... Running make install in /usr/src/sys/boot ...)
> 
> root at rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada0
> partcode written to ada0p1
> root at rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada1
> partcode written to ada1p1
> root at rescue:/ # exit
> [root at rescue ~]# umount /mnt/dev
> [root at rescue ~]# zpool export zroot
> [root at rescue ~]# reboot
> 
> Why zpool metadata get corrupted?

TBH, I have no idea. Can you also paste the log of the system (Xen + 
FreeBSD) when it fails to boot? If that's not possible, can you at least 
paste the output of `xl dmesg` when booted correctly under Xen?

What operations did you perform when the system booted correctly using 
FreeBSD/Xen?

Does the disk get corrupted even if the system fails to boot? AFAICT, it 
seems like it's only the bootcode that gets corrupted, is that right?

Roger.