Re: Stable/13 doesn't boot with xen

From: Brian Buhrow <buhrow_at_nfbcal.org>
Date: Tue, 07 Jun 2022 06:05:05 UTC
	hello Roger.  I tried booting my system from a head kernel and it boots ans runs
successfully, boot logs shown below.  However, there is an issue both with the working
releng/13.1 and the head kernel, which is that upon shutdown, the system panics.  It's mostly
harmless since it seems to do it after all buffers are synced, but it doesn't panic if FreeBSD
is running on bare metal and it doesn't look nice when  operators see it.

I've captured the logs from the failure on head, but I've reverted the system back to
releng/13.1 with the netback patch installed, so it's back in production mode.
For this message, I'll include the boot log for the head kernel.

I'll followup with a second message on this thread with the shutdown log, which fails.

Hope this helps you track down the issue with stable/13 as well.

-thanks
-Brian


Script started on Mon Jun  6 09:38:54 2022

[  Not too informative shutdown log from releng/13.1, but shows the panic. ]

login: Jun  6 09:39:28 xen-lothlorien shutdown[7223]: reboot by buhrow: 
Stopping sshd.
Waiting for PIDS: 1299.
Stopping cron.
Waiting for PIDS: 1283.
Stopping devd.
Waiting for PIDS: 544.
Writing entropy file: .
Writing early boot entropy file: .
Terminated
.
Jun  6 09:39:29 xen-lothlorien syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop... 
Syncing disks, vnodes remaining... 2 1 0 done
All buffers synced.
Uptime: 2d0h10m24s
GEOM_MIRROR: Device gptback: provider destroyed.
GEOM_MIRROR: Device gptback destroyed.
GEOM_MIRROR: Device gptswap: provider destroyed.
GEOM_MIRROR: Device gptswap destroyed.
GEOM_MIRROR: Device gptroot: provider destroyed.
GEOM_MIRROR: Device gptroot destroyed.
uhub4: detached
uhub1: detached
uhub0: detached
uhub3: detached
uhub2: detached


Fatal trap 12: page fault while in kernel mode
cpuid = 1

[ ... cuts to boot log with head kernel ... ]

 Xen 4.16.0
(XEN) Xen version 4.16.0 (buhrow@) (FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)) debug=n Thu Jun  2 11:30:17 PDT 2022
(XEN) Latest ChangeSet: 
(XEN) Bootloader: FreeBSD Loader
(XEN) Command line: dom0_mem=8192m dom0_max_vcpus=2 dom0=pvh pv-l1tf=off,domu=off console=com1,vga com1=9600,8n1
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  [0000000000000000, 00000000000997ff] (usable)
(XEN)  [0000000000099800, 000000000009ffff] (reserved)
(XEN)  [00000000000e0000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 000000001fffffff] (usable)
(XEN)  [0000000020000000, 00000000201fffff] (reserved)
(XEN)  [0000000020200000, 000000003fffffff] (usable)
(XEN)  [0000000040000000, 00000000401fffff] (reserved)
(XEN)  [0000000040200000, 00000000bc855fff] (usable)
(XEN)  [00000000bc856000, 00000000bc85efff] (ACPI data)
(XEN)  [00000000bc85f000, 00000000bc8a9fff] (ACPI NVS)
(XEN)  [00000000bc8aa000, 00000000bc8b1fff] (usable)
(XEN)  [00000000bc8b2000, 00000000bc9a4fff] (reserved)
(XEN)  [00000000bc9a5000, 00000000bc9a6fff] (usable)
(XEN)  [00000000bc9a7000, 00000000bcbc5fff] (reserved)
(XEN)  [00000000bcbc6000, 00000000bcbc6fff] (usable)
(XEN)  [00000000bcbc7000, 00000000bcbd6fff] (reserved)
(XEN)  [00000000bcbd7000, 00000000bcbf4fff] (ACPI NVS)
(XEN)  [00000000bcbf5000, 00000000bcc18fff] (reserved)
(XEN)  [00000000bcc19000, 00000000bcc5bfff] (ACPI NVS)
(XEN)  [00000000bcc5c000, 00000000bce7bfff] (reserved)
(XEN)  [00000000bce7c000, 00000000bcffffff] (usable)
(XEN)  [00000000bd800000, 00000000bf9fffff] (reserved)
(XEN)  [00000000fed1c000, 00000000fed3ffff] (reserved)
(XEN)  [00000000ff000000, 00000000ffffffff] (reserved)
(XEN)  [0000000100000000, 000000083e5fffff] (usable)
(XEN) New Xen image base address: 0xbc200000
(XEN) ACPI: RSDP 000F0450, 0024 (r2  INTEL)
(XEN) ACPI: XSDT BC856070, 0064 (r1 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: FACP BC85DBC0, 00F4 (r4 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: DSDT BC856168, 7A54 (r2 INTEL  DQ67SW         16 INTL 20051117)
(XEN) ACPI: FACS BCBDBF80, 0040
(XEN) ACPI: APIC BC85DCB8, 0072 (r3 INTEL  DQ67SW    1072009 AMI     10013)
(XEN) ACPI: TCPA BC85DD30, 0032 (r2 INTEL  DQ67SW          1 MSFT  1000013)
(XEN) ACPI: SSDT BC85DD68, 0102 (r1 INTEL  DQ67SW          1 MSFT  3000001)
(XEN) ACPI: MCFG BC85DE70, 003C (r1 INTEL  DQ67SW    1072009 MSFT       97)
(XEN) ACPI: HPET BC85DEB0, 0038 (r1 INTEL  DQ67SW    1072009 AMI.        4)
(XEN) ACPI: ASF! BC85DEE8, 00A0 (r32 INTEL  DQ67SW          1 TFSM    F4240)
(XEN) ACPI: DMAR BC85DF88, 00E8 (r1 INTEL  DQ67SW          1 INTL        1)
(XEN) System RAM: 32683MB (33467896kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - bcbdbf80/0000000000000000, using 32
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) PCI: Not using MCFG for segment 0000 bus 00-3f
(XEN) Switched to APIC driver x2apic_cluster
(XEN) CPU0: 1600 ... 3100 MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) Speculative mitigation facilities:
(XEN)   Hardware hints:
(XEN)   Hardware features: IBPB IBRS STIBP SSBD L1D_FLUSH
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- STIBP- SSBD-, Other: IBPB L1D_FLUSH BRANCH_HARDEN
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe address 1000000000
(XEN)   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 3092.985 MHz processor.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 4 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Dom0 has maximum 440 PIRQs
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) WARNING: PVH is an experimental mode with limited functionality
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 600kB init memory
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2022 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-71df58aef: Mon Jun  6 07:46:07 PDT 2022
    buhrow@fbsd_dev.nfbcal.org:/usr/home/buhrow/obj/usr/home/buhrow/src/fbsd-src/current/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 14.0.4 (https://github.com/llvm/llvm-project.git llvmorg-14.0.4-0-g29f1039a7285)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
XEN: Hypervisor version 4.16 detected.
CPU microcode: no matching update found
CPU: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (3092.98-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
  Features=0x1fc3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0x9fba2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  AMD Extended Feature Extensions ID EBX=0x101000<IBPB>
  TSC: P-state invariant
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 9725026304 (9274 MB)
avail memory = 8252633088 (7870 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <INTEL  DQ67SW  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-23
Launching APs: 1
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: <VT VGA driver>
smbios0: <System Management BIOS> at iomem 0xf0480-0xf049e
smbios0: Version: 2.6, BCD Revision: 2.6
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <INTEL DQ67SW>
acpi0: Power Button (fixed)
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
hpet0: memory region width 1024 too small for 32 timers
device_attach: hpet0 attach returned 6
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mps0: <Avago Technologies (LSI) SAS2008> port 0xe000-0xe0ff mem 0xfe6c0000-0xfe6c3fff,0xfe680000-0xfe6bffff irq 16 at device 0.0 on pci1
mps0: Firmware: 15.00.00.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
pci0: <simple comms> at device 22.0 (no driver attached)
atapci0: <Intel ATA controller> port 0xf130-0xf137,0xf120-0xf123,0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0ff irq 18 at device 22.2 on pci0
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
uart2: <Intel AMT - KT Controller> port 0xf0e0-0xf0e7 mem 0xfe729000-0xfe729fff irq 17 at device 22.3 on pci0
uart2: Using 1 MSI message
em0: <Intel(R) 82579LM> port 0xf080-0xf09f mem 0xfe700000-0xfe71ffff,0xfe728000-0xfe728fff irq 20 at device 25.0 on pci0
em0: EEPROM V0.13-4
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:22:4d:56:3c:f7
em0: netmap queues/slots: TX 1/1024, RX 1/1024
ehci0: <Intel Cougar Point USB 2.0 controller> mem 0xfe727000-0xfe7273ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
hdac0: <Intel Cougar Point HDA Controller> mem 0xfe720000-0xfe723fff irq 22 at device 27.0 on pci0
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci3: <ACPI PCI bus> on pcib3
xhci0: <NEC uPD720200 USB 3.0 controller> mem 0xfe500000-0xfe501fff irq 18 at device 0.0 on pci3
xhci0: 32 bytes context size, 32-bit DMA
usbus1 on xhci0
usbus1: 5.0Gbps Super Speed USB v3.0
ehci1: <Intel Cougar Point USB 2.0 controller> mem 0xfe726000-0xfe7263ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
usbus2: 480Mbps High Speed USB v2.0
pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci4: <serial bus, FireWire> at device 3.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f mem 0xfe725000-0xfe7257ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
acpi_button0: <Power Button> on acpi0
tpm0: <Trusted Platform Module> iomem 0xfed40000-0xfed44fff on acpi0
tpm: device 0xffffffff rev 0xff
WARNING: Device "tpm" is Giant locked and may be deleted before FreeBSD 14.0.
xenpv0: <Xen PV bus>
granttable0: <Xen Grant-table Device> on xenpv0
xc0: <Xen Console> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xen_et0: registered as a time-of-day clock, resolution 0.000001s
xenstore0: <XenStore> on xenpv0
xsd_dev0: <Xenstored user-space device> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
gntdev0: <Xen grant-table user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
orm0: <ISA Option ROM> at iomem 0xd4800-0xd57ff pnpid ORM0000 on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbdc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14.
Timecounter "TSC-low" frequency 1546492500 Hz quality 1000
Timecounters tick every 10.000 msec
hdacc0: <Realtek ALC888 HDA CODEC> at cad 2 on hdac0
hdaa0: <Realtek ALC888 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC888 (Rear Analog)> at nid 20 and 24,26 on hdaa0
pcm1: <Realtek ALC888 (Front Analog)> at nid 27 and 25 on hdaa0
pcm2: <Realtek ALC888 (Onboard Digital)> at nid 30 on hdaa0
pcm3: <Realtek ALC888 (Onboard Analog Speaker)> at nid 23 on hdaa0
hdacc1: <Intel Cougar Point HDA CODEC> at cad 3 on hdac0
hdaa1: <Intel Cougar Point Audio Function Group> at nid 1 on hdacc1
pcm4: <Intel Cougar Point (HDMI/DP 8ch)> at nid 6 on hdaa1
WARNING: WITNESS option enabled, expect reduced performance.
ugen2.1: <Intel EHCI root HUB> at usbus2
ugen0.1: <Intel EHCI root HUB> at usbus0
uhub0 on usbus2
uhub1 on usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <(0x1033) XHCI root HUB> at usbus1
uhub2 on usbus1
Trying to mount root from ufs:/dev/mirror/gptroot []...
uhub2: <(0x1033) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
uhub2: 4 ports with 4 removable, self powered
Root mount waiting for: CAM usbus0 usbus2
uhub0: 3 ports with 3 removable, self powered
uhub1: 3 ports with 3 removable, self powered
ugen0.2: <vendor 0x8087 product 0x0024> at usbus0
uhub3 on uhub1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen2.2: <vendor 0x8087 product 0x0024> at usbus2
uhub4 on uhub0
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus2
ses0 at ahciem0 bus 0 scbus4 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses0: cd0 in 'Slot 00', SATA Slot: scbus3 target 0
cd0 at ahcich0 bus 0 scbus3 target 0 lun 0
cd0: <ATAPI iHAS224   B GL0B> Removable CD-ROM SCSI device
cd0: Serial Number 3524220 228217500534
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2: <ATA WDC WD82PURZ-85T 0A82> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number VGKUGHSG
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 7630885MB (15628053168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <ATA WDC WD81PURZ-85L 0A80> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number VAJ3RZJL
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 7630885MB (15628053168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3: <ATA WDC WD81PURZ-85L 0A80> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number VAJYL6HL
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 7630885MB (15628053168 512 byte sectors)
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <ATA WDC WD81PURZ-85L 0A80> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number VAJ3Y51L
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 7630885MB (15628053168 512 byte sectors)
GEOM_MIRROR: Device mirror/gptroot launched (2/2).
GEOM_MIRROR: Device mirror/gptswap launched (2/2).
GEOM_MIRROR: Device mirror/gptback launched (2/2).
Root mount waiting for: usbus0 usbus2
uhub3: 6 ports with 6 removable, self powered
uhub4: 8 ports with 8 removable, self powered
Setting hostuuid: 49434d53-0200-9009-2500-09902500dac8.
Setting hostid: 0x0535156f.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM: zvol/xendisks/loth_9_install: invalid disklabel.
Starting file system checks:
/dev/mirror/gptroot: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mirror/gptroot: clean, 56544108 free (28156 frags, 7064494 blocks, 0.0% fragmentation)
Mounting local filesystems:.
Setting hostname: xen-lothlorien.nfbcal.org.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/gcc10 /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path: /usr/lib32
Setting up harvesting: [CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
Autoloading module: ichsmb
ichsmb0: <Intel Cougar Point SMBus controller> port 0xf040-0xf05f mem 0xfe724000-0xfe7240ff irq 18 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
bridge0: Ethernet address: 58:9c:fc:10:94:73
Created clone interfaces: bridge0.
lo0: link state changed to UP
em0: link state changed to UP
em0: promiscuous mode enabled
bridge0: link state changed to UP
Starting Network: lo0 em0 bridge0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
	ether 00:22:4d:56:3c:f7
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 58:9c:fc:10:94:73
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 20000
	groups: bridge
	nd6 options=9<PERFORMNUD,IFDISABLED>
Starting devd.
Sleeping 60  seconds before starting...
Waiting 60s for the default route interface: .....(no carrier)
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add net default: gateway 10.40.60.1
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating /var/run/os-release done.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating motd:.
Starting syslogd.
Setting date via ntp.
 6 Jun 09:48:08 ntpdate[767]: step time server 209.81.9.7 offset +4.800691 sec
Mounting late filesystems:.
Starting xenservices: xenstored, xenconsoled.Jun  6 09:48:09 xen-lothlorien xenstored[812]: Checking store ...
Jun  6 09:48:09 xen-lothlorien xenstored[812]: Checking store complete.
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_back0: <Xen Backend Devices> on xenstore0

Setting domain 0 name, domid and JSON config...
Done setting up Dom0
Starting xenservices: xenconsoled.
Setting domain 0 name, domid and JSON config...
Dom0 is already set up