FreeBSD-13.1 and Xen-4.15.0 crash formerly working NetBSD-99.77 domu's
Date: Thu, 02 Jun 2022 09:16:35 UTC
hello. I just finished updating one of my FreeBSD Xen servers from 12.2-stable to 13.1-release. After a very painful upgrade process, which I'll detail in another e-mail, I discovered that my domu's running NetBSD-99.77, which had been running without any trouble whatsoever in pv mode, are failing with an unhandled general protection fault. Is this a known issue, a configuration error on my part, a bug in the NetBSD kernel, or something completely unknown? The details from xen from a sample crash are listed below. I'm happy to get more details, if they would be helpful. Just tell me what you would like to see and how I might go about getting it. Any light anyone can shed on this issue would be helpful. -thanks -Brian Xen host details: %uname -a FreeBSD xen-lothlorien.nfbcal.org 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64 Xen dmesg output: BC85DEB0, 0038 (r1 INTEL DQ67SW 1072009 AMI. 4) (XEN) ACPI: ASF! BC85DEE8, 00A0 (r32 INTEL DQ67SW 1 TFSM F4240) (XEN) ACPI: DMAR BC85DF88, 00E8 (r1 INTEL DQ67SW 1 INTL 1) (XEN) System RAM: 32683MB (33467896kB) (XEN) Domain heap initialised (XEN) ACPI: 32/64X FACS address mismatch in FADT - bcbdbf80/0000000000000000, using 32 (XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) PCI: Not using MCFG for segment 0000 bus 00-3f (XEN) Switched to APIC driver x2apic_cluster (XEN) CPU0: 1600 ... 3100 MHz (XEN) xstate: size: 0x340 and states: 0x7 (XEN) Speculative mitigation facilities: (XEN) Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD (XEN) Compiled-in support: SHADOW_PAGING (XEN) Xen settings: BTI-Thunk N/A, SPEC_CTRL: IBRS+ SSBD-, Other: IBPB L1D_FLUSH BRANCH_HARDEN (XEN) L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe address 1000000000 (XEN) Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU (XEN) Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU (XEN) XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID) (XEN) PV L1TF shadowing: Dom0 disabled, DomU disabled (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2) (XEN) Initializing Credit2 scheduler (XEN) Platform timer is 14.318MHz HPET (XEN) Detected 3093.000 MHz processor. (XEN) Intel VT-d iommu 0 supported page sizes: 4kB (XEN) Intel VT-d iommu 1 supported page sizes: 4kB (XEN) Intel VT-d Snoop Control not enabled. (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. (XEN) Intel VT-d Queued Invalidation enabled. (XEN) Intel VT-d Interrupt Remapping enabled. (XEN) Intel VT-d Posted Interrupt not enabled. (XEN) Intel VT-d Shared EPT tables not enabled. (XEN) I/O virtualisation enabled (XEN) - Dom0 mode: Relaxed (XEN) Interrupt remapping enabled (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) Allocated console ring of 16 KiB. (XEN) VMX: Supported advanced features: (XEN) - APIC MMIO access virtualisation (XEN) - APIC TPR shadow (XEN) - Extended Page Tables (EPT) (XEN) - Virtual-Processor Identifiers (VPID) (XEN) - Virtual NMI (XEN) - MSR direct-access bitmap (XEN) - Unrestricted Guest (XEN) HVM: ASIDs enabled. (XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207 (XEN) HVM: VMX enabled (XEN) HVM: Hardware Assisted Paging (HAP) detected (XEN) HVM: HAP page sizes: 4kB, 2MB (XEN) Brought up 4 CPUs (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource (XEN) Dom0 has maximum 440 PIRQs (XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0 (XEN) WARNING: PVH is an experimental mode with limited functionality (XEN) Initial low memory virq threshold set at 0x4000 pages. (XEN) Scrubbing Free RAM in background (XEN) Std. Loglevel: Errors and warnings (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings) (XEN) Xen is relinquishing VGA console. (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) (XEN) Freed 604kB init memory <Crash details here> (XEN) d2v0 Unhandled general protection fault fault/trap [#13, ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167 (XEN) Domain 2 (vcpu#0) crashed on cpu#1: (XEN) ----[ Xen-4.15.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e033:[<ffffffff8021f27a>] (XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest (d2v0) (XEN) rax: 0000000000070106 rbx: 000000000197d000 rcx: 0000000000000277 (XEN) rdx: 0000000000070106 rsi: 0000000000000002 rdi: ffffffff80718ec0 (XEN) rbp: ffffffff8197bf00 rsp: ffffffff8197be70 r8: 000000001fc9cbf5 (XEN) r9: 0000000000000000 r10: 000000000000000c r11: ffffffff80718f14 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: ffffffff81977000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000042660 (XEN) cr3: 0000000421342000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffffffff80718ec0 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff8197be70: (XEN) 0000000000000277 ffffffff80718f14 0000000000000000 ffffffff8021f27a (XEN) 000000010000e030 0000000000010002 ffffffff8197beb8 000000000000e02b (XEN) ffffffff8197bf00 ffffffff80232be9 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 00000000756e6547 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) d4v0 Unhandled general protection fault fault/trap [#13, ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167 (XEN) Domain 4 (vcpu#0) crashed on cpu#3: (XEN) ----[ Xen-4.15.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e033:[<ffffffff8021f27a>] (XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest (d4v0) (XEN) rax: 0000000000070106 rbx: 0000000000f7b000 rcx: 0000000000000277 (XEN) rdx: 0000000000070106 rsi: 0000000000000002 rdi: ffffffff80716ec0 (XEN) rbp: ffffffff80f79f00 rsp: ffffffff80f79e70 r8: 000000001fc9cbf5 (XEN) r9: 0000000000000000 r10: 000000000000000c r11: ffffffff80716f14 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: ffffffff80f75000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000042660 (XEN) cr3: 000000047b140000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffffffff80716ec0 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff80f79e70: (XEN) 0000000000000277 ffffffff80716f14 0000000000000000 ffffffff8021f27a (XEN) 000000010000e030 0000000000010002 ffffffff80f79eb8 000000000000e02b (XEN) ffffffff80f79f00 ffffffff80232be9 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 00000000756e6547 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) d5v0 Unhandled general protection fault fault/trap [#13, ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167 (XEN) Domain 5 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.15.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff8021f27a>] (XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest (d5v0) (XEN) rax: 0000000000070106 rbx: 000000000197d000 rcx: 0000000000000277 (XEN) rdx: 0000000000070106 rsi: 0000000000000002 rdi: ffffffff80718ec0 (XEN) rbp: ffffffff8197bf00 rsp: ffffffff8197be70 r8: 000000001fc9cbf5 (XEN) r9: 0000000000000000 r10: 000000000000000c r11: ffffffff80718f14 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: ffffffff81977000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000042660 (XEN) cr3: 0000000421f42000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffffffff80718ec0 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff8197be70: (XEN) 0000000000000277 ffffffff80718f14 0000000000000000 ffffffff8021f27a (XEN) 000000010000e030 0000000000010002 ffffffff8197beb8 000000000000e02b (XEN) ffffffff8197bf00 ffffffff80232be9 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 00000000756e6547 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) d6v0 Unhandled general protection fault fault/trap [#13, ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167 (XEN) Domain 6 (vcpu#0) crashed on cpu#3: (XEN) ----[ Xen-4.15.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e033:[<ffffffff8021f27a>] (XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest (d6v0) (XEN) rax: 0000000000070106 rbx: 0000000000f7b000 rcx: 0000000000000277 (XEN) rdx: 0000000000070106 rsi: 0000000000000002 rdi: ffffffff80716ec0 (XEN) rbp: ffffffff80f79f00 rsp: ffffffff80f79e70 r8: 000000001fc9cbf5 (XEN) r9: 0000000000000000 r10: 000000000000000c r11: ffffffff80716f14 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: ffffffff80f75000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000042660 (XEN) cr3: 0000000422940000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffffffff80716ec0 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff80f79e70: (XEN) 0000000000000277 ffffffff80716f14 0000000000000000 ffffffff8021f27a (XEN) 000000010000e030 0000000000010002 ffffffff80f79eb8 000000000000e02b (XEN) ffffffff80f79f00 ffffffff80232be9 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 00000000756e6547 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) d7v0 Unhandled general protection fault fault/trap [#13, ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167 (XEN) Domain 7 (vcpu#0) crashed on cpu#3: (XEN) ----[ Xen-4.15.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e033:[<ffffffff8021f27a>] (XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest (d7v0) (XEN) rax: 0000000000070106 rbx: 0000000000f7b000 rcx: 0000000000000277 (XEN) rdx: 0000000000070106 rsi: 0000000000000002 rdi: ffffffff80716ec0 (XEN) rbp: ffffffff80f79f00 rsp: ffffffff80f79e70 r8: 000000001fc9cbf5 (XEN) r9: 0000000000000000 r10: 000000000000000c r11: ffffffff80716f14 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: ffffffff80f75000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000042660 (XEN) cr3: 0000000523140000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffffffff80716ec0 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff80f79e70: (XEN) 0000000000000277 ffffffff80716f14 0000000000000000 ffffffff8021f27a (XEN) 000000010000e030 0000000000010002 ffffffff80f79eb8 000000000000e02b (XEN) ffffffff80f79f00 ffffffff80232be9 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 00000000756e6547 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) d2v0 Unhandled general protection fault fault/trap [#13, ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d040300b58 x86_64/entry.S#create_bounce_frame+0x14f/0x167 (XEN) Domain 2 (vcpu#0) crashed on cpu#1: (XEN) ----[ Xen-4.15.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e033:[<ffffffff8021f27a>] (XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest (d2v0) (XEN) rax: 0000000000070106 rbx: 000000000197d000 rcx: 0000000000000277 (XEN) rdx: 0000000000070106 rsi: 0000000000000002 rdi: ffffffff80718ec0 (XEN) rbp: ffffffff8197bf00 rsp: ffffffff8197be70 r8: 000000001fc9cbf5 (XEN) r9: 0000000000000000 r10: 000000000000000c r11: ffffffff80718f14 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: ffffffff81977000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000042660 (XEN) cr3: 0000000421342000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffffffff80718ec0 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff8197be70: (XEN) 0000000000000277 ffffffff80718f14 0000000000000000 ffffffff8021f27a (XEN) 000000010000e030 0000000000010002 ffffffff8197beb8 000000000000e02b (XEN) ffffffff8197bf00 ffffffff80232be9 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 00000000756e6547 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff8022a0b1 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000