Xen kernel fails to boot, d0v1 triple fault looks like the cuplrit

[Roger Pau Monné]

On Wed, May 23, 2018 at 12:57:59PM +0530, Pratyush Yadav wrote:
> On Mon, May 21, 2018 at 2:33 PM, Roger Pau Monné <roger.pau at citrix.com>
> wrote:
> > Please try to avoid top posting.
> 
> Sorry, I didn't know. I Googled top posting just now, and realized it is
> inconvenient for the reader.
> 
> > Hm, it seems like dbg_stack is not properly allocated. Can you please
> > try the above debug patch and paste the boot log?
> >
> > ---8<---
> > diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c
> > index 301461420874..8854242b4bf5 100644
> > --- a/sys/amd64/amd64/mp_machdep.c
> > +++ b/sys/amd64/amd64/mp_machdep.c
> > @@ -304,6 +304,7 @@ init_secondary(void)
> >
> >         /* Save the per-cpu pointer for use by the DB# handler. */
> >         np = ((struct nmi_pcpu *) &dbg_stack[PAGE_SIZE]) - 1;
> > +printf("ID %d dbg_stack: %p per-cpu: %p\n", cpu, dbg_stack, np);
> >         np->np_pcpu = (register_t) pc;
> >
> >         wrmsr(MSR_FSBASE, 0);           /* User value */
> > @@ -403,6 +404,7 @@ native_start_all_aps(void)
> >                     M_WAITOK | M_ZERO);
> >                 dbg_stack = (char *)kmem_malloc(kernel_arena, PAGE_SIZE,
> >                     M_WAITOK | M_ZERO);
> > +printf("allocated dbg_stack: %p\n", dbg_stack);
> >                 dpcpu = (void *)kmem_malloc(kernel_arena, DPCPU_SIZE,
> >                     M_WAITOK | M_ZERO);
> >
> 
> I applied your patch. The boot stops at the same spot. Also, I can't see
> any of the two print messages added in the patch. I booted from a Live CD
> to look at the logs, but they are not stored. I checked dmesg.boot and
> messages, and /var/log/xen/ none of them have the logs. They all have logs
> of the previous boot.

It's too early for the logs to be stored anywhere. The point where you
get the crash is when the APs are started, which is way before FreeBSD
starts proving for disk devices.

Can you please try the patch below?

Thanks, Roger.
---8<---
diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c
index 54184898e9bf..52391e2e7c08 100644
--- a/sys/x86/xen/pv.c
+++ b/sys/x86/xen/pv.c
@@ -113,6 +113,7 @@ static int xen_pv_start_all_aps(void);
 extern char *doublefault_stack;
 extern char *mce_stack;
 extern char *nmi_stack;
+extern char *dbg_stack;
 #endif
 
 /*
@@ -329,6 +330,8 @@ start_xen_ap(int cpu)
 	    (char *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
 	nmi_stack =
 	    (char *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
+	dbg_stack =
+	    (void *)kmem_malloc(kernel_arena, PAGE_SIZE, M_WAITOK | M_ZERO);
 	dpcpu =
 	    (void *)kmem_malloc(kernel_arena, DPCPU_SIZE, M_WAITOK | M_ZERO);