Panic: 32-bit PAE enabled OS with 4GB RAM and 2 vCPUs on Bhyve

[Prakhar Deep]

Hi,


I am facing kernel panic when 2nd processor is executing it's reset
code while running a 32-bit OS as a VM (with PAE enabled) on ‘bhyve’ with
following parameters:

1.       Number of vCPUs 2 or more,  and

2.       Memory 4GB or more

When number of vCPU =1 and memory > 4GB i.e. PAE mode with only one CPU or
 vCPU = 2 or more and memory < 4GB i.e non-PAE mode then VM comes up
successfully. The OS is able to bring up the boot processor but it panics
when AP is executing it's reset code.


The OS uses fixed kernel virtual addresses which are mapped by a level 2
page table which is replicated for each CPU to allow some of these mappings
to be different on a
per-CPU basis. These pages act as storage for many global variables which
are used when processor is executing reset code. When a second (non-boot)
CPU comes up, we are seeing that per-CPU pages for this CPU are reset to
all zeroes when the second CPU sets the PGE flag in CR4. This then leads to
incorrect code paths.

I am not able to understand why the pages mapped to AP are getting set to 0
when PGE flag is set in CR4 as this does not happen for boot processor. If
i do not set PGE flag then pages are not set to 0 but the code panics some
where else but not due to zeroed out global values.

I have tried FreeBSD i386 with PAE support compiled into the kernel on
Bhyve with >4GB RAM and 2 vCPUS and VM came up successfully. Also, the PGE
bit is set in the CR4 register for FreeBSD.


It will be helpful if anyone could throw some light on following points:



1. Why are the pages mapped to AP are being zeroed when PGE flag is set for
AP in it's CR4 ?

2. Are there are any changes required in a 32-bit OS to boot it in PAE mode
with 4GB or more RAM and 2 or more vCPUS on Bhyve ?

3. Was FreeBSD (i386) kernel updated to boot on Bhyve with 4GB or more RAM
and 2 or more vCPUS ?

Thanks.

Prakhar