High vCPU Counts in bhyve

Peter Grehan grehan at freebsd.org
Sat Sep 17 23:55:26 UTC 2016


Hi Trent,

> If anyone has any hints on how to get more information from this or ideas
> on the apparent APIC error would be greatly appreciated.

  Thanks for the boot logs. The bhyve bug can be seen from:

ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, 
should be 0x0A (20160527/tbprint-229)
[7/1843]
MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9

  .. which is pointing to the MADT table being overwritten. There is 
only 256 bytes allocated for this table when it is being created:

acpi.c
  *       MADT  ->   0xf2500  (depends on #CPUs)
  *       FADT  ->   0xf2600  (268 bytes)
...
#define MADT_OFFSET             0x100
#define FADT_OFFSET             0x200

  The MADT* for bhyve has a 44-byte fixed header, followed by a array of 
8-byte 'Processor Local APIC' entries, one for each vCPU. The end of the 
table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source 
Override' entries, and a 6-byte 'Local APIC NMI' entry.

Looking at the max #CPUs that can fit into 256 bytes:

256 = 44 + N*8 + 12 + 2*10 + 6, which gives N = 21. The fact that it 
worked for slightly larger values is probably due to the table entries 
at the end being ignored, until eventually the I/O APIC table entry was 
corrupted by the FADT overwrite.

A quick fix to get more vCPUs is to bump the addresses of the tables in 
acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs.

+#define FADT_OFFSET             0x700
+#define HPET_OFFSET             0x840
+#define MCFG_OFFSET             0x880
+#define FACS_OFFSET             0x8C0
+#define DSDT_OFFSET             0x900
-#define FADT_OFFSET             0x200
-#define HPET_OFFSET             0x340
-#define MCFG_OFFSET             0x380
-#define FACS_OFFSET             0x3C0
-#define DSDT_OFFSET             0x400

I'll create a bug for this so that the overwrite will be detected at 
run-time, and also bump up the space to allow for some growth.

later,

Peter.

* see the ACPI spec at http://www.acpi.info/spec.htm for table details.


More information about the freebsd-virtualization mailing list