High vCPU Counts in bhyve
Peter Grehan
grehan at freebsd.org
Sat Sep 17 23:55:26 UTC 2016
Hi Trent,
> If anyone has any hints on how to get more information from this or ideas
> on the apparent APIC error would be greatly appreciated.
Thanks for the boot logs. The bhyve bug can be seen from:
ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C,
should be 0x0A (20160527/tbprint-229)
[7/1843]
MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9
.. which is pointing to the MADT table being overwritten. There is
only 256 bytes allocated for this table when it is being created:
acpi.c
* MADT -> 0xf2500 (depends on #CPUs)
* FADT -> 0xf2600 (268 bytes)
...
#define MADT_OFFSET 0x100
#define FADT_OFFSET 0x200
The MADT* for bhyve has a 44-byte fixed header, followed by a array of
8-byte 'Processor Local APIC' entries, one for each vCPU. The end of the
table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source
Override' entries, and a 6-byte 'Local APIC NMI' entry.
Looking at the max #CPUs that can fit into 256 bytes:
256 = 44 + N*8 + 12 + 2*10 + 6, which gives N = 21. The fact that it
worked for slightly larger values is probably due to the table entries
at the end being ignored, until eventually the I/O APIC table entry was
corrupted by the FADT overwrite.
A quick fix to get more vCPUs is to bump the addresses of the tables in
acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs.
+#define FADT_OFFSET 0x700
+#define HPET_OFFSET 0x840
+#define MCFG_OFFSET 0x880
+#define FACS_OFFSET 0x8C0
+#define DSDT_OFFSET 0x900
-#define FADT_OFFSET 0x200
-#define HPET_OFFSET 0x340
-#define MCFG_OFFSET 0x380
-#define FACS_OFFSET 0x3C0
-#define DSDT_OFFSET 0x400
I'll create a bug for this so that the overwrite will be detected at
run-time, and also bump up the space to allow for some growth.
later,
Peter.
* see the ACPI spec at http://www.acpi.info/spec.htm for table details.
More information about the freebsd-virtualization
mailing list