High vCPU Counts in bhyve

[Allan Jude]

On 2016-09-17 19:55, Peter Grehan wrote:
> Hi Trent,
> 
>> If anyone has any hints on how to get more information from this or ideas
>> on the apparent APIC error would be greatly appreciated.
> 
>  Thanks for the boot logs. The bhyve bug can be seen from:
> 
> ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C,
> should be 0x0A (20160527/tbprint-229)
> [7/1843]
> MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9
> 
>  .. which is pointing to the MADT table being overwritten. There is only
> 256 bytes allocated for this table when it is being created:
> 
> acpi.c
>  *       MADT  ->   0xf2500  (depends on #CPUs)
>  *       FADT  ->   0xf2600  (268 bytes)
> ...
> #define MADT_OFFSET             0x100
> #define FADT_OFFSET             0x200
> 
>  The MADT* for bhyve has a 44-byte fixed header, followed by a array of
> 8-byte 'Processor Local APIC' entries, one for each vCPU. The end of the
> table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source
> Override' entries, and a 6-byte 'Local APIC NMI' entry.
> 
> Looking at the max #CPUs that can fit into 256 bytes:
> 
> 256 = 44 + N*8 + 12 + 2*10 + 6, which gives N = 21. The fact that it
> worked for slightly larger values is probably due to the table entries
> at the end being ignored, until eventually the I/O APIC table entry was
> corrupted by the FADT overwrite.
> 
> A quick fix to get more vCPUs is to bump the addresses of the tables in
> acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs.
> 
> +#define FADT_OFFSET             0x700
> +#define HPET_OFFSET             0x840
> +#define MCFG_OFFSET             0x880
> +#define FACS_OFFSET             0x8C0
> +#define DSDT_OFFSET             0x900
> -#define FADT_OFFSET             0x200
> -#define HPET_OFFSET             0x340
> -#define MCFG_OFFSET             0x380
> -#define FACS_OFFSET             0x3C0
> -#define DSDT_OFFSET             0x400
> 
> I'll create a bug for this so that the overwrite will be detected at
> run-time, and also bump up the space to allow for some growth.
> 
> later,
> 
> Peter.
> 
> * see the ACPI spec at http://www.acpi.info/spec.htm for table details.
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscribe at freebsd.org"

Ha, so the only reason I didn't run into this, is the biggest VM i've
created just happened to be 20 vCPUs on my 24 CPU system.

Dodged a bullet, or, delay the discovery of this bug by 2 years. One of
the two.

-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-virtualization/attachments/20160917/fe9cfb89/attachment.sig>