Poor virtio performance on Scaleway ARM systems
John Baldwin
jhb at FreeBSD.org
Thu Oct 25 15:58:43 UTC 2018
On 10/23/18 11:49 AM, Peter Jeremy wrote:
> On 2018-Oct-23 10:04:04 -0700, John Baldwin <jhb at FreeBSD.org> wrote:
>> On 10/23/18 8:46 AM, Greg V wrote:
>>> On Sun, Sep 16, 2018 at 9:46 PM, Peter Jeremy <peter at rulingia.com>
>>> wrote:
>>>> I have been playing with the 4-core ARM64 VPSs on
>>>> https://www.scaleway.com
>>>> and notice that the disk I/O performance (using virtio_blk) is
>>>> abyssmal.
>>>>
>>>> The only oddity I've found is that FreeBSD is reporting very high
>>>> interrupt
>>>> rates on gic0,p11, gic0,s4 and gic0,s5 whilst disk I/O is occurring.
>>>> Unfortunately, I can't tell what is attached to those interrupts (it's
>>>> not obvious from the dmesg and reported as "+").
>>> gic0,s4 correlates to network activity.
>>> gic0,s5 correlates to disk activity.
>>>
>>> echo 'hw.pci.honor_msi_blacklist="0"' >> /boot/loader.conf
>
>> Can you make 'pciconf -lc' output available? There is a "generic"
>> blacklist that assumes we will see a host bridge device with PCI-e
>> or PCI-X and if neither of those is present we also blacklist MSI.
>> However, we can certainly figure out how to make this work out of
>> the box.
>
> $ sudo pciconf -lc
> hostb0 at pci0:0:0:0: class=0x060000 card=0x11001af4 chip=0x00081b36 rev=0x00 hdr=0x00
> virtio_pci0 at pci0:0:1:0: class=0x020000 card=0x00011af4 chip=0x10001af4 rev=0x00 hdr=0x00
> cap 11[98] = MSI-X supports 3 messages
> Table in map 0x14[0x0], PBA in map 0x14[0x800]
> cap 09[84] = vendor (length 20)
> cap 09[70] = vendor (length 20)
> cap 09[60] = vendor (length 16)
> cap 09[50] = vendor (length 16)
> cap 09[40] = vendor (length 16)
> virtio_pci1 at pci0:0:2:0: class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00
> cap 11[98] = MSI-X supports 2 messages
> Table in map 0x14[0x0], PBA in map 0x14[0x800]
> cap 09[84] = vendor (length 20)
> cap 09[70] = vendor (length 20)
> cap 09[60] = vendor (length 16)
> cap 09[50] = vendor (length 16)
> cap 09[40] = vendor (length 16)
The relevant code is here in sys/dev/pci/pci.c:
/*
* Determine if MSI is blacklisted globally on this system. Currently,
* we just check for blacklisted chipsets as represented by the
* host-PCI bridge at device 0:0:0. In the future, it may become
* necessary to check other system attributes, such as the kenv values
* that give the motherboard manufacturer and model number.
*/
static int
pci_msi_blacklisted(void)
{
device_t dev;
if (!pci_honor_msi_blacklist)
return (0);
/* Blacklist all non-PCI-express and non-PCI-X chipsets. */
if (!(pcie_chipset || pcix_chipset)) {
if (vm_guest != VM_GUEST_NO) {
/*
* Whitelist older chipsets in virtual
* machines known to support MSI.
*/
dev = pci_find_bsf(0, 0, 0);
if (dev != NULL)
return (!pci_has_quirk(pci_get_devid(dev),
PCI_QUIRK_ENABLE_MSI_VM));
}
return (1);
}
dev = pci_find_bsf(0, 0, 0);
if (dev != NULL)
return (pci_msi_device_blacklisted(dev));
return (0);
}
In this case no devices have PCI-express or PCI-X capabilities. We
could perhaps whitelist your hostb device, but I think a more general
solution might be to whitelist MSI if there are any virtio devices as
virtio devices can always do MSI. I can work on a patch unless someone
else beats me to it, but for the virtio approach what I would do is
have a global 'virtio_seen' similar to 'pcie_chipset' that gets set to
true if we find any virtio devices during the initial bus scans and
then add '|| virtio_seen' to the second 'if' there.
--
John Baldwin
More information about the freebsd-arm
mailing list