Poor virtio performance on Scaleway ARM systems

[John Baldwin]

On 10/23/18 8:46 AM, Greg V wrote:
> 
> 
> On Sun, Sep 16, 2018 at 9:46 PM, Peter Jeremy <peter at rulingia.com> 
> wrote:
>> I have been playing with the 4-core ARM64 VPSs on 
>> https://www.scaleway.com
>> and notice that the disk I/O performance (using virtio_blk) is 
>> abyssmal.
>> Using "dd if=/dev/{vda|vtbd0} of=/dev/null bs=256k count=4096", I get
>> 400-500MBps, whilst under FreeBSD-12, I get about 5MBps.  I've 
>> checked on a
>> couple of instances and both Linux & FreeBSD on the same instance and 
>> get
>> similar results.  Linux & FreeBSD are both using a virtio block device
>> attached to the PCI bus.  Rebuilding FreeBSD to turn off all the 
>> debugging
>> has no effect.
>>
>> The only oddity I've found is that FreeBSD is reporting very high 
>> interrupt
>> rates on gic0,p11, gic0,s4 and gic0,s5 whilst disk I/O is occurring.
>> Unfortunately, I can't tell what is attached to those interrupts (it's
>> not obvious from the dmesg and reported as "+").
> gic0,s4 correlates to network activity.
> gic0,s5 correlates to disk activity.
> 
>> I've done some searching and have only found general FUD ("FreeBSD 
>> virtio
>> isn't any good") and nothing specifically related to Scaleway.
>>
>> Can anyone suggest where to look for a solution?
> 
> I FIGURED IT OUT!!
> 
> echo 'hw.pci.honor_msi_blacklist="0"' >> /boot/loader.conf
> 
> SOMEHOW FreeBSD ended up using legacy interrupts instead of MSI-X.
> 
> I have no idea how.
> Looking at pci_quirks, MSI disable is for old Intel stuff, some old 
> VMWare stuff... NOTHING about QEMU/KVM or Cavium!
> 
> With the fix, we're getting ~1000 interrupts on 'its0,4: virtio_pci1' 
> (instead of ~30000 on gic), latency is no longer awful, and bandwidth 
> is about 220 MB/s. YAY!!

Can you make 'pciconf -lc' output available?  There is a "generic"
blacklist that assumes we will see a host bridge device with PCI-e
or PCI-X and if neither of those is present we also blacklist MSI.
However, we can certainly figure out how to make this work out of
the box.

-- 
John Baldwin