Poor virtio performance on Scaleway ARM systems

[Greg V]

On Sun, Sep 16, 2018 at 9:46 PM, Peter Jeremy <peter at rulingia.com> 
wrote:
> I have been playing with the 4-core ARM64 VPSs on 
> https://www.scaleway.com
> and notice that the disk I/O performance (using virtio_blk) is 
> abyssmal.
> Using "dd if=/dev/{vda|vtbd0} of=/dev/null bs=256k count=4096", I get
> 400-500MBps, whilst under FreeBSD-12, I get about 5MBps.  I've 
> checked on a
> couple of instances and both Linux & FreeBSD on the same instance and 
> get
> similar results.  Linux & FreeBSD are both using a virtio block device
> attached to the PCI bus.  Rebuilding FreeBSD to turn off all the 
> debugging
> has no effect.
> 
> The only oddity I've found is that FreeBSD is reporting very high 
> interrupt
> rates on gic0,p11, gic0,s4 and gic0,s5 whilst disk I/O is occurring.
> Unfortunately, I can't tell what is attached to those interrupts (it's
> not obvious from the dmesg and reported as "+").
gic0,s4 correlates to network activity.
gic0,s5 correlates to disk activity.

> I've done some searching and have only found general FUD ("FreeBSD 
> virtio
> isn't any good") and nothing specifically related to Scaleway.
> 
> Can anyone suggest where to look for a solution?

I FIGURED IT OUT!!

echo 'hw.pci.honor_msi_blacklist="0"' >> /boot/loader.conf

SOMEHOW FreeBSD ended up using legacy interrupts instead of MSI-X.

I have no idea how.
Looking at pci_quirks, MSI disable is for old Intel stuff, some old 
VMWare stuff... NOTHING about QEMU/KVM or Cavium!

With the fix, we're getting ~1000 interrupts on 'its0,4: virtio_pci1' 
(instead of ~30000 on gic), latency is no longer awful, and bandwidth 
is about 220 MB/s. YAY!!