I/O streams capped at 128KB/t?

[Andre Goree]

I'm trying to come up with a solution to an issue I'm seeing on FreeBSD 
VMs installed as qemu/KVM guests (I'm sure the fact that they're guests 
have nothing to do with my issue, but I thought I'd mention it anyway).

 From within a VM, I seem to be hitting a ~128KB/s limit on I/O streams, 
that only seems to be present in FreeBSD.  The first output below is to 
the local disk (local to the VM) on which the OS is installed, you can 
see that the KB/t (kilobytes per transfer) never go above ~128KB/t, 
though the IOPS (iostat tps field) go up as far as allowed for the VM.  
I ran a simple 'dd' with 'bs=1M count=1024' (columns are KB/t,tps,MB/s):


   0.00   0  0.00   0.00   0  0.00
  126.90 322 39.91   0.00   0  0.00
  127.59 1094 136.31   0.00   0  0.00
  126.79 1089 134.88   0.00   0  0.00
  127.85 1314 164.03   0.00   0  0.00
  127.57 1340 166.91   0.00   0  0.00
  127.56 1322 164.69   0.00   0  0.00
  127.64 1331 165.87   0.00   0  0.00
  126.87 425 52.71   0.00   0  0.00
   7.00   1  0.01   0.00   0  0.00
   0.00   0  0.00   0.00   0  0.00


The second disk I added (which is an RBD network disk, though I don't 
think that matters, thought I'd mention it) has it's IOPS limited to 75 
(with some bursting allowed).  You can see that the KB/t still remains 
below 128KB/t, which creates a bottleneck as the IOPS cannot increase 
for this disk, giving it a very low throughput of ~9MB/s, which is 
horrible write speed for my case.


   0.00   0  0.00  124.69  75  9.17   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.86  74  9.05   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.64  75  9.19   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.81  75  9.09   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.61  76  9.23   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.80  74  9.04   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.77  75  9.14   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.56  76  9.22   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  123.24  75  9.03   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.97  75  9.15   0.00   0  0.00   0.00   0  0.00
   4.00   1  0.00  124.73  75  9.13   0.00   0  0.00   0.00   0  0.00
   0.00   0  0.00  124.73  75  9.14   0.00   0  0.00   0.00   0  0.00


 From within any number of Linux VMs (I've tried Ubuntu, Arch, and 
Fedora) on the same platform, using the RBD disk I do not have the same 
issue with speed, as those systems seem to scale up their data per 
second -- discounting the fact that the IOPS in this case burst up a bit 
to 118tps (this is from a VM running Arch Linux; columns are 
tps,KB_read/s,KB_wrtn/s,KB_read,KB_wrtn):

sdb               4.00         8.00        28.00          8         28
sdb              26.00         4.00     23820.00          4      23820
sdb              31.00         0.00     31488.00          0      31488
sdb             118.00         0.00    120832.00          0     120832
sdb              90.10         0.00     93275.25          0      94208
sdb               0.00         0.00         0.00          0          0



My question is, am I able to increase the KB/t from 128KB/t?  Or is this 
simply a limit of the FreeBSD kernel?  I'd prefer to be able to increase 
the KB/t (from within the VM) rather than IOPS (which I can do from the 
host system) if at all possible.

Thanks in advance for any insight.


-- 
Andre Goree
-=-=-=-=-=-
Email     - andre at drenet.net
Website   - http://blog.drenet.net
PGP key   - http://www.drenet.net/pubkey.html
-=-=-=-=-=-