Gmirror performanc

[Ivan Voras]

Guido van Rooij wrote:

> But a single-threaded process accessing a large file should also be
> able to see a speed increase. I really do not see why a split or roundrobin
> approach to a mirror would get only half the performance of a raw access
> to a non-mirror. Somehow there must a limit of 128KB of outstanding requests.
> That limit is software imposed and should IMHO scale with the number of
> spindles in a mirror...

I've recently toyed with creating RAID arrays from 2 USB memory sticks
(for fun :) ), and this is what I've found out (I may create a web page
with the results if I find the time):

- diskinfo -t doesn't work on small drives (needs at least 1 GB drive,
maybe larger)
- maximum read data rate of my USB sticks is ~~ 7.5 MB/s (with dd), but
it can be achieved only with large requests (64 KB/s, as seen by iostat)
- the speed declines when using smaller requests, to ~~ 4 MB/s with 4 KB
requests (CPU utilisation is still ~~ 0 - the slowdown is in the
controller or the drive)
- gmirror split balance algorithm works as advertised - requests larger
than split size are delegated to multiple devices. This has the expected
effect of achieving double throughput (15 MB/s) with large enough requests.
- there's a (somewhat expected but still surprising) interaction with
MAXPHYS (maximum IO size) in the kernel. MAXPHYS is 128 KB, which means
requests from TO gmirror (and other GEOM devices) will be <= 128 KB,
which gmirror will split in two, so we arrive to maximum size of 64 KB
that's requested from the hardware. In case of a four-way mirror,
maximum size to hardware would be 32 KB, etc. I don't know if this is
what's slowing gmirror down, but it might, since current drives are
apparently best tuned for large requests.

It probably makes sense to use gmirror split size of around FS block
size (16 KB) to make use of split access but not to make too small requests.

I've done a round of testing with gstripe and it behaves as expected
(and doesn't have the side-effect of splitting requests).