N-way mirror read speedup in zfsonlinux

[Steven Hartland]

So based on mav's comments I've created a new version of the load balanced
patch which takes into account the last IO's locality on the disk.

I've added detection of non-rotating media, that also triggers the locality
bonus, which significantly improves mixed HDD and SSD mirrors.

I've also removed the time based switching as this was flawed for N-Way
mirrors where N != 2, and didn't see to provide any benefit either.

One question I did have; is it necessary to use protect the call to
avl_numnodes with the vq_lock mutex, as this seems like it may not
actually be needed given its not iterating or modifying the avl?

My tests show this version provides a significant performance increases
on both from the original strip balancing as well as the zfsonlinux load
balancing version; resulting in up to 3 x the read rates on an 3 way
mirror with 2 x HDD's and 1 x SSD.

The patch is attached, comments?

Here is my full set of test results:

== Setup ==
3 Way Mirror with 2 x HD's and 1 x SSD

=== Prefetch Disabled ==
==== Load Balanced (locality) ====
Read 15360MB using bs: 1048576, readers: 3, took 54 seconds @ 284 MB/s
Read 30720MB using bs: 1048576, readers: 6, took 77 seconds @ 398 MB/s
Read 46080MB using bs: 1048576, readers: 9, took 89 seconds @ 517 MB/s

==== Stripe Balanced (default) ====
Read 15360MB using bs: 1048576, readers: 3, took 161 seconds @ 95 MB/s

==== Load Balanced (zfslinux) ====
Read 15360MB using bs: 1048576, readers: 3, took 297 seconds @ 51 MB/s

=== Prefetch Enabled ===
==== Load Balanced (locality) ====
Read 15360MB using bs: 1048576, readers: 3, took 48 seconds @ 320 MB/s

==== Stripe Balanced (default) ====
Read 15360MB using bs: 1048576, readers: 3, took 91 seconds @ 168 MB/s

==== Load Balanced (zfslinux) ====
Read 15360MB using bs: 1048576, readers: 3, took 108 seconds @ 142 MB/s

== Setup ==
2 Way Mirror with 2 x HD's

=== Prefetch Disabled ==
==== Load Balanced (locality) ====
Read 10240MB using bs: 1048576, readers: 2, took 131 seconds @ 78 MB/s

==== Stripe Balanced (default) ====
Read 10240MB using bs: 1048576, readers: 2, took 160 seconds @ 64 MB/s

==== Load Balanced (zfslinux) ====
Read 10240MB using bs: 1048576, readers: 2, took 207 seconds @ 49 MB/s

=== Prefetch Enabled ===
==== Load Balanced (locality) ====
Read 10240MB using bs: 1048576, readers: 2, took 85 seconds @ 120 MB/s

==== Stripe Balanced (default) ====
Read 10240MB using bs: 1048576, readers: 2, took 109 seconds @ 93 MB/s

==== Load Balanced (zfslinux) ====
Read 10240MB using bs: 1048576, readers: 2, took 94 seconds @ 108 MB/s

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zfs-mirror-load.patch
Type: application/octet-stream
Size: 9445 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/zfs-devel/attachments/20130814/8da14737/attachment.obj>