N-way mirror read speedup in zfsonlinux

[Steven Hartland]

Interesting stuff.

I created a little test scenario here today to run this through its passes.

Its very basic, running 10 x dd's from 5 * 5GB tests to /dev/null on a
pool made up of a 4 SSD's and 1 HDD in a mirror:

  pool: tpool
 state: ONLINE
  scan: resilvered 38.5K in 0h0m with 0 errors on Sun Aug  4 18:13:59 2013
config:

        NAME        STATE     READ WRITE CKSUM
        tpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada5    ONLINE       0     0     0
            ada1    ONLINE       0     0     0

The results are quite telling:-

== Without Patch ==
=== SSDs & HD ===
Read of 51200MB using bs 1048576 took 51 seconds @ 1003 MB/s
Read of 51200 MB using bs 4096 took 51 seconds @ 1003 MB/s
Read of 51200 MB using bs 512 took 191 seconds @ 268 MB/s

=== SSDs Only ===
Read of 51200MB using bs 1048576 took 40 seconds @ 1280 MB/s
Read of 51200MB using bs 4096 took 41 seconds @ 1248 MB/s
Read of 51200MB using bs 512 took 188 seconds @ 272 MB/s

== With Patch ==
=== SSDs & HD ===
Read of 51200MB using bs 1048576 took 32 seconds @ 1600 MB/s
Read of 51200MB using bs 4096 took 31 seconds @ 1651 MB/s
Read of 51200MB using bs 512 took 184 seconds @ 278 MB/s

=== SSDs Only ===
Read of 51200MB using bs 1048576 took 28 seconds @ 1828 MB/s
Read of 51200MB using bs 4096 took 29 seconds @ 1765 MB/s
Read of 51200MB using bs 512 took 185 seconds @ 276 MB/s

Even with only the SSD's the patched version performs
noticeably better. I suspect this is down to the fact
the SSD's are various makes so have slightly different IO
characteristics.

N.B. The bs 512 tests can be mostly discounted as it was CPU
limited in dd on the 8 core test machine.

    Regards
    Steve
----- Original Message ----- 
From: "Martin Matuska" <mm at FreeBSD.org>
To: <zfs-devel at freebsd.org>
Cc: "Xin Li" <delphij at FreeBSD.org>; "Steven Hartland" <smh at FreeBSD.org>
Sent: Sunday, August 04, 2013 10:25 AM
Subject: Re: N-way mirror read speedup in zfsonlinux


> Attached is a FreeBSD version of this patch for testing and comments,
> including sysctl tunable:
> http://people.freebsd.org/~mm/patches/zfs/vdev_mirror.c.patch
>
> On 2013-07-12 11:21, Martin Matuška wrote:
>> Hi everyone,
>>
>> zfsonlinux has implemented a change in the N-way mirror device selection
>> algorithm by selecting the device with the least pending I/O instead of
>> random selection. They measured an increased read bandwidth increase
>> up to
>> 50% and IOPS increase up to 10%.
>>
>> this might be useful for common ZFS code and we might consider porting
>> this
>> to illumos and FreeBSD:
>> https://github.com/zfsonlinux/zfs/issues/1461
>> https://github.com/zfsonlinux/zfs/commit/556011dbec2d10579819078559a77630fc559112
>>
>
> 


================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.