N-way mirror read speedup in zfsonlinux

Tue Aug 6 15:37:48 UTC 2013

>----- Original Message ----- 
>From: "Alexander Motin" <mav at FreeBSD.org>
>On 04.08.2013 21:22, Alexander Motin wrote:
>> On 04.08.2013 21:18, Steven Hartland wrote:
>>> Interesting stuff.
>>>
>>> I created a little test scenario here today to run this through its
>>> passes.
>>>
>>> Its very basic, running 10 x dd's from 5 * 5GB tests to /dev/null on a
>>> pool made up of a 4 SSD's and 1 HDD in a mirror:
>>>
>>>   pool: tpool
>>> state: ONLINE
>>>   scan: resilvered 38.5K in 0h0m with 0 errors on Sun Aug  4 18:13:59
>>> 2013
>>> config:
>>>
>>>         NAME        STATE     READ WRITE CKSUM
>>>         tpool       ONLINE       0     0     0
>>>           mirror-0  ONLINE       0     0     0
>>>             ada2    ONLINE       0     0     0
>>>             ada3    ONLINE       0     0     0
>>>             ada4    ONLINE       0     0     0
>>>             ada5    ONLINE       0     0     0
>>>             ada1    ONLINE       0     0     0
>>>
>>> The results are quite telling:-
>>>
>>> == Without Patch ==
>>> === SSDs & HD ===
>>> Read of 51200MB using bs 1048576 took 51 seconds @ 1003 MB/s
>>> Read of 51200 MB using bs 4096 took 51 seconds @ 1003 MB/s
>>> Read of 51200 MB using bs 512 took 191 seconds @ 268 MB/s
>>>
>>> === SSDs Only ===
>>> Read of 51200MB using bs 1048576 took 40 seconds @ 1280 MB/s
>>> Read of 51200MB using bs 4096 took 41 seconds @ 1248 MB/s
>>> Read of 51200MB using bs 512 took 188 seconds @ 272 MB/s
>>>
>>> == With Patch ==
>>> === SSDs & HD ===
>>> Read of 51200MB using bs 1048576 took 32 seconds @ 1600 MB/s
>>> Read of 51200MB using bs 4096 took 31 seconds @ 1651 MB/s
>>> Read of 51200MB using bs 512 took 184 seconds @ 278 MB/s
>>>
>>> === SSDs Only ===
>>> Read of 51200MB using bs 1048576 took 28 seconds @ 1828 MB/s
>>> Read of 51200MB using bs 4096 took 29 seconds @ 1765 MB/s
>>> Read of 51200MB using bs 512 took 185 seconds @ 276 MB/s
>>>
>>> Even with only the SSD's the patched version performs
>>> noticeably better. I suspect this is down to the fact
>>> the SSD's are various makes so have slightly different IO
>>> characteristics.
>>>
>>> N.B. The bs 512 tests can be mostly discounted as it was CPU
>>> limited in dd on the 8 core test machine.
>>
>> Could you also run test with HDDs only and with different (lower) number
>> of dd's? SSDs are much more forgiving due to lack of seek time.
>
> I couldn't wait and did it myself with 4xHDDs in mirror:
> Without patch:
>  1xdd 360MB/s
>  2xdd 434MB/s
>  4xdd 448MB/s
>
> With patch:
>  1xdd 167MB/s
>  2xdd 310MB/s
>  4xdd 455MB/s
>
> So yes, while it helps with multi-threaded read, sequential low-threaded 
> read is heavily harmed. I would not call it a win.
>

Hmmm, my results for two HDD's disagree with your results:-

== 2 disks - Without Patch (max_pending=10) ==
Read of 5120MB using bs: 1048576, readers: 1, took 34 seconds @ 150 MB/s
Read of 10240MB using bs: 1048576, readers: 2, took 71 seconds @ 144 MB/s
Read of 25600MB using bs: 1048576, readers: 5, took 188 seconds @ 136 MB/s
Read of 51200MB using bs: 1048576, readers: 10, took 188 seconds @ 272 MB/s

== 2 disks - With Patch (max_pending=10) ==
Read of 5120MB using bs: 1048576, readers: 1, took 24 seconds @ 213 MB/s
Read of 10240MB using bs: 1048576, readers: 2, took 59 seconds @ 173 MB/s
Read of 25600MB using bs: 1048576, readers: 5, took 114 seconds @ 224 MB/s
Read of 51200MB using bs: 1048576, readers: 10, took 168 seconds @ 304 MB/s

== 2 disks - Without Patch (max_pending=100) ==
Read of 5120MB using bs: 1048576, readers: 1, took 37 seconds @ 138 MB/s
Read of 10240MB using bs: 1048576, readers: 2, took 80 seconds @ 128 MB/s
Read of 25600MB using bs: 1048576, readers: 5, took 206 seconds @ 124 MB/s
Read of 51200MB using bs: 1048576, readers: 10, took 208 seconds @ 246 MB/s

== 2 disks - With Patch (max_pending=100) ==
Read of 5120MB using bs: 1048576, readers: 1, took 24 seconds @ 213 MB/s
Read of 10240MB using bs: 1048576, readers: 2, took 49 seconds @ 208 MB/s
Read of 25600MB using bs: 1048576, readers: 5, took 113 seconds @ 226 MB/s
Read of 51200MB using bs: 1048576, readers: 10, took 116 seconds @ 441 MB/s

Visually watching gstat output shows the first disk ada1 @ 50% busy and the 
second disk (ada5) @ 100% most of the time without the patch, where as with
they both take an even amount of the load.

Note: I believe any values over ~240MB/s are due to ARC replay.

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.