N-way mirror read speedup in zfsonlinux

[Alexander Motin]

On 04.08.2013 21:18, Steven Hartland wrote:
> Interesting stuff.
>
> I created a little test scenario here today to run this through its passes.
>
> Its very basic, running 10 x dd's from 5 * 5GB tests to /dev/null on a
> pool made up of a 4 SSD's and 1 HDD in a mirror:
>
>   pool: tpool
> state: ONLINE
>   scan: resilvered 38.5K in 0h0m with 0 errors on Sun Aug  4 18:13:59 2013
> config:
>
>         NAME        STATE     READ WRITE CKSUM
>         tpool       ONLINE       0     0     0
>           mirror-0  ONLINE       0     0     0
>             ada2    ONLINE       0     0     0
>             ada3    ONLINE       0     0     0
>             ada4    ONLINE       0     0     0
>             ada5    ONLINE       0     0     0
>             ada1    ONLINE       0     0     0
>
> The results are quite telling:-
>
> == Without Patch ==
> === SSDs & HD ===
> Read of 51200MB using bs 1048576 took 51 seconds @ 1003 MB/s
> Read of 51200 MB using bs 4096 took 51 seconds @ 1003 MB/s
> Read of 51200 MB using bs 512 took 191 seconds @ 268 MB/s
>
> === SSDs Only ===
> Read of 51200MB using bs 1048576 took 40 seconds @ 1280 MB/s
> Read of 51200MB using bs 4096 took 41 seconds @ 1248 MB/s
> Read of 51200MB using bs 512 took 188 seconds @ 272 MB/s
>
> == With Patch ==
> === SSDs & HD ===
> Read of 51200MB using bs 1048576 took 32 seconds @ 1600 MB/s
> Read of 51200MB using bs 4096 took 31 seconds @ 1651 MB/s
> Read of 51200MB using bs 512 took 184 seconds @ 278 MB/s
>
> === SSDs Only ===
> Read of 51200MB using bs 1048576 took 28 seconds @ 1828 MB/s
> Read of 51200MB using bs 4096 took 29 seconds @ 1765 MB/s
> Read of 51200MB using bs 512 took 185 seconds @ 276 MB/s
>
> Even with only the SSD's the patched version performs
> noticeably better. I suspect this is down to the fact
> the SSD's are various makes so have slightly different IO
> characteristics.
>
> N.B. The bs 512 tests can be mostly discounted as it was CPU
> limited in dd on the 8 core test machine.

Could you also run test with HDDs only and with different (lower) number 
of dd's? SSDs are much more forgiving due to lack of seek time.

> ----- Original Message ----- From: "Martin Matuska" <mm at FreeBSD.org>
> To: <zfs-devel at freebsd.org>
> Cc: "Xin Li" <delphij at FreeBSD.org>; "Steven Hartland" <smh at FreeBSD.org>
> Sent: Sunday, August 04, 2013 10:25 AM
> Subject: Re: N-way mirror read speedup in zfsonlinux
>
>
>> Attached is a FreeBSD version of this patch for testing and comments,
>> including sysctl tunable:
>> http://people.freebsd.org/~mm/patches/zfs/vdev_mirror.c.patch
>>
>> On 2013-07-12 11:21, Martin Matuška wrote:
>>> Hi everyone,
>>>
>>> zfsonlinux has implemented a change in the N-way mirror device selection
>>> algorithm by selecting the device with the least pending I/O instead of
>>> random selection. They measured an increased read bandwidth increase
>>> up to
>>> 50% and IOPS increase up to 10%.
>>>
>>> this might be useful for common ZFS code and we might consider porting
>>> this
>>> to illumos and FreeBSD:
>>> https://github.com/zfsonlinux/zfs/issues/1461
>>> https://github.com/zfsonlinux/zfs/commit/556011dbec2d10579819078559a77630fc559112


-- 
Alexander Motin