N-way mirror read speedup in zfsonlinux

[Alexander Motin]

Hi.

I like the idea of load-aware balancing. I went that way in both gmirror 
and graid and seen substantial performance improvements. But pure 
load-based balance may be ineffective in case of spinning HDDs and 
sequential I/Os. Doing that way will mitigate effect of the device 
read-ahead. If sequential read operations will be handled on triple 
mirror as ABCABC, we may get situation when all three devices are busy, 
but giving only read speed of one. vdev_mirror_shift used in the 
original balance code has its reason -- it will make all I/Os to the 
consequential offsets (within 2MB) go to the same disk. You may find 
G_RAID_SUBDISK_TRACK_SIZE constant in graid and TRACK_SIZE in gmirror 
doing exactly the same.

I am not have doubt in effectiveness of proposed time-based magic. If 
there is only one request issued at a time, this algorithm indeed may 
give benefit by more effective using of the device read-ahead. But if we 
have more then one request active, the result will become purely random 
since first request sent will affect later ones and (d % lowest_nr) will 
give precedence to completely random device.

Also I would like to note that getting precise time still can be 
expensive operation on some hardware to do it on every I/O. I think in 
many cases there will be only one device with lowest load, and reading 
of time could be skipped.

On 04.08.2013 12:25, Martin Matuska wrote:
> Attached is a FreeBSD version of this patch for testing and comments,
> including sysctl tunable:
> http://people.freebsd.org/~mm/patches/zfs/vdev_mirror.c.patch
>
> On 2013-07-12 11:21, Martin Matuška wrote:
>> zfsonlinux has implemented a change in the N-way mirror device selection
>> algorithm by selecting the device with the least pending I/O instead of
>> random selection. They measured an increased read bandwidth increase
>> up to
>> 50% and IOPS increase up to 10%.
>>
>> this might be useful for common ZFS code and we might consider porting
>> this
>> to illumos and FreeBSD:
>> https://github.com/zfsonlinux/zfs/issues/1461
>> https://github.com/zfsonlinux/zfs/commit/556011dbec2d10579819078559a77630fc559112

-- 
Alexander Motin