12-CURRENT: ZFS single stream/large files read performance degradation

[Alex Tutubalin]

Hi,

I'm using freebsd/zfs box to store/access media files (mostly RAW photos 
/thousands/ and some video), so it is 'single user NAS' (connected via 
10G to my workstation).

This box has upgraded to 12-current several weeks ago, it was also 
updated with new disks and after that I see serious read speed 
performance degradation.

This hurts me so much, that I've run tests with separate set of HDDs 
yesterday, using FreeBSD 10.1, 10.2, 10.3, 11.0, 12-20161130 booted from 
memstick (so, default settings, no adjustments).

In details:

  Box configuration:  Intel i5-2400 (3.1Ghz), 16GB RAM, LSI-9211-8i 
controller in dumb IT-mode. Disks: WD1003FBYX (6-drive and 7-drive 
RAIDZ2 were tested)

Testing methodology:

boot from memstick
zpool create -O recordsize=nn ztest raidz2 /dev/gpt/DISK[0-5/6] (for 6 
and 7 disk RAIDZ2)
zfs create ztest/pool
dd if=/dev/zero of=/ztest/pool/100gfile bs=1m count=100k
dd if=/ztest/pool/100gfile of=/dev/null bs=1m

  And write numbers (first free digits :) into excel table.

I've tested 5 versions of FreeBSD (all memstick images are from 
ftp.freebsd.org), two recordsizes (128k and 1m) and two RAIDZ2 disc 
count: 6 and 7.

Results are here in table: 
http://blog.lexa.ru/2016/12/06/freebsd_zfs_read_speed_oni_ubili_kenni.html
My blog post is in Russian, but the table has english captions, so 
speaks for itself.

In short, for 6-drive array and 11.0 against 12-20163011

128k recordsize:

    11.0  503Mb/s(read)/462 Mb/s(write)

     12-Current: 380/189

1m recordsize:

    11.0: 507/521

    12-current: 515/203.

It is possible to get single-stream read speed back in 12-Current by 
setting vfs.zfs.zfetch.max_distance to very high number (1 gigabyte or 
even more, there is significant difference between 800m and 1800m), but 
this is not acceptable solution because any small read will become very 
large.

For now, I'm downgrading back to 11.0, but it looks like I need to 
report this degradation to whom it may concern :)

-- 
Alex Tutubalin
Web: http://blog.lexa.ru
mailto:lexa at lexa.ru