12-CURRENT: ZFS single stream/large files read performance degradation

[Konstantin Belousov]

On Tue, Dec 06, 2016 at 10:52:12AM +0300, Alex Tutubalin wrote:
> Hi,
> 
> I'm using freebsd/zfs box to store/access media files (mostly RAW photos 
> /thousands/ and some video), so it is 'single user NAS' (connected via 
> 10G to my workstation).
> 
> This box has upgraded to 12-current several weeks ago, it was also 
> updated with new disks and after that I see serious read speed 
> performance degradation.
> 
> This hurts me so much, that I've run tests with separate set of HDDs 
> yesterday, using FreeBSD 10.1, 10.2, 10.3, 11.0, 12-20161130 booted from 
> memstick (so, default settings, no adjustments).
> 
> In details:
> 
>   Box configuration:  Intel i5-2400 (3.1Ghz), 16GB RAM, LSI-9211-8i 
> controller in dumb IT-mode. Disks: WD1003FBYX (6-drive and 7-drive 
> RAIDZ2 were tested)
> 
> Testing methodology:
> 
> boot from memstick
> zpool create -O recordsize=nn ztest raidz2 /dev/gpt/DISK[0-5/6] (for 6 
> and 7 disk RAIDZ2)
> zfs create ztest/pool
> dd if=/dev/zero of=/ztest/pool/100gfile bs=1m count=100k
> dd if=/ztest/pool/100gfile of=/dev/null bs=1m
> 
>   And write numbers (first free digits :) into excel table.
> 
> I've tested 5 versions of FreeBSD (all memstick images are from 
> ftp.freebsd.org), two recordsizes (128k and 1m) and two RAIDZ2 disc 
> count: 6 and 7.
> 
> Results are here in table: 
> http://blog.lexa.ru/2016/12/06/freebsd_zfs_read_speed_oni_ubili_kenni.html
> My blog post is in Russian, but the table has english captions, so 
> speaks for itself.
> 
> In short, for 6-drive array and 11.0 against 12-20163011
> 
> 128k recordsize:
> 
>     11.0  503Mb/s(read)/462 Mb/s(write)
> 
>      12-Current: 380/189
> 
> 1m recordsize:
> 
>     11.0: 507/521
> 
>     12-current: 515/203.
> 
> It is possible to get single-stream read speed back in 12-Current by 
> setting vfs.zfs.zfetch.max_distance to very high number (1 gigabyte or 
> even more, there is significant difference between 800m and 1800m), but 
> this is not acceptable solution because any small read will become very 
> large.
> 
> For now, I'm downgrading back to 11.0, but it looks like I need to 
> report this degradation to whom it may concern :)

Are you using memstick image from the download site ?
Note that all HEAD images produced by re@ have full debugging enabled,
among which both kernel-side INVARIANTS and WITNESS, and usermode malloc
debugging, have potentially huge impact on the system performance.