12-CURRENT AND 11-stable: ZFS single stream/large files read performance degradation

[Alex Tutubalin]

On 12/6/2016 10:52 AM, Alex Tutubalin wrote:

> This box has upgraded to 12-current several weeks ago, it was also 
> updated with new disks and after that I see serious read speed 
> performance degradation.
It is looks even worse, 11-stable is affected too:

- Another box, different hardware:  i3-6300T, 32Gb ram, C236 Chipset, 6 
x 6Tb HGST drives (3x7k6000 and 3xHe8) connected to motherboard SATA ports.
The only zfs related tunables are about ARC size:
vm.kmem_size="30G"
vfs.zfs.arc_max="29G"

11-STABLE:
FreeBSD home-gw.lexa.ru 11.0-STABLE FreeBSD 11.0-STABLE #1 r308516: Fri 
Nov 11 13:15:21 MSK 2016
lexa at home-gw.lexa.ru:/usr/obj/usr/src/sys/GENERIC  amd64

Read speed of 100Gb file:
lexa at home-gw:~# dd if=/ztest/100g of=/dev/null bs=1m
102400+0 records in
102400+0 records out
107374182400 bytes transferred in 640.324508 secs (167687135 bytes/sec)

(zfs recordsize is 1m for this dataset)

(Note, this is about single disk speed, He8 drives are capable to get 
175Mb/sec at faster tracks, 7k600 are about 195Mb/s).

Downgraded to 11-RELENG:
FreeBSD home-gw.lexa.ru 11.0-RELEASE-p3 FreeBSD 11.0-RELEASE-p3 #2 
r309605: Tue Dec  6 12:31:16 MSK 2016 
lexa at home-gw.lexa.ru:/usr/obj/usr/src/sys/GENERIC  amd64

read same file: same shit, about 160Mb/s.

Re-created the file using downgdaded ZFS code:

lexa at home-gw:/home/lexa# dd if=/dev/zero of=/ztest/100gb bs=1m count=100k
102400+0 records in
102400+0 records out
107374182400 bytes transferred in 170.845743 secs (628486146 bytes/sec)

Rebooted to get cold read and run read test:

dd if=/ztest/100gb of=/dev/null bs=1m
102400+0 records in
102400+0 records out
107374182400 bytes transferred in 168.924032 secs (635635919 bytes/sec)


It is still not perfect, because only 4 disks are used in read
iostat -x 5:
device     r/s   w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen %b
ada1         0     0      0.0      0.0     0     0     0     0 0   0
ada2      1229     0 157362.7      0.0     1     0     0     1    0 44
ada3      1231     0 157373.1      0.0     0     0     0     0    0 38
ada4      1229     0 157337.1      0.0     4     0     0     4    5 98
ada5      1231     0 157372.3      0.0     1     0     0     1    0 45
ada6         0     0      0.8      0.0    27     0     0    27 0   1

ada1 and ada6 are idle, while it is possible to read some data from it.

But this is MUCH MUCH better, than on 11-STABLE and 12-CURRENT

-- 
Alex Tutubalin
Web: http://blog.lexa.ru
mailto:lexa at lexa.ru