[EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs

[Caza, Aaron]

>
> From: Freddie Cash [mailto:fjwcash at gmail.com]
> Sent: Tuesday, June 20, 2017 1:06 PM
> To: Caza, Aaron
> Cc: Karl Denninger; freebsd-fs at freebsd.org
> Subject: [EXTERNAL] Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs
>
> On Tue, Jun 20, 2017 at 11:50 AM, Caza, Aaron <Aaron.Caza at ca.weatherford.com<mailto:Aaron.Caza at ca.weatherford.com>> wrote:
I've observed this performance degradation on 6 different hardware systems using 4 differents SSDS (2x Intel 510 120GB, 2x Intel 520 120GB, 2x Intel 540 120GB, 2x Samsung 850 Pro SSDs) on FreeBSD10.3 RELEASE, FreeBSD 10.3 RELEASEp6, FreeBSD 10.3RELEASEp19, FreeBSD 10-Stable, FreeBSD11.0 RELEASE, FreeBSD 11-Stable and now FreeBSD11.1 Beta 2.  This latest testing I'm not doing much in the way of writing - only logging the output of the 'dd' command along with 'zfs-stats -a' and 'uptime' to go along with it once an hour.   Ran for ~20hrs before performance drop kicked in though why it happens is inexplicable as this server isn't doing anything other than running this test hourly.

I have a FreeBSD9.0 system using 2x Intel 520 120GB SSDs that doesn't exhibit this performance degradation, maintaining ~400MB/s speeds even after many days of uptime.  This is using the GEOM ELI layer to provide 4k sector emulation for the mirrored zpool as I previously described.
>
​> I don't remember if this has been mentioned yet in either of your threads on this, but what is the output of this command on all your poorly performing systems:
>
> sysctl ​vfs.zfs.trim.enabled
>
> If it's set to 1 (the default), set it to 0 and re-run your tests.
>
> ZFS Trim support for SSDs was added to 10.0, so any system running FreeBSD 10+ will show a performance drop after awhile when the trim function kicks in to clear out deleted/unused blocks.  Especially if it's > an SSD that can't run Trim commands in parallel.
>
> You can look at the various ZFS trim-related stats to see what it's doing:
>
> sysctl vfs.zfs | grep trim
>
> --
> Freddie Cash
> fjwcash at gmail.com<mailto:fjwcash at gmail.com>

When using the GEOM ELI layer to provide the 4k sector emulation for the mirrored zpool, I tried disabling Trim; however, it had no effect.  I’ve now modified my FreeBSD 11.1 Beta 2 test server to remove the vfs.zfs.arc_min and vfs.zfs.arc_max settings so that the system uses the defaults as well as set vfs.zfs.trim.enabled=”0”.  Initial results are more in line with what I was seeing using GEOM ELI layer to provide 4k sector emulation:

test at f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m
16000+0 records in
16000+0 records out
16777216000 bytes transferred in 19.631485 secs (854607567 bytes/sec)

Perhaps the forced vfs.zfs.arc_min and vfs.zfs.arc_max of 50MB, while fine for FreeBSD 9.0 (which never supported Trim), need more liberal settings for FreeBSD 10 and later?  I have a number of systems with only 2GB ram running PostgreSQL along with a number of other daemons which is why I chose such low ARC settings in the first place.

Still, it seems odd that the system runs fine for several hours before performance degrades.  With the original GEOM ELI layering I utilized, implemented before ashift was available and retained, Trim was not an issue as it was never passed to the drives.

This test will probably take a while to complete.
This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. PLEASE NOTE that all incoming e-mails sent to Weatherford e-mail accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional e-mails (spam). This process could result in deletion of a legitimate e-mail before it is read by its intended recipient at our organization. Moreover, based on the scanning results, the full text of e-mails and attachments may be made available to Weatherford security and other personnel for review and appropriate action. If you have any concerns about this process, please contact us at dataprivacy at weatherford.com.