ZSTD Project Weekly Status Update

[Allan Jude]

This is another weekly status report on my FreeBSD Foundation sponsored
project to complete the integration of ZSTD compression into OpenZFS.

I have continued to work on the benchmarks, including attempting to
compare compression and performance across various settings with pgbench
(PostgreSQL). However the results are not as meaningful as I would like
so far.

I am still trying some other configurations to try to get clearer
results. Using multiple md(4) devices has helped avoid that being the
bottleneck.

https://docs.google.com/spreadsheets/d/1TvCAIDzFsjuLuea7124q-1UtMd0C9amTgnXm2yPtiUQ/edit?usp=sharing

Additionally, I completed rewriting my earlier commit to convert the
FreeBSD versions of zstd to use FreeBSD's UMA kernel memory allocator,
rather than the custom mempool used on Linux due to its kmem_cache
abstraction not being able to handle large allocations:

https://github.com/openzfs/zfs/pull/10975

I am working on some benchmarks of this change as well.

Thank you.

On 2020-09-15 22:11, Allan Jude wrote:
> This is another weekly status report on my FreeBSD Foundation sponsored
> project to complete the integration of ZSTD compression into OpenZFS.
> 
> The first batch of benchmarks are complete, although they took longer
> than expected to get good data.
> 
> I am still not entirely pleased with the data, as in some cases I am
> running up against limitations of my device-under-test rather than the
> performance limits of ZFS.
> 
> Here is what I have so far:
> 
> https://docs.google.com/spreadsheets/d/1TvCAIDzFsjuLuea7124q-1UtMd0C9amTgnXm2yPtiUQ/edit?usp=sharing
> 
> A number of these tests were initially done on both FreeBSD and Linux on
> the same machine, and the results were consistent within a 2% margin of
> error, so I've taken to doing most of the tests only on FreeBSD, because
> it is easier. I've struggled to get a good ramdisk solution on Ubuntu etc.
> 
> To walk you through the different tabs in the spreadsheet so far:
> 
> #1: fio SSD
> This is a random write test to my pool made of 4 SSDs. This ran into the
> performance limitations of the SSDs when testing the very fast
> algorithms. Since the data generated by fio is completely
> uncompressible, there is no gain from the higher compression levels.
> 
> #2: fio to ramdisk
> To overcome the limitations of the first test, I did it again with a
> ramdisk. Obviously this had to be a smaller dataset, since there is
> limited memory available, but it does a much better job of showing how
> the zstd-fast levels scale, and how they outperform LZ4, although you
> cannot compare the compression, because the data is uncompressible.
> 
> #3: zfs recv to SSD
> For this test, I created a dataset by extracting the FreeBSD src.txz
> file 8 times (each to a different directory), then created a snapshot of
> that, and send it to a file on a tmpfs.
> 
> I then timed zfs recv < /tmpfs/snapshot.zfs with each compression
> algorithm. This allows you to compare the compression gain for the time
> trade-off, but again ran into the throughput limitations of the SSDs, so
> provides a bit less information about the performance of the higher
> zstd-fast levels, but you can see the compression tradeoff.
> 
> I need to reconfigure my setup to re-do this benchmark using a ramdisk.
> 
> #4: large image file 128k
> For this, i created an approximately 20GB tar file, by unxz'ding the
> FreeBSD 12.1 src.txz and concatenating it 16 times. This provides the
> best possible case for compression.
> 
> One of the major advantages of ZSTD is that the decompression throughput
> stays relatively the same even as the compression level is increased. So
> while writing a zstd-19 compressed block takes a lot longer than a
> zstd-3 compressed block, both decompress at nearly the same speed.
> 
> This time I measured fio random read performance. Putting the
> limitations of the SSDs to good use, this test shows the read
> performance gains from reading compressed. Even though the disks top out
> around 1.5 GB/sec, zstd-compressed data can be read at an effective rate
> of over 5 GB/sec.
> 
> #5: large image file 1m
> This is the same test, but done with zfs recordsize=1m
> 
> The larger record size unlocks higher compression ratios, and achieves
> throughputs in excess of 6 GB/sec.
> 
> #6: large image file 16k
> This is again the same test, but with zfs recordsize=16k
> This is an approximation of reading from a large database with a 16k
> page size.
> The lower record size provides much less compression, and the smaller
> blocks result in more overhead, but, there are still large performance
> gains to be had from the compression, although they are much less drastic.
> 
> I would be interested in what other tests people might be interested in
> seeing before I finish wearing these SSDs out.
> 
> 
> Thanks again to the FreeBSD Foundation for sponsoring this work.
> 
> 

-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20200923/bf162d34/attachment.sig>