ZSTD Project Weekly Status Update

[Allan Jude]

This is the tenth weekly status report on my FreeBSD Foundation
sponsored project to complete the integration of ZSTD compression into
OpenZFS.

Late last week the main pull request was merged, and ZSTD support is now
part of OpenZFS's trunk branch.

Last night, OpenZFS with ZSTD was imported into FreeBSD's -current branch.

I am continuing to work on a number of things related to ZSTD, including
future-proofing support (so upgrading ZSTD won't cause problems with
features like nopwrite), and improving the integration of ZSTD into
FreeBSD, including enabling support for booting from ZSTD compressed
datasets, and improving the performance of ZSTD on FreeBSD.

I'll also be adding some additional tests to make sure we detect any
issues when we do look at updating ZSTD. Additionally, I am working on a
bunch of documentation around using ZSTD in ZFS.

For my benchmarking of ZSTD, I have been using a zfs recv of a stream in
a file on a tmpfs, and recording how long it takes to receive and sync
the data. The test data is a copy of the FreeBSD 12.1 source code, since
that is easily reproducible.

Does anyone have experience or a better suggestion on how to get the
most consistent and repeatable results when benchmarking like this?


On 2020-08-18 18:51, Allan Jude wrote:
> This is the ninth weekly status report on my FreeBSD Foundation
> sponsored project to complete the integration of ZSTD compression into
> OpenZFS.
> 
> https://github.com/openzfs/zfs/pull/10693 - The L2ARC fixes (for when
> compressed ARC is disabled) have been merged.
> 
> https://github.com/openzfs/zfs/pull/10278/ - A number of other cleanups
> and fixes for the ZSTD have been integrated and squashed, and it looks
> like the completed ZSTD feature will be merged very soon.
> 
> This included a bunch of fixes for makefiles and runfiles to hook the
> tests I added up to the ZFS test suite so they are run properly.
> 
> It looks like this will mean that the ZSTD feature will be included in
> OpenZFS 2.0. Thanks for everyone who has tested, reviewed, or
> contributed to this effort, especially those who kept it alive while I
> was working on other things.
> 
> Post-merge, the remaining work is to develop future-proofing around ZSTD
> so that we will be able to more seamlessly upgrade to newer releases of
> ZSTD. Recompression of the same input resulting in the same on-disk
> checksum is the main concern, as without this upgrading the compression
> algorithm will break features like nop-write.
> 
> This project is sponsored by the FreeBSD Foundation.
> 


-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20200825/01e99fab/attachment.sig>