ZSTD Project Weekly Status Update

[Allan Jude]

This is the fifth weekly status report on the project to integrate ZSTD
into OpenZFS.

https://github.com/c0d3z3r0/zfs/pull/14/commits/9807c99169e5931a754bb0df68267ffa2f289474
- Created a new test case to ensure that ZSTD compressed blocks survive
replication with the -c flag. We wanted to make sure the on-disk
compression header survived the trip.

https://github.com/c0d3z3r0/zfs/pull/14/commits/94bef464fc304e9d6f5850391e41720c3955af11
- I split the zstd.c file into OS specific bits
(module/os/{linux,freebsd}/zstd_os.c) and also split the .h file into
zstd.h and zstd_impl.h. This was done so that FreeBSD can use the
existing kmem_cache mechanism, while Linux can use the vmem_alloc pool
created in the earlier versions of this patch. I significantly changed
the FreeBSD implementation from my earlier work, to reuse the power of 2
zio_data_buf_cache[]'s that already exist, only adding a few additional
kmem_caches for large blocks with high compression levels. This should
avoid creating as many unnecessary kmem caches.

https://github.com/c0d3z3r0/zfs/pull/14/commits/3d48243b77e6c8c3bf562c7a2315dd6cc571f28c
- Lastly, in my testing I was seeing a lot of hits on the new
compression failure kstat I added. This was caused by the ZFS "early
abort" feature, where we give the compressor an output buffer that is
smaller than the input, so it will fail if the block will not compress
enough to be worth it. This helps avoid wasting CPU on uncompressible
blocks. However, it seems the 'one-file' version of zstd we are using
does not expose the ZSTD_ErrorCode enum. This needs to be investigated
further to avoid issues if the value changes (although it is apparently
stable after version 1.3.1).

I am still working on a solution for zfs send stream compatibility. I am
leaning towards creating a new flag, --zstd, to enable ZSTD compressed
output. If the -e or -c flag are used without the --zstd flag, and the
dataset has the zstd feature active, the idea would be to emit a warning
but send the blocks uncompressed, so that the stream remains compatible
with older versions of ZFS. I will be discussing this on the OpenZFS
Leadership call tomorrow, and am open to suggestions on how to best
handle this.


On 2020-07-14 22:26, Allan Jude wrote:
> In my continuing effort to complete the integration of ZSTD into
> OpenZFS, here is my fourth weekly status report:
> 
> https://github.com/allanjude/zfs/commit/b0b1270d4e7835ecff413208301375e3de2a4153
> - Create a new test case to make sure that the ZSTD header we write
> along with the data is correct. Verify that the physical size of the
> compressed data is less than the psize for the block pointer, and verify
> that the level matches. It uses a random level between 1 and 19 and then
> verifies with zdb that the block was compressed with that level.
> 
> I am still working on a solution for setting the zstd feature flag to
> 'active' as soon as it is set, rather than only once a block is born. As
> well as fixing up compatibility around zfs send/recv with the embedded
> block points flag.
> 
> This project is sponsored by the FreeBSD Foundation.
> 
> 

-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20200720/ced3b82a/attachment.sig>