ZSTD Project Weekly Status Update

[Allan Jude]

In my continuing effort to complete the integration of ZSTD into
OpenZFS, here is my fourth weekly status report:

https://github.com/allanjude/zfs/commit/b0b1270d4e7835ecff413208301375e3de2a4153
- Create a new test case to make sure that the ZSTD header we write
along with the data is correct. Verify that the physical size of the
compressed data is less than the psize for the block pointer, and verify
that the level matches. It uses a random level between 1 and 19 and then
verifies with zdb that the block was compressed with that level.

I am still working on a solution for setting the zstd feature flag to
'active' as soon as it is set, rather than only once a block is born. As
well as fixing up compatibility around zfs send/recv with the embedded
block points flag.

This project is sponsored by the FreeBSD Foundation.


On 2020-07-06 20:07, Allan Jude wrote:
> In my continuing effort to complete the integration of ZSTD into
> OpenZFS, here is my third weekly status report:
> 
> https://github.com/allanjude/zfs/commit/87bb538bdbc4bb8848ed5791b7b0de84a026ebbe
> - Completed the rewrite of the way the compression property is handled,
> moving away from the initial approach of storing the compression
> property (enum zio_compress) and the level (uint64_t) separately.
> 
> Previously we exposed the list of compression algorithms and levels to
> userland as the corresponding value from the enum in the lower 7 bits,
> and the level in the remaining upper bits. Then, as part of the property
> GET and SET IOCTLs, we read the separate compression= and
> compress_level= properties from the ZAP and returned the combined value,
> or split the combined value into those two separate properties. This
> worked but caused a lot of headache around property inheritance.
> 
> Instead I've changed to doing the combine/split when reading/writing
> from the dataset properties ZAP, via the compression_changed_cb()
> function. So the properties ZAP contains the combined value (lower 7
> bits are the compression algorithm, as defined in the enum zio_compress,
> and the upper bits are the compression level). Elsewhere in ZFS we keep
> the two values separate (os_compress and os_complevel, and related
> variables in all of the different parts of ZFS).
> 
> So now, inheritance of the property is handled correctly, and avoids
> issues where a dataset with compression=zstd-12, would say 'inherited
> from' a dataset with zstd at some other compression level (since both
> actually just had compression=zstd, but different compress_level= values).
> 
> 
> I have also further extended zdb to inspect the compression settings
> when looking at an object:
> https://github.com/allanjude/zfs/commit/3fef3c83b8ce90149110ed989bd9fd3e289798e0
> 
> 
> I am still working on a solution for setting the zstd feature flag to
> 'active' as soon as it is set, rather than only once a block is born.
> 
> 
> Additionally, I am investigating how to best handle the fact that
> embedded block pointers compressed with ZSTD will make 'zfs send -e'
> streams backwards incompatible, without a way for the user to opt-out of
> sending a stream that contains zstd compressed blocks that the receiving
> side may not be able to read. The same can be said for 'zfs send -c' as
> well. I am open to ideas on how best to handle this. I have thought
> about only sending ZSTD compressed blocks if the user specifies the -Z
> (--zstd) flag, but this can lead to confusion where using -c without -Z
> would have to either error out, or send the ZSTD compressed blocks
> uncompressed.
> 
> 
> This project is sponsored by the FreeBSD Foundation.
> 


-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20200714/eeaa590d/attachment.sig>