Re: {* 05.00 *}Re: Desperate with 870 QVO and ZFS

Am 06.04.22 um 23:19 schrieb Jan Bramkamp:
> On 06.04.22 22:43, mike tancsa wrote:
>> On 4/6/2022 4:18 PM, Bob Friesenhahn wrote:
>>> On Wed, 6 Apr 2022, egoitz@ramattack.net wrote:
>>>>>
>>>>> WE DON'T USE COMPRESSION AS IT'S NOT SET BY DEFAULT. SOME PEOPLE SAY YOU
>>>>> SHOULD HAVE IT ENABLED.... BUT.... JUST FOR AVOID HAVING SOME DATA
>>>>> COMPRESSED SOME OTHER NOT (IN CASE YOU ENABLE AND LATER DISABLE) AND
>>>>> FINALLY FOR AVOID ACCESSING TO INFORMATION WITH DIFFERENT CPU COSTS OF
>>>>> HANDLING... WE HAVE NOT TOUCHED COMPRESSION....
>>>
>>> There seems to be a problem with your caps-lock key.
>>>
>>> Since it seems that you said that you are using maildir for your mail
>>> server, it is likely very useful if you do enable even rather mild
>>> compression (e.g. lz4) since this will reduce the write work-load and even
>>> short files will be stored more efficiently.
>>>
>> FYI, a couple of our big zfs  mailspools sees a 1.24x and 1.23x compress
>> ratio with lz4.  We use Maildir format as well.  They are not RELENG_13 so
>> not sure how zstd would fair.
> I've found that Dovecot's mdbox format compresses a lot better than Maildir (or
> sdbox), because it stores multiple messages per file resulting in files large
> enough to contain enough exploitable reduncancy to compress down to the next
> smaller blocksize. In a corporate or education environment where users tend to
> send the same medium to large attachments multiple times to multiple recipients
> on the same server Dovecot's single instance storage is a game changer. It
> reduced my IMAP storage requirements by a *factor* of 4.7 which allowed me to
> get rid of spinning disks for the mail servers instead of playing losing games
> with hybrid storage. Dovecot also supports zlib compression in the application
> instead of punting it to the file system. I don't know if Cyrus IMAP offers
> similar features, but if it does I would recommend evaluating them instead of
> compressing or deduplicating at the file system level.

I have not compared dovecot's zlib compression with zstd-2 on the file system,
but since I use the latter on all my ZFS file systems (excepts those that
exclusively hold compressed files and media), I'm using it for Dovecot mdbox
files, too. I get a compression ratio of 2,29 with ZFS zstd-2, maybe I should
copy the files over into a zlib compressed mdbox for comparison ...

One large advantage of the mdbox format in the context of the mail server
set-up at the start of this thread is that deletions are only registered in
an index file (while mbox needs a rewrite of potentially large parts of the
mail folder and mdir immediately deletes files (TRIM) and updates inodes and
directory entries, causing multiple writes per deleted message).

With mdbox you can delay all "expensive" file system operations to the
point of least load each day, for example. Such a compression run is also
well suited for SSDs, since it does not perform random updates that punch
holes in a large number of erase blocks (which then will need to be garbage
collected, causing write amplification to put further load and stress on
the SSD).