From nobody Thu Apr 07 13:53:06 2022 X-Original-To: performance@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 240921A90AA1; Thu, 7 Apr 2022 13:53:17 +0000 (UTC) (envelope-from egoitz@ramattack.net) Received: from cu1208c.smtpx.saremail.com (cu1208c.smtpx.saremail.com [195.16.148.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KZ2s80CnXz4bfR; Thu, 7 Apr 2022 13:53:14 +0000 (UTC) (envelope-from egoitz@ramattack.net) Received: from www.saremail.com (unknown [194.30.0.183]) by sieve-smtp-backend02.sarenet.es (Postfix) with ESMTPA id 5746F60C4A4; Thu, 7 Apr 2022 15:53:06 +0200 (CEST) List-Id: Performance/tuning List-Archive: https://lists.freebsd.org/archives/freebsd-performance List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_9966c949ad7be3a0f79023829a1cf786" Date: Thu, 07 Apr 2022 15:53:06 +0200 From: egoitz@ramattack.net To: Stefan Esser Cc: Jan Bramkamp , performance@freebsd.org, freebsd-fs@freebsd.org, FreeBSD Hackers Subject: Re: {* 05.00 *}Re: Re: Desperate with 870 QVO and ZFS In-Reply-To: <49f43af5-e145-c793-959d-ab1596421d81@FreeBSD.org> References: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> <665236B1-8F61-4B0E-BD9B-7B501B8BD617@ultra-secure.de> <0ef282aee34b441f1991334e2edbcaec@ramattack.net> <28e11d7ec0ac5dbea45f9f271fc28f06@ramattack.net> <7aa95cb4bf1fd38b3fce93bc26826042@ramattack.net> <49f43af5-e145-c793-959d-ab1596421d81@FreeBSD.org> Message-ID: X-Sender: egoitz@ramattack.net User-Agent: Saremail webmail X-Rspamd-Queue-Id: 4KZ2s80CnXz4bfR X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=reject) header.from=ramattack.net; spf=pass (mx1.freebsd.org: domain of egoitz@ramattack.net designates 195.16.148.183 as permitted sender) smtp.mailfrom=egoitz@ramattack.net X-Spamd-Result: default: False [-3.79 / 15.00]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; XM_UA_NO_VERSION(0.01)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.16.148.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[ramattack.net,reject]; FROM_NO_DN(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-fs,freebsd-hackers,performance]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:3262, ipnet:195.16.128.0/19, country:ES]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --=_9966c949ad7be3a0f79023829a1cf786 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Hi Stefan, Thanks a lot again, mate. Answering below in bold blue... El 2022-04-07 09:59, Stefan Esser escribió: > I have not compared dovecot's zlib compression with zstd-2 on the file system, > but since I use the latter on all my ZFS file systems (excepts those that > exclusively hold compressed files and media), I'm using it for Dovecot mdbox > files, too. I get a compression ratio of 2,29 with ZFS zstd-2, maybe I should > copy the files over into a zlib compressed mdbox for comparison ... > > WE ARE RUNNING CYRUS HERE... ALTHOUGH THAT CHECK SOUNDS INTERESTING... > > One large advantage of the mdbox format in the context of the mail server > set-up at the start of this thread is that deletions are only registered in > an index file (while mbox needs a rewrite of potentially large parts of the > mail folder and mdir immediately deletes files (TRIM) and updates inodes and > directory entries, causing multiple writes per deleted message). > > I SEE... REALLY SAID... I LOVE CYRUS... IT'S REPLICATION IS EXTREMELY RELIABLE... > > SOME TIME NOW... DOVECOT DIDN'T HAD REPLICATION... AND WE HAVE SOME DEVELOPMENTS DONE FROM SOME TIME NOW FOR CYRUS IMAP... > > BUT GOOD TO KNOW TO ABOUT OTHER SOFTWARE'S ADVANTAGES... > > With mdbox you can delay all "expensive" file system operations to the > point of least load each day, for example. Such a compression run is also > well suited for SSDs, since it does not perform random updates that punch > holes in a large number of erase blocks (which then will need to be garbage > collected, causing write amplification to put further load and stress on > the SSD). > > WE DON'T DELETE MAIL DURING DAY HOURS. WE USE A FEATURE OF CYRUS CALLED EXPUNGE DELAYED. THE DELETED EMAIL IS DELETED FROM DISK AT 04AM (UNTIL THAT MOMENT IS JUST TAGGED AS DELETED IN A CYRUS DATABASE). THE EXCEPTION HAPPENS WHEN YOU RENAME A FOLDER OR DELETE AN ENTIRE FOLDER. IF YOU DELETE AN ENTIRE FOLDER, I THINK IT GETS COPIED TO A DELETED/..WHATEVER.. FOLDER AND THEN YES... IT COPIES AND LATER DELETES... > > APART FROM THAT, CYRUS DOES A LOT OF DATABASE CHECKPOINTING, CAUSING DATABASES TO BE COPIED TO A NEW CREATED ONE AND THE OLD ONE TO BE DELETED. THIS ARE THE ONLY REMOVALS WE DO DURING DAY TIME. THE REST IS DONE FROM 04AM TO 05AM, WHEN THERE'S NO LOAD. > > CHEERS!!! --=_9966c949ad7be3a0f79023829a1cf786 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8

Hi Stefan,


Thanks a lot again, mate. Answering below in bold blue...

 


El 2022-04-07 09:59, Stefan Esser escribió:

=
I have not compared dovecot's zlib compression with zstd-2 on the fi= le system,
but since I use the latter on all my ZFS file systems (exc= epts those that
exclusively hold compressed files and media), I'm usi= ng it for Dovecot mdbox
files, too. I get a compression ratio of 2,29= with ZFS zstd-2, maybe I should
copy the files over into a zlib comp= ressed mdbox for comparison ...
=  
= We are running Cyrus here... althou= gh that check sounds interesting...

One large = advantage of the mdbox format in the context of the mail server
set-u= p at the start of this thread is that deletions are only registered in
an index file (while mbox needs a rewrite of potentially large parts of t= he
mail folder and mdir immediately deletes files (TRIM) and updates = inodes and
directory entries, causing multiple writes per deleted mes= sage).
=  
= I see... really said... I love Cyru= s... it's replication is extremely reliable...
=  
= Some time now... Dovecot didn't had= replication... and we have some developments done from some time now for C= yrus IMAP...
=  
= But good to know to about other sof= tware's advantages...

With mdbox you can delay= all "expensive" file system operations to the
point of least load ea= ch day, for example. Such a compression run is also
well suited for S= SDs, since it does not perform random updates that punch
holes in a l= arge number of erase blocks (which then will need to be garbage
colle= cted, causing write amplification to put further load and stress on
t= he SSD).
=  
= We don't delete mail during day hou= rs. We use a feature of Cyrus called expunge delayed. The deleted email is = deleted from disk at 04am (until that moment is just tagged as deleted in a= Cyrus database). The exception happens when you rename a folder or delete = an entire folder. If you delete an entire folder, I think it gets copied to= a DELETED/..whatever.. folder and then yes... it copies and later deletes= =2E..
=  
= Apart from that, Cyrus does a lot o= f database checkpointing, causing databases to be copied to a new created o= ne and the old one to be deleted. This are the only removals we do during d= ay time. The rest is done from 04am to 05am, when there's no load.
=  
= Cheers!!!
--=_9966c949ad7be3a0f79023829a1cf786--