From nobody Thu Apr 07 07:59:27 2022 X-Original-To: performance@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 05C4C1A97DC7; Thu, 7 Apr 2022 07:59:35 +0000 (UTC) (envelope-from se@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KYv126R78z3KyW; Thu, 7 Apr 2022 07:59:34 +0000 (UTC) (envelope-from se@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1649318375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=m2HUVrbEq7o9ase+zDUCti07l48rJ+nU2EpdAQMDQ38=; b=NMnF19bzpHf2nHHVB1YxAjcApV7iALDxLIvW6idngan6XSg0GeYmvFdpcwm2/zvQ/+Ga4J 7ek0Evu1q19J1y6m+FT/veoZWm7xFnz6pm8EKkXfay5gVNWeCpyihq0zFetyPVKrEXXYR8 yHcEibj27FCtdbcevPD3Huzq+9UgVKon1u8uPyO8MCPaqO52ZV5LU72ERviW6LSs80ewxK eYIs9tydKw1vh2ZQY/fd8lGOv4pJXhIThInO1WiOFFXkK60CWjIuLkJ9Vqh9KQYrWVbkBx Jel3ssLOEUyAw+tVxjCASoU6//o3kZdel7elQKdv7Ei/vNs+k/mySkoWbINFMQ== Received: from [IPV6:2003:cd:5f22:6f00:34ed:cacb:3b28:daf] (p200300cd5f226f0034edcacb3b280daf.dip0.t-ipconnect.de [IPv6:2003:cd:5f22:6f00:34ed:cacb:3b28:daf]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id D6D7EA066; Thu, 7 Apr 2022 07:59:33 +0000 (UTC) (envelope-from se@FreeBSD.org) Message-ID: <49f43af5-e145-c793-959d-ab1596421d81@FreeBSD.org> Date: Thu, 7 Apr 2022 09:59:27 +0200 List-Id: Performance/tuning List-Archive: https://lists.freebsd.org/archives/freebsd-performance List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: {* 05.00 *}Re: Desperate with 870 QVO and ZFS Content-Language: en-US To: egoitz@ramattack.net References: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> <665236B1-8F61-4B0E-BD9B-7B501B8BD617@ultra-secure.de> <0ef282aee34b441f1991334e2edbcaec@ramattack.net> <28e11d7ec0ac5dbea45f9f271fc28f06@ramattack.net> <7aa95cb4bf1fd38b3fce93bc26826042@ramattack.net> From: Stefan Esser Cc: Jan Bramkamp , performance@freebsd.org, "freebsd-fs@freebsd.org" , FreeBSD Hackers In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------o4K5Sngmd3lLPnmlzLRzMtEK" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1649318374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=m2HUVrbEq7o9ase+zDUCti07l48rJ+nU2EpdAQMDQ38=; b=IWML0cH5d0UJDKTcEQqANYFGYOX7ABIcJAQ+YDeHZ4QAbaUZSpBJrCNxrZZsrvCAkwnMTY K74KzXtXJVTxHPp78gm2hLlPomWFSHeFMV3VJPRXJ+PFFuX4SA+nhQcOh28iO9JCM7hyJX PLLdH0kXGvvIq2o0ovvwQ0Liy7kQTp0BKz19FyYcMsGRzHAg7kzduV535q1Pw7JFIHoR2T IYbeGPMhAXrLbNSZyxLz6AKgw95LS8srBqWqkSKFmlr22ZpoEjXdnYCUy2EnPz/7WON5RX RxN3OqalkAJKygy71QOovZXkBhNZUej5Mwz4dAaEzOaAOJDe5HgTWrD17kqW+Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1649318374; a=rsa-sha256; cv=none; b=rApmUo+lAfX/P6o4SLQ68AGoD/Sx5BGrWCkFyCWD0/bXXVIBQWGs+Pi6shFlSIB6ZB5gIj ElK74KW/VSc62QR/K6s27Ji9I9iZy36zaE+AviLY5VURhHI0tyjl2jAF/AjsHaQQj0zqoh qNk++uhsi7umP6jnoQXI0DIXi8k98Jx48smYnExyveQfodOG60etpx2AgF/fT50iwOCtVQ bYbSc2Q5JFOBf4CRadmSUa8EMDNTVBnWlrJK6FGjN6mzMkisWg+s+N+S7z0ZK+jxMyXyAH NDvOmGPsf+5g3aWQz7Y5dNLIGe8PsGQ1Yx4n6FftSPBRddbI0b5akns9ejNw3w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------o4K5Sngmd3lLPnmlzLRzMtEK Content-Type: multipart/mixed; boundary="------------ce4Isv9Wdp6yhBkTY0YsttW0"; protected-headers="v1" From: Stefan Esser To: egoitz@ramattack.net Cc: Jan Bramkamp , performance@freebsd.org, "freebsd-fs@freebsd.org" , FreeBSD Hackers Message-ID: <49f43af5-e145-c793-959d-ab1596421d81@FreeBSD.org> Subject: Re: {* 05.00 *}Re: Desperate with 870 QVO and ZFS References: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> <665236B1-8F61-4B0E-BD9B-7B501B8BD617@ultra-secure.de> <0ef282aee34b441f1991334e2edbcaec@ramattack.net> <28e11d7ec0ac5dbea45f9f271fc28f06@ramattack.net> <7aa95cb4bf1fd38b3fce93bc26826042@ramattack.net> In-Reply-To: --------------ce4Isv9Wdp6yhBkTY0YsttW0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Am 06.04.22 um 23:19 schrieb Jan Bramkamp: > On 06.04.22 22:43, mike tancsa wrote: >> On 4/6/2022 4:18 PM, Bob Friesenhahn wrote: >>> On Wed, 6 Apr 2022, egoitz@ramattack.net wrote: >>>>> >>>>> WE DON'T USE COMPRESSION AS IT'S NOT SET BY DEFAULT. SOME PEOPLE SA= Y YOU >>>>> SHOULD HAVE IT ENABLED.... BUT.... JUST FOR AVOID HAVING SOME DATA >>>>> COMPRESSED SOME OTHER NOT (IN CASE YOU ENABLE AND LATER DISABLE) AN= D >>>>> FINALLY FOR AVOID ACCESSING TO INFORMATION WITH DIFFERENT CPU COSTS= OF >>>>> HANDLING... WE HAVE NOT TOUCHED COMPRESSION.... >>> >>> There seems to be a problem with your caps-lock key. >>> >>> Since it seems that you said that you are using maildir for your mail= >>> server, it is likely very useful if you do enable even rather mild >>> compression (e.g. lz4) since this will reduce the write work-load and= even >>> short files will be stored more efficiently. >>> >> FYI, a couple of our big zfs=C2=A0 mailspools sees a 1.24x and 1.23x c= ompress >> ratio with lz4.=C2=A0 We use Maildir format as well.=C2=A0 They are no= t RELENG_13 so >> not sure how zstd would fair. > I've found that Dovecot's mdbox format compresses a lot better than Mai= ldir (or > sdbox), because it stores multiple messages per file resulting in files= large > enough to contain enough exploitable reduncancy to compress down to the= next > smaller blocksize. In a corporate or education environment where users = tend to > send the same medium to large attachments multiple times to multiple re= cipients > on the same server Dovecot's single instance storage is a game changer.= It > reduced my IMAP storage requirements by a *factor* of 4.7 which allowed= me to > get rid of spinning disks for the mail servers instead of playing losin= g games > with hybrid storage. Dovecot also supports zlib compression in the appl= ication > instead of punting it to the file system. I don't know if Cyrus IMAP of= fers > similar features, but if it does I would recommend evaluating them inst= ead of > compressing or deduplicating at the file system level. I have not compared dovecot's zlib compression with zstd-2 on the file sy= stem, but since I use the latter on all my ZFS file systems (excepts those that= exclusively hold compressed files and media), I'm using it for Dovecot md= box files, too. I get a compression ratio of 2,29 with ZFS zstd-2, maybe I sh= ould copy the files over into a zlib compressed mdbox for comparison ... One large advantage of the mdbox format in the context of the mail server= set-up at the start of this thread is that deletions are only registered = in an index file (while mbox needs a rewrite of potentially large parts of t= he mail folder and mdir immediately deletes files (TRIM) and updates inodes = and directory entries, causing multiple writes per deleted message). With mdbox you can delay all "expensive" file system operations to the point of least load each day, for example. Such a compression run is also= well suited for SSDs, since it does not perform random updates that punch= holes in a large number of erase blocks (which then will need to be garba= ge collected, causing write amplification to put further load and stress on the SSD). --------------ce4Isv9Wdp6yhBkTY0YsttW0-- --------------o4K5Sngmd3lLPnmlzLRzMtEK Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEo3HqZZwL7MgrcVMTR+u171r99UQFAmJOmd8FAwAAAAAACgkQR+u171r99UQz fQgAj/0scy7zAbl1SoRPExnKQSTSk320RX81cVGflFCk2hDHRKeF9bScO22aZil0nYKaCKPR1Mps kyujxrmFpTgWjrjhNe7noHe5sz3LlGplXB7YNMKr0eujF1VC9YrlSvQLGTDFJeJyIRcI7EjSAgoy 8aLZjMG8rI7XCiMo1y+bpJqyWsElxYFoiomi2h2fkZ5MFZtWfyqPNaCFd2e4YHW6x6WYS9/H9ZDc k3E6GoFUaoRIuHC5dw2HbeUxBr72TYRLZgSH5pMBlk0cN60QNHkOBiBahbyzPVA/U1IJqCSVTWJP 7ZUNURO6nTn71io0o148Jho31IitSdxA4cuaJOOIVg== =DqKu -----END PGP SIGNATURE----- --------------o4K5Sngmd3lLPnmlzLRzMtEK--