From nobody Thu Aug 19 09:37:39 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7DCA0176E575 for ; Thu, 19 Aug 2021 09:38:03 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "vtr.rulingia.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gr07G0qJmz4WZC for ; Thu, 19 Aug 2021 09:38:01 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (2001-44b8-31fc-0d00-41ad-e251-cf00-b0b1.static.ipv6.internode.on.net [IPv6:2001:44b8:31fc:d00:41ad:e251:cf00:b0b1]) by vtr.rulingia.com (8.16.1/8.15.2) with ESMTPS id 17J9biZ4026313 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 19 Aug 2021 19:37:50 +1000 (AEST) (envelope-from peter@rulingia.com) DKIM-Filter: OpenDKIM Filter v2.10.3 vtr.rulingia.com 17J9biZ4026313 X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.16.1/8.16.1) with ESMTPS id 17J9bdnD039514 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 19 Aug 2021 19:37:39 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.16.1/8.16.1/Submit) id 17J9bdta039513 for freebsd-fs@freebsd.org; Thu, 19 Aug 2021 19:37:39 +1000 (AEST) (envelope-from peter) Date: Thu, 19 Aug 2021 19:37:39 +1000 From: Peter Jeremy To: freebsd-fs@freebsd.org Subject: ZFS on high-latency devices Message-ID: List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="/blLqS9gkC3/xMqZ" Content-Disposition: inline X-PGP-Key: http://www.rulingia.com/keys/peter.pgp X-Rspamd-Queue-Id: 4Gr07G0qJmz4WZC X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=quarantine) header.from=rulingia.com; spf=pass (mx1.freebsd.org: domain of peter@rulingia.com designates 2001:19f0:5801:ebe:5400:1ff:fe53:30fd as permitted sender) smtp.mailfrom=peter@rulingia.com X-Spamd-Result: default: False [-5.87 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[peter]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.965]; DMARC_POLICY_ALLOW(-0.50)[rulingia.com,quarantine]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20473, ipnet:2001:19f0:5800::/38, country:US]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --/blLqS9gkC3/xMqZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I'm looking at backing up my local ZFS pools to a remote system without the remote system having access to the pool content. The options for the backup pools seem to be: a) Use OpenZFS native encryption with ZFS running on the remote system backed up using "zfs send --raw". b) Run ZFS over geli locally over geom_gate[1] to the remote system. The first approach removes RTT as an issue but requires that the local pools first be converted to native encryption - a process that seems to be generously defined as "very difficult". The second approach removes the need to encrypt the local pool but is far more sensitive to RTT issues and I've been unable to get a reasonable throughput. The main problems I've found are: * Even with a quite high write aggregation limit, I still get lots of relatively small writes. * Snapshot boundaries appear to wait for all queued writes to be flushed. I've found https://openzfs.org/wiki/ZFS_on_high_latency_devices but I can't get the procedure to work. "zfs send" of a zvol seems to bear very little resemblance to a "zfs send" of a "normal" filesystem. Sending a zvol, I can't get ndirty _down_ to the suggested 70-80%, whereas with (eg) my mail spool, I can't get ndirty _up_ to the suggested 70-80%. And most of the suggested adjustments are system- wide so the suggested changes are likely to adversely impact local ZFS performance. Does anyone have any suggestions as to a way forward? Either a reliable process to encrypt an existing pool or a way to improve throughput doing "zfs recv" to a pool with a high RTT. --=20 Peter Jeremy --/blLqS9gkC3/xMqZ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAmEeJlxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzS6LRAAiBR1BHdBtlf7jnNmPuvMn+oaYwW1HeEmLzdcD/YrGwJN6vP0d7LV2AP9 ceY3E2SiRTN/VDcYAYuWBk4UzNbV16tYbs6f6qQpAYH0diF0lFzjYe+hsnvDZFMo NXskXqAyhGPqciDTn+lRbNZETDJbnonNVmGUw7HfzG7wTPZOyr/5odq+YXyep6s0 zmpFTheLStYKMXnbaxBazoV6ydzQ9y4Elc/aCqswMV29vHxDSp5j/fdDwG6ZQRFs B2+4AqzF9Ea8eASzDeX0qzwhiH8jXNDCP0Xp0uPusSnzYJBu2i/x42wbT7Dp9DVE oTIXYMYlX2WZTLrUyB7evgEdce/DLH01LP/ZPLTPSK+vHPtoLIV/FiVre0AqUO3j ilJli+O+OKeITyWFjI60bXOoWWJpl+zyirwCtC2viKwtwyj4nj1jTrSHK9d8dCbq KI///r89DivnU2tKgSIbvCgwtE8YUi4T5dc0VJzr2c+b0Vf39R3LbC+C4FdvperW AqU6kuN9sVgqsJK5kLUBh4buAWDGhpfNxllbEh9sGioEodnnEi1nM5MmWqPhLnAM Lz44UMeSJEu4+XoQt9MO5QObWfi86382oN9qA7khMIad1aEN4eqwLKH9f3QutfR5 kkCbG+BRq7StBkeV8YzqPBJe2BKgBDA3BUHZe5p+5/kIXe+OC+k= =CRZo -----END PGP SIGNATURE----- --/blLqS9gkC3/xMqZ--