Re: ZFS on high-latency devices

From: Johannes Totz <>
Date: Fri, 20 Aug 2021 00:16:28 +0100
On 19/08/2021 10:37, Peter Jeremy wrote:
> I'm looking at backing up my local ZFS pools to a remote system
> without the remote system having access to the pool content.  The
> options for the backup pools seem to be:
> a) Use OpenZFS native encryption with ZFS running on the remote system
>     backed up using "zfs send --raw".
> b) Run ZFS over geli locally over geom_gate[1] to the remote system.
> The first approach removes RTT as an issue but requires that the local
> pools first be converted to native encryption - a process that seems
> to be generously defined as "very difficult".
> The second approach removes the need to encrypt the local pool but
> is far more sensitive to RTT issues and I've been unable to get
> a reasonable throughput.  The main problems I've found are:
> * Even with a quite high write aggregation limit, I still get lots of
>    relatively small writes.
> * Snapshot boundaries appear to wait for all queued writes to be flushed.
> I've found but I
> can't get the procedure to work.  "zfs send" of a zvol seems to bear
> very little resemblance to a "zfs send" of a "normal" filesystem.
> Sending a zvol, I can't get ndirty _down_ to the suggested 70-80%,
> whereas with (eg) my mail spool, I can't get ndirty _up_ to the
> suggested 70-80%.  And most of the suggested adjustments are system-
> wide so the suggested changes are likely to adversely impact local
> ZFS performance.
> Does anyone have any suggestions as to a way forward?  Either a
> reliable process to encrypt an existing pool or a way to improve
> throughput doing "zfs recv" to a pool with a high RTT.

Do you have geli included in those perf tests? Any difference if you 
leave it out?

What's making the throughout slow? zfs issuing a bunch of small writes 
and then trying to read something (unrelated)? Is there just not enough 
data to be written to saturate the link?

Totally random thought: there used to be a vdev cache (not sure if 
that's still around) that would inflate read requests to hopefully drag 
in more data that might be useful soon.

Have you tried hastd?
Received on Thu Aug 19 2021 - 23:16:28 UTC

Original text of this message