speeding up zfs send | recv

Alan Somers asomers at freebsd.org
Thu May 13 15:37:37 UTC 2021

On Thu, May 13, 2021 at 8:45 AM mike tancsa <mike at sentex.net> wrote:

> For offsite storage, I have been doing a zfs send across a 10G link and
> noticed something I don't understand with respect to speed.  I have a
> number of datasets I am sending, one which contains a great many
> Maildirs / files and others that a few large 30-60G files (vm disk
> images).  When I am sending the data set with the mail spool (many small
> files and directories), transfer tends to be markedly slower.  Looking
> at the cacti graph, it seems to hover around 500Mb/s through an
> aes128-gcm cipher when sending the mail spool, vs sending the dataset
> that has the VMs on it, around 2.5Gb/s (both on a 5min average)...
> Why would the mail spool send be so slow compared to the sends where
> datasets only have a few large files ?
> One thing I am wondering is, could it be due to the amount of snapshots
> I have ? For each, I have about 60-100 snapshots. I am only sending a
> copy based on the latest snapshot, but I guess that's a lot of
> calculations to go through in order to get a complete image. However, I
> would have thought that would impact both types of datasets equally ?
> e.g. on my oldest mailspool snapshot, I see 60G of difference from the
> oldest snapshot on a dataset that's about 600GB in size
> By contrast, the dataset with VM images, is 300G and the oldest snapshot
> shows just 16G of difference and has a total of 93 snapshots.
> Is there anything I can do to speed up the send ? The recv side has lots
> of spare CPU. I dont see the disk blocking at all.  The sending side is
> pretty busy, but I would imagine equally busy across all data sets.
> sender is a recent RELENG_12, recv side is RELENG_13
> as a side note, is zstd ever nice!  On a different dataset that has a
> lot of big ass json files. I am seeing refcompressratio  22.15x   vs
> 13.19x  for the old lz4.
>     ---Mike

Is this a high latency link?  ZFS send streams can be bursty.  Piping the
stream through mbuffer helps with that.  Just google "zfs send mbuffer" for
some examples.  And be aware that your speed may be limited by the sender.
Especially if those small files are randomly spread across the platter,
your sending server's disks may be the limiting factor.  Use gstat to check.

More information about the freebsd-fs mailing list