Odd behaviour of two identical ZFS servers mirroring via rsync

From: andy thomas <andy_at_time-domain.co.uk>
Date: Fri, 11 Nov 2022 17:42:44 UTC
I have two identical servers, called clustor2 and clustor-backup, each 
with a ZFS RAIDZ-1 pool containing 9 SAS hard disks plus one spare and two 
SSDs for the ZIL and ARC functions. clustor2 stores user data from a 
HPC while clustor2-backup uses rsync to mirrors all the data from clustor2 
every 24 hours.

However, the disk usage on the mirror server is considerably more than on 
the other server - attached is a screenshot showing the two servers side 
by side, with the mirror server on the right, and displaying the contents 
of the same subdirectory choen at random (named 'ratio_10.0' in this 
instance); as you can see, the sizes of the files within each of the 
folders are identical but 'du' reports very different 
space usages for each folder and 'zpool list' also reports a significant 
difference in ZFS pool size.

I'm not sure if this is relevant but both servers have ZFS pools with no 
compression although lz4 compression is enabled on the ZFS filesystems & 
both run FreeBSD 11.3 with ZFS version 5.

Perhaps using zfs send/receive instead of rsync for mirroring might solve 
this disparity?

Thanks in advance for any suggestions,

Andy