Unexpected zvol usage
Paul Esson
paul.esson at redstor.com
Fri May 18 15:42:09 UTC 2018
Hi Folks,
I have an 11.1-RELEASE system being used as a host for a bhyve guest. There is a large zpool on the host created from 12 x 10TB HDDs using raidz2 redundancy with ashift12. I have created a sparse zvol within the pool using default settings and presented that to the bhyve vm as an ahci-hd disk type. The guest has a zpool and filesystem dataset built on this disk. When I start to write to the filesystem on the guest I am finding that the used/referenced on the host's zvol are more than double those on the guest. The logicalused/referenced values on the host zvol are more in line with the equivalent guest values, but my problem is that the host zvol is likely to fill before I have written all intended data to the guest.
I have included below information from both the host and guest before and after writing. This output shows that the zvol uses a default 8K blocksize and that the guest zfs is therefore ashift13. I also tried creating the zvol with a 4K blocksize and the guest zfs ashift12 so that 4K blocks were consistent across hosts and guest, but still saw the amplification on writes to the zvol.
Any insight greatly appreciated.
HOST
Zpool
RAIDZ2 12 x HDDs, ashift 12
NAME PROPERTY VALUE SOURCE
dc1-hn-01 type filesystem -
dc1-hn-01 creation Mon Apr 23 14:35 2018 -
dc1-hn-01 used 32.0G -
dc1-hn-01 available 78.2T -
dc1-hn-01 referenced 201K -
dc1-hn-01 compressratio 1.00x -
dc1-hn-01 mounted yes -
dc1-hn-01 quota none default
dc1-hn-01 reservation none default
dc1-hn-01 recordsize 128K default
dc1-hn-01 mountpoint /export/data/dc1-hn-01 local
dc1-hn-01 sharenfs off default
dc1-hn-01 checksum on default
dc1-hn-01 compression off default
dc1-hn-01 atime on default
dc1-hn-01 devices on default
dc1-hn-01 exec on default
dc1-hn-01 setuid on default
dc1-hn-01 readonly off default
dc1-hn-01 jailed off default
dc1-hn-01 snapdir hidden default
dc1-hn-01 aclmode discard default
dc1-hn-01 aclinherit restricted default
dc1-hn-01 canmount on default
dc1-hn-01 xattr off temporary
dc1-hn-01 copies 1 default
dc1-hn-01 version 5 -
dc1-hn-01 utf8only off -
dc1-hn-01 normalization none -
dc1-hn-01 casesensitivity sensitive -
dc1-hn-01 vscan off default
dc1-hn-01 nbmand off default
dc1-hn-01 sharesmb off default
dc1-hn-01 refquota none default
dc1-hn-01 refreservation none default
dc1-hn-01 primarycache all default
dc1-hn-01 secondarycache all default
dc1-hn-01 usedbysnapshots 0 -
dc1-hn-01 usedbydataset 201K -
dc1-hn-01 usedbychildren 32.0G -
dc1-hn-01 usedbyrefreservation 0 -
dc1-hn-01 logbias latency default
dc1-hn-01 dedup off default
dc1-hn-01 mlslabel -
dc1-hn-01 sync standard default
dc1-hn-01 refcompressratio 1.00x -
dc1-hn-01 written 201K -
dc1-hn-01 logicalused 2.89G -
dc1-hn-01 logicalreferenced 36.5K -
dc1-hn-01 volmode default default
dc1-hn-01 filesystem_limit none default
dc1-hn-01 snapshot_limit none default
dc1-hn-01 filesystem_count none default
dc1-hn-01 snapshot_count none default
dc1-hn-01 redundant_metadata all default
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
dc1-hn-01 78.2T 32.0G 0 201K 0 32.0G
dc1-hn-01/vm 78.2T 31.9G 0 990M 0 30.9G
dc1-hn-01/vm/dc1-olbp-sn-11 78.2T 30.9G 0 238K 0 30.9G
dc1-hn-01/vm/dc1-olbp-sn-11/disk0 78.2T 30.9G 0 4.35G 26.6G 0
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 78.2T 4.50M 0 4.50M 0 0
Sparse ZVOL - baseline
NAME PROPERTY VALUE SOURCE
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 type volume -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 creation Fri May 18 15:36 2018 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 used 4.50M -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 available 78.2T -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 referenced 4.50M -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compressratio 1.00x -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 reservation none default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volsize 28T local
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volblocksize 8K -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 checksum on default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compression off default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 readonly off default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 copies 1 default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refreservation none default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 primarycache all default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 secondarycache all default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbysnapshots 0 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbydataset 4.50M -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbychildren 0 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbyrefreservation 0 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logbias latency default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 dedup off default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 mlslabel -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 sync standard default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refcompressratio 1.00x -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 written 4.50M -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalused 1.89M -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalreferenced 1.89M -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volmode dev local
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_limit none default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_count none default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 redundant_metadata all default
GUEST - baseline
1 x vdisk from host ZVOL ashift 13
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
dc1-sn-11 26.9T 632K 0 176K 0 456K
NAME PROPERTY VALUE SOURCE
dc1-sn-11 type filesystem -
dc1-sn-11 creation Fri May 18 15:40 2018 -
dc1-sn-11 used 632K -
dc1-sn-11 available 26.9T -
dc1-sn-11 referenced 176K -
dc1-sn-11 compressratio 1.00x -
dc1-sn-11 mounted yes -
dc1-sn-11 quota none default
dc1-sn-11 reservation none default
dc1-sn-11 recordsize 128K default
dc1-sn-11 mountpoint /export/data/dc1-sn-11 local
dc1-sn-11 sharenfs off default
dc1-sn-11 checksum on default
dc1-sn-11 compression off default
dc1-sn-11 atime on default
dc1-sn-11 devices on default
dc1-sn-11 exec on default
dc1-sn-11 setuid on default
dc1-sn-11 readonly off default
dc1-sn-11 jailed off default
dc1-sn-11 snapdir hidden default
dc1-sn-11 aclmode discard default
dc1-sn-11 aclinherit restricted default
dc1-sn-11 canmount on default
dc1-sn-11 xattr off temporary
dc1-sn-11 copies 1 default
dc1-sn-11 version 5 -
dc1-sn-11 utf8only off -
dc1-sn-11 normalization none -
dc1-sn-11 casesensitivity sensitive -
dc1-sn-11 vscan off default
dc1-sn-11 nbmand off default
dc1-sn-11 sharesmb off default
dc1-sn-11 refquota none default
dc1-sn-11 refreservation none default
dc1-sn-11 primarycache all default
dc1-sn-11 secondarycache all default
dc1-sn-11 usedbysnapshots 0 -
dc1-sn-11 usedbydataset 176K -
dc1-sn-11 usedbychildren 456K -
dc1-sn-11 usedbyrefreservation 0 -
dc1-sn-11 logbias latency default
dc1-sn-11 dedup off default
dc1-sn-11 mlslabel -
dc1-sn-11 sync standard default
dc1-sn-11 refcompressratio 1.00x -
dc1-sn-11 written 176K -
dc1-sn-11 logicalused 49K -
dc1-sn-11 logicalreferenced 11.5K -
dc1-sn-11 volmode default default
dc1-sn-11 filesystem_limit none default
dc1-sn-11 snapshot_limit none default
dc1-sn-11 filesystem_count none default
dc1-sn-11 snapshot_count none default
dc1-sn-11 redundant_metadata all default
After writing some data to the guest
HOST ZVOL
NAME PROPERTY VALUE SOURCE
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 type volume -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 creation Fri May 18 15:36 2018 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 used 99.7G -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 available 78.1T -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 referenced 99.7G -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compressratio 1.00x -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 reservation none default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volsize 28T local
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volblocksize 8K -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 checksum on default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 compression off default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 readonly off default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 copies 1 default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refreservation none default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 primarycache all default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 secondarycache all default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbysnapshots 0 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbydataset 99.7G -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbychildren 0 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 usedbyrefreservation 0 -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logbias latency default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 dedup off default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 mlslabel -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 sync standard default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 refcompressratio 1.00x -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 written 99.7G -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalused 43.6G -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 logicalreferenced 43.6G -
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 volmode dev local
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_limit none default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 snapshot_count none default
dc1-hn-01/vm/dc1-olbp-sn-11/disk1 redundant_metadata all default
GUEST ZFS
NAME PROPERTY VALUE SOURCE
dc1-sn-11 type filesystem -
dc1-sn-11 creation Fri May 18 15:40 2018 -
dc1-sn-11 used 44.3G -
dc1-sn-11 available 26.8T -
dc1-sn-11 referenced 176K -
dc1-sn-11 compressratio 1.00x -
dc1-sn-11 mounted no -
dc1-sn-11 quota none default
dc1-sn-11 reservation none default
dc1-sn-11 recordsize 128K default
dc1-sn-11 mountpoint /export/data/dc1-sn-11 local
dc1-sn-11 sharenfs off default
dc1-sn-11 checksum on default
dc1-sn-11 compression off default
dc1-sn-11 atime on default
dc1-sn-11 devices on default
dc1-sn-11 exec on default
dc1-sn-11 setuid on default
dc1-sn-11 readonly off default
dc1-sn-11 jailed off default
dc1-sn-11 snapdir hidden default
dc1-sn-11 aclmode discard default
dc1-sn-11 aclinherit restricted default
dc1-sn-11 canmount on default
dc1-sn-11 xattr on default
dc1-sn-11 copies 1 default
dc1-sn-11 version 5 -
dc1-sn-11 utf8only off -
dc1-sn-11 normalization none -
dc1-sn-11 casesensitivity sensitive -
dc1-sn-11 vscan off default
dc1-sn-11 nbmand off default
dc1-sn-11 sharesmb off default
dc1-sn-11 refquota none default
dc1-sn-11 refreservation none default
dc1-sn-11 primarycache all default
dc1-sn-11 secondarycache all default
dc1-sn-11 usedbysnapshots 0 -
dc1-sn-11 usedbydataset 176K -
dc1-sn-11 usedbychildren 44.3G -
dc1-sn-11 usedbyrefreservation 0 -
dc1-sn-11 logbias latency default
dc1-sn-11 dedup off default
dc1-sn-11 mlslabel -
dc1-sn-11 sync standard default
dc1-sn-11 refcompressratio 1.00x -
dc1-sn-11 written 176K -
dc1-sn-11 logicalused 44.2G -
dc1-sn-11 logicalreferenced 11.5K -
dc1-sn-11 volmode default default
dc1-sn-11 filesystem_limit none default
dc1-sn-11 snapshot_limit none default
dc1-sn-11 filesystem_count none default
dc1-sn-11 snapshot_count none default
dc1-sn-11 redundant_metadata all default
Regards,
Paul Esson
t +44 (0)118 951 5235 | m +44 (0)776 690 6514
e paul.esson at redstor.com<mailto:paul.esson at redstor.com>
More information about the freebsd-fs
mailing list