zpool list show nonsense on raidz pools, at least it looks like it for me

Eugene M. Zheganin emz at norma.perm.ru
Wed Apr 12 18:01:12 UTC 2017


It's not my first letter where I fail to understand the space usage from 
zfs utilities, and in previous ones I was kind of convinced that I just 
read it wrong, but not this time I guess. See for yourself:

[emz at san01:~]> zpool list data
data  17,4T  7,72T  9,66T         -    46%    44%  1.00x  ONLINE -

Here' as I understand it, zpool says that less than a half of the pool 
is used. As far as I know this is very complicated when it comes to the 
radiz pools. Let's see:

[emz at san01:~]> zfs list -t all data
data  13,3T   186G  27,2K  /data

So, if we won't investigate further, it looks like that only 186G is 
free. Spoiling - this is the real free space amount, because I've just 
managed to free 160 gigs of data, and I really know I was short on space 
when sending 30 Gb dataset, because zfs was saying "Not enough free 
space". So, let's investigate further:

[emz at san01:~]> zfs list -t all | more
NAME                                USED  AVAIL  REFER  MOUNTPOINT
data                               13,3T   186G  27,2K  /data
data/esx                           5,23T   186G  27,2K  /data/esx
data/esx/boot-esx01                8,25G   193G   561M  -
data/esx/boot-esx02                8,25G   193G   561M  -
data/esx/boot-esx03                8,25G   193G   561M  -
data/esx/boot-esx04                8,25G   193G   561M  -
data/esx/boot-esx05                8,25G   193G   561M  -
data/esx/boot-esx06                8,25G   193G   561M  -
data/esx/boot-esx07                8,25G   193G   962M  -
data/esx/boot-esx08                8,25G   193G   562M  -
data/esx/boot-esx09                8,25G   193G   562M  -
data/esx/boot-esx10                8,25G   193G   595M  -
data/esx/boot-esx11                8,25G   193G   539M  -
data/esx/boot-esx12                8,25G   193G   539M  -
data/esx/boot-esx13                8,25G   193G   539M  -
data/esx/boot-esx14                8,25G   193G   539M  -
data/esx/boot-esx15                8,25G   193G   539M  -
data/esx/boot-esx16                8,25G   193G   541M  -
data/esx/boot-esx17                8,25G   193G   540M  -
data/esx/boot-esx18                8,25G   193G   539M  -
data/esx/boot-esx19                8,25G   193G   542M  -
data/esx/boot-esx20                8,25G   194G  12,8K  -
data/esx/boot-esx21                8,25G   194G  12,8K  -
data/esx/boot-esx22                8,25G   193G   913M  -
data/esx/boot-esx23                8,25G   193G   558M  -
data/esx/boot-esx24                8,25G   194G  12,8K  -
data/esx/boot-esx25                8,25G   194G  12,8K  -
data/esx/boot-esx26                8,25G   194G  12,8K  -
data/esx/shared                    5,02T  2,59T  2,61T  -
data/reference                     6,74T  4,17T  2,73T  -
data/reference at ver7_214             127M      -  2,73T  -
data/reference at ver2_739            12,8M      -  2,73T  -
data/reference at ver2_740            5,80M      -  2,73T  -
data/reference at ver2_741            4,55M      -  2,73T  -
data/reference at ver2_742             993K      -  2,73T  -
data/reference-ver2_739-worker100  1,64G   186G  2,73T  -
data/reference-ver2_739-worker101   254M   186G  2,73T  -
data/reference-ver2_739-worker102   566K   186G  2,73T  -
data/reference-ver2_739-worker103   260M   186G  2,73T  -
data/reference-ver2_739-worker104  8,74G   186G  2,73T  -
data/reference-ver2_739-worker105  4,19G   186G  2,73T  -
data/reference-ver2_739-worker106  1,72G   186G  2,73T  -
data/reference-ver2_739-worker107   282M   186G  2,73T  -
data/reference-ver2_739-worker108  1,27M   186G  2,73T  -
data/reference-ver2_739-worker109  8,74G   186G  2,73T  -
data/reference-ver2_739-worker110  8,74G   186G  2,73T  -
data/reference-ver2_739-worker111  8,74G   186G  2,73T  -
data/reference-ver2_739-worker112  8,74G   186G  2,73T  -
data/reference-ver2_739-worker113   838K   186G  2,73T  -
data/reference-ver2_739-worker114  8,74G   186G  2,73T  -
data/reference-ver2_739-worker115  8,74G   186G  2,73T  -
data/reference-ver2_739-worker116  8,74G   186G  2,73T  -
data/reference-ver2_739-worker117  8,74G   186G  2,73T  -
data/reference-ver2_739-worker118  8,74G   186G  2,73T  -
data/reference-ver2_739-worker119  8,74G   186G  2,73T  -
data/reference-ver2_739-worker120  8,74G   186G  2,73T  -
data/reference-ver2_739-worker121  8,74G   186G  2,73T  -
data/reference-ver2_739-worker122  8,74G   186G  2,73T  -
data/reference-ver2_739-worker123  8,74G   186G  2,73T  -
data/reference-ver2_739-worker124  8,74G   186G  2,73T  -
data/reference-ver2_739-worker125  8,74G   186G  2,73T  -
data/reference-ver2_739-worker126  8,74G   186G  2,73T  -
data/reference-ver2_739-worker127  8,74G   186G  2,73T  -
data/reference-ver2_739-worker128  8,74G   186G  2,73T  -
data/reference-ver2_739-worker129  8,74G   186G  2,73T  -
data/reference-ver2_739-worker130  8,74G   186G  2,73T  -
data/reference-ver2_739-worker131  8,74G   186G  2,73T  -
data/reference-ver2_739-worker132  8,74G   186G  2,73T  -
data/reference-ver2_739-worker133  8,74G   186G  2,73T  -
data/reference-ver2_739-worker134  8,74G   186G  2,73T  -
data/reference-ver2_739-worker135  8,74G   186G  2,73T  -
data/reference-ver2_739-worker136  8,74G   186G  2,73T  -
data/reference-ver2_739-worker137  8,74G   186G  2,73T  -
data/reference-ver2_739-worker138  8,74G   186G  2,73T  -
data/reference-ver2_739-worker139  8,74G   186G  2,73T  -
data/reference-ver2_739-worker140  8,74G   186G  2,73T  -
data/reference-ver2_739-worker141  8,74G   186G  2,73T  -
data/reference-ver2_739-worker142  8,74G   186G  2,73T  -
data/reference-ver2_739-worker143  8,73G   186G  2,73T  -
data/reference-ver2_739-worker144  9,16G   186G  2,73T  -
data/reference-ver2_739-worker145  8,74G   186G  2,73T  -
data/reference-ver2_739-worker146  8,74G   186G  2,73T  -
data/reference-ver2_739-worker147  8,74G   186G  2,73T  -
data/reference-ver2_739-worker148  8,74G   186G  2,73T  -
data/reference-ver2_739-worker149  8,74G   186G  2,73T  -
data/reference-ver2_739-worker150  8,74G   186G  2,73T  -
data/reference-ver2_739-worker151  8,74G   186G  2,73T  -
data/reference-ver2_739-worker152  8,74G   186G  2,73T  -
data/reference-ver2_739-worker53    385K   186G  2,73T  -
data/reference-ver2_739-worker58   9,49M   186G  2,73T  -
data/reference-ver2_739-worker81   8,20G   186G  2,73T  -
data/reference-ver2_739-worker82   7,64G   186G  2,73T  -
data/reference-ver2_739-worker83   6,84G   186G  2,73T  -
data/reference-ver2_739-worker84   5,95G   186G  2,73T  -
data/reference-ver2_739-worker85   5,09G   186G  2,73T  -
data/reference-ver2_739-worker86   3,76G   186G  2,73T  -
data/reference-ver2_739-worker87   4,06G   186G  2,73T  -
data/reference-ver2_739-worker88   1,92G   186G  2,73T  -
data/reference-ver2_739-worker89   6,83G   186G  2,73T  -
data/reference-ver2_739-worker90   6,55G   186G  2,73T  -
data/reference-ver2_739-worker91   5,25G   186G  2,73T  -
data/reference-ver2_739-worker92   4,18G   186G  2,73T  -
data/reference-ver2_739-worker93   3,18G   186G  2,73T  -
data/reference-ver2_739-worker94    983M   186G  2,73T  -
data/reference-ver2_739-worker95    525M   186G  2,73T  -
data/reference-ver2_739-worker96    650M   186G  2,73T  -
data/reference-ver2_739-worker97   5,78G   186G  2,73T  -
data/reference-ver2_739-worker98   4,81G   186G  2,73T  -
data/reference-ver2_739-worker99   4,93G   186G  2,73T  -
data/reference-ver2_740-worker56    361K   186G  2,73T  -
data/reference-ver2_741-worker74   3,98M   186G  2,73T  -
data/reference-ver2_742-worker02    934K   186G  2,73T  -
data/reference-ver2_742-worker03    935K   186G  2,73T  -
data/reference-ver2_742-worker04    935K   186G  2,73T  -
data/reference-ver2_742-worker05    935K   186G  2,73T  -
data/reference-ver2_742-worker06    935K   186G  2,73T  -
data/reference-ver2_742-worker08    934K   186G  2,73T  -
data/reference-ver2_742-worker09   15,9M   186G  2,73T  -
data/reference-ver2_742-worker10   16,2M   186G  2,73T  -
data/reference-ver2_742-worker11   15,4M   186G  2,73T  -
data/reference-ver2_742-worker12    935K   186G  2,73T  -
data/reference-ver2_742-worker13   15,9M   186G  2,73T  -
data/reference-ver2_742-worker14    935K   186G  2,73T  -
data/reference-ver2_742-worker15   15,5M   186G  2,73T  -
data/reference-ver2_742-worker16    934K   186G  2,73T  -
data/reference-ver2_742-worker17    934K   186G  2,73T  -
data/reference-ver2_742-worker18    935K   186G  2,73T  -
data/reference-ver2_742-worker19    531M   186G  2,73T  -
data/reference-ver2_742-worker20    935K   186G  2,73T  -
data/reference-ver2_742-worker21    935K   186G  2,73T  -
data/reference-ver2_742-worker22   5,27G   186G  2,73T  -
data/reference-ver2_742-worker23    934K   186G  2,73T  -
data/reference-ver2_742-worker24   5,27G   186G  2,73T  -
data/reference-ver2_742-worker25    934K   186G  2,73T  -
data/reference-ver2_742-worker26    934K   186G  2,73T  -
data/reference-ver2_742-worker27   15,8M   186G  2,73T  -
data/reference-ver2_742-worker28    935K   186G  2,73T  -
data/reference-ver2_742-worker29    934K   186G  2,73T  -
data/reference-ver2_742-worker30    934K   186G  2,73T  -
data/reference-ver2_742-worker31    934K   186G  2,73T  -
data/reference-ver2_742-worker32    444M   186G  2,73T  -
data/reference-ver2_742-worker33    935K   186G  2,73T  -
data/reference-ver2_742-worker34   15,9M   186G  2,73T  -
data/reference-ver2_742-worker35   1,12M   186G  2,73T  -
data/reference-ver2_742-worker36    935K   186G  2,73T  -
data/reference-ver2_742-worker37   5,27G   186G  2,73T  -
data/reference-ver2_742-worker38    935K   186G  2,73T  -
data/reference-ver2_742-worker39    935K   186G  2,73T  -
data/reference-ver2_742-worker40    935K   186G  2,73T  -
data/reference-ver2_742-worker41   15,5M   186G  2,73T  -
data/reference-ver2_742-worker42    934K   186G  2,73T  -
data/reference-ver2_742-worker43   15,4M   186G  2,73T  -
data/reference-ver2_742-worker44    429M   186G  2,73T  -
data/reference-ver2_742-worker45   15,5M   186G  2,73T  -
data/reference-ver2_742-worker46    935K   186G  2,73T  -
data/reference-ver2_742-worker47    934K   186G  2,73T  -
data/reference-ver2_742-worker48   15,5M   186G  2,73T  -
data/reference-ver2_742-worker49    935K   186G  2,73T  -
data/reference-ver2_742-worker50    934K   186G  2,73T  -
data/reference-ver2_742-worker51    935K   186G  2,73T  -
data/reference-ver2_742-worker52    935K   186G  2,73T  -
data/reference-ver2_742-worker54    935K   186G  2,73T  -
data/reference-ver2_742-worker55    934K   186G  2,73T  -
data/reference-ver2_742-worker57    935K   186G  2,73T  -
data/reference-ver2_742-worker59    934K   186G  2,73T  -
data/reference-ver2_742-worker60   15,5M   186G  2,73T  -
data/reference-ver2_742-worker61    934K   186G  2,73T  -
data/reference-ver2_742-worker62    934K   186G  2,73T  -
data/reference-ver2_742-worker63    934K   186G  2,73T  -
data/reference-ver2_742-worker64   15,5M   186G  2,73T  -
data/reference-ver2_742-worker65    934K   186G  2,73T  -
data/reference-ver2_742-worker66    935K   186G  2,73T  -
data/reference-ver2_742-worker67    934K   186G  2,73T  -
data/reference-ver2_742-worker68    935K   186G  2,73T  -
data/reference-ver2_742-worker69    934K   186G  2,73T  -
data/reference-ver2_742-worker70    935K   186G  2,73T  -
data/reference-ver2_742-worker71   15,5M   186G  2,73T  -
data/reference-ver2_742-worker72    935K   186G  2,73T  -
data/reference-ver2_742-worker73    935K   186G  2,73T  -
data/reference-ver2_742-worker75    934K   186G  2,73T  -
data/reference-ver2_742-worker76    935K   186G  2,73T  -
data/reference-ver2_742-worker77    934K   186G  2,73T  -
data/reference-ver2_742-worker78   15,7M   186G  2,73T  -
data/reference-ver2_742-worker79   15,4M   186G  2,73T  -
data/reference-ver2_742-worker80    935K   186G  2,73T  -
data/reference-ver7_214-worker07   2,19M   186G  2,73T  -
data/reference-ver7_214-worker53   2,18M   186G  2,73T  -
data/userdata                       825G   186G  27,2K /data/userdata
data/userdata/userdata             27,2K   186G  27,2K 
data/userdata/worker01             8,25G   190G  3,77G  -
data/userdata/worker02             8,25G   190G  3,48G  -
data/userdata/worker03             8,25G   190G  4,07G  -
data/userdata/worker04             8,25G   191G  2,99G  -
data/userdata/worker05             8,25G   190G  3,46G  -
data/userdata/worker06             8,25G   191G  2,89G  -
data/userdata/worker07             8,25G   190G  3,65G  -
data/userdata/worker08             8,25G   190G  3,55G  -
data/userdata/worker09             8,25G   190G  3,64G  -
data/userdata/worker10             8,25G   190G  3,85G  -
data/userdata/worker11             8,25G   191G  3,28G  -
data/userdata/worker12             8,25G   191G  3,19G  -
data/userdata/worker13             8,25G   191G  2,93G  -
data/userdata/worker14             8,25G   191G  2,89G  -
data/userdata/worker15             8,25G   190G  3,57G  -
data/userdata/worker153            8,25G   194G  12,8K  -
data/userdata/worker154            8,25G   194G  12,8K  -
data/userdata/worker155            8,25G   194G  12,8K  -
data/userdata/worker156            8,25G   194G  12,8K  -
data/userdata/worker157            8,25G   194G  12,8K  -
data/userdata/worker158            8,25G   194G  12,8K  -
data/userdata/worker159            8,25G   194G  12,8K  -
data/userdata/worker16             8,25G   190G  4,23G  -
data/userdata/worker160            8,25G   194G  12,8K  -
data/userdata/worker161            8,25G   194G  12,8K  -
data/userdata/worker162            8,25G   194G  12,8K  -
data/userdata/worker163            8,25G   194G  12,8K  -
data/userdata/worker164            8,25G   194G  12,8K  -
data/userdata/worker165            8,25G   194G  12,8K  -
data/userdata/worker166            8,25G   194G  12,8K  -
data/userdata/worker167            8,25G   194G  12,8K  -
data/userdata/worker168            8,25G   194G  12,8K  -
data/userdata/worker169            8,25G   194G  12,8K  -
data/userdata/worker17             8,25G   190G  3,63G  -
data/userdata/worker170            8,25G   194G  12,8K  -
data/userdata/worker171            8,25G   194G  12,8K  -
data/userdata/worker172            8,25G   194G  12,8K  -
data/userdata/worker18             8,25G   191G  3,19G  -
data/userdata/worker19             8,25G   191G  3,02G  -
data/userdata/worker20             8,25G   192G  2,35G  -
data/userdata/worker21             8,25G   188G  5,42G  -
data/userdata/worker22             8,25G   191G  2,76G  -
data/userdata/worker23             8,25G   191G  2,61G  -
data/userdata/worker24             8,25G   191G  3,22G  -
data/userdata/worker25             8,25G   189G  4,97G  -
data/userdata/worker26             8,25G   191G  3,23G  -
data/userdata/worker27             8,25G   189G  4,90G  -
data/userdata/worker28             8,25G   189G  5,31G  -
data/userdata/worker29             8,25G   189G  4,67G  -
data/userdata/worker30             8,25G   190G  3,46G  -
data/userdata/worker31             8,25G   191G  3,36G  -
data/userdata/worker32             8,25G   190G  4,11G  -
data/userdata/worker33             8,25G   190G  4,31G  -
data/userdata/worker34             8,25G   191G  3,31G  -
data/userdata/worker35             8,25G   190G  4,23G  -
data/userdata/worker36             8,25G   191G  3,24G  -
data/userdata/worker37             8,25G   191G  3,16G  -
data/userdata/worker38             8,25G   191G  2,89G  -
data/userdata/worker39             8,25G   189G  4,49G  -
data/userdata/worker40             8,25G   190G  4,21G  -
data/userdata/worker41             8,25G   190G  3,42G  -
data/userdata/worker42             8,25G   191G  3,32G  -
data/userdata/worker43             8,25G   190G  3,71G  -
data/userdata/worker44             8,25G   191G  3,33G  -
data/userdata/worker45             8,25G   189G  5,04G  -
data/userdata/worker46             8,25G   190G  3,76G  -
data/userdata/worker47             8,25G   189G  5,13G  -
data/userdata/worker48             8,25G   190G  3,81G  -
data/userdata/worker49             8,25G   191G  2,72G  -
data/userdata/worker50             8,25G   190G  3,78G  -
data/userdata/worker51             8,25G   191G  3,08G  -
data/userdata/worker52             8,25G   190G  3,38G  -
data/userdata/worker53             8,25G   190G  4,10G  -
data/userdata/worker54             8,25G   190G  3,48G  -
data/userdata/worker55             8,25G   189G  4,39G  -
data/userdata/worker56             8,25G   190G  3,58G  -
data/userdata/worker57             8,25G   190G  3,97G  -
data/userdata/worker58             8,25G   190G  4,02G  -
data/userdata/worker59             8,25G   190G  3,57G  -
data/userdata/worker60             8,25G   190G  4,04G  -
data/userdata/worker61             8,25G   190G  3,73G  -
data/userdata/worker62             8,25G   191G  3,21G  -
data/userdata/worker63             8,25G   191G  3,37G  -
data/userdata/worker64             8,25G   189G  4,77G  -
data/userdata/worker65             8,25G   189G  5,34G  -
data/userdata/worker66             8,25G   190G  3,50G  -
data/userdata/worker67             8,25G   190G  3,84G  -
data/userdata/worker68             8,25G   189G  4,73G  -
data/userdata/worker69             8,25G   191G  3,20G  -
data/userdata/worker70             8,25G   190G  3,96G  -
data/userdata/worker71             8,25G   189G  5,11G  -
data/userdata/worker72             8,25G   191G  2,92G  -
data/userdata/worker73             8,25G   189G  4,53G  -
data/userdata/worker74             8,25G   189G  4,39G  -
data/userdata/worker75             8,25G   190G  4,26G  -
data/userdata/worker76             8,25G   189G  4,67G  -
data/userdata/worker77             8,25G   189G  5,30G  -
data/userdata/worker78             8,25G   190G  3,98G  -
data/userdata/worker79             8,25G   190G  4,10G  -
data/userdata/worker80             8,25G   190G  3,48G  -

This are getting really complicated now.

What I don't understand is:

- why the amount of free space changes from dataset to dataset ? I mean 
they all share the same free space pool, all have the same 
refreservation=none, but the AVAIL differs. When it comes to workerX 
datasets, it differs slightly, but when it comes to the large zvols, 
like esx/shared or reference, it differs a lot !

- why the esx/shared and reference datasets are shown like they can be 
enlarged ? I mean, I really don't have THAT much of free space.

Here are their properties:

[emz at san01:~]> zfs get all data/esx/shared
NAME             PROPERTY VALUE                         SOURCE
data/esx/shared  type volume                        -
data/esx/shared  creation              вт февр. 21 10:19 2017  -
data/esx/shared  used 5,02T                         -
data/esx/shared  available 2,59T                         -
data/esx/shared  referenced 2,61T                         -
data/esx/shared  compressratio 1.81x                         -
data/esx/shared  reservation none                          default
data/esx/shared  volsize 5T                            local
data/esx/shared  volblocksize 64K                           -
data/esx/shared  checksum on                            default
data/esx/shared  compression lzjb                          inherited 
from data
data/esx/shared  readonly off                           default
data/esx/shared  copies 1                             default
data/esx/shared  refreservation 5,02T                         local
data/esx/shared  primarycache all                           default
data/esx/shared  secondarycache all                           default
data/esx/shared  usedbysnapshots 0                             -
data/esx/shared  usedbydataset 2,61T                         -
data/esx/shared  usedbychildren 0                             -
data/esx/shared  usedbyrefreservation 2,41T                         -
data/esx/shared  logbias latency                       default
data/esx/shared  dedup off                           default
data/esx/shared mlslabel                                            -
data/esx/shared  sync standard                      default
data/esx/shared  refcompressratio 1.81x                         -
data/esx/shared  written 2,61T                         -
data/esx/shared  logicalused 4,71T                         -
data/esx/shared  logicalreferenced 4,71T                         -
data/esx/shared  volmode default                       default
data/esx/shared  snapshot_limit none                          default
data/esx/shared  snapshot_count none                          default
data/esx/shared  redundant_metadata all                           default
[emz at san01:~]> zfs get all data/reference
NAME            PROPERTY VALUE                         SOURCE
data/reference  type volume                        -
data/reference  creation              вт февр. 21 11:12 2017  -
data/reference  used 6,74T                         -
data/reference  available 4,17T                         -
data/reference  referenced 2,73T                         -
data/reference  compressratio 1.08x                         -
data/reference  reservation none                          default
data/reference  volsize 3,97T                         local
data/reference  volblocksize 64K                           -
data/reference  checksum on                            default
data/reference  compression lzjb                          local
data/reference  readonly off                           default
data/reference  copies 1                             default
data/reference  refreservation 3,98T                         local
data/reference  primarycache all                           default
data/reference  secondarycache all                           default
data/reference  usedbysnapshots 21,6G                         -
data/reference  usedbydataset 2,73T                         -
data/reference  usedbychildren 0                             -
data/reference  usedbyrefreservation 3,98T                         -
data/reference  logbias latency                       default
data/reference  dedup off                           default
data/reference mlslabel                                            -
data/reference  sync standard                      default
data/reference  refcompressratio 1.08x                         -
data/reference  written 1,10M                         -
data/reference  logicalused 2,98T                         -
data/reference  logicalreferenced 2,95T                         -
data/reference  volmode default                       default
data/reference  snapshot_limit none                          default
data/reference  snapshot_count none                          default
data/reference  redundant_metadata all                           default

Could please someone explain why they show as having like half of the 
total pool space as AVAIL ? I thing this is directly related to the fact 
that zpool list shows only 44% of the total pool space is used. And I 
use this value to monitor the pool space usage, looks like I'm totally 
failing with this.

I also don't understand whe the zvol of the size 3.97T really uses 6.74T 
of the space. I found an article, explaing that the volblocksize and the 
sector size has to do something with this, and this happens when the 
device block size is 4k, and volblocksize is default, thus 8k. Mine 
disks sector size is 512 native, so this is really not the case. I'm 
also having equal number of disks in vdevs, and they are 5:

[root at san01:~]# zpool status data
   pool: data
  state: ONLINE
   scan: scrub repaired 111K in 40h42m with 0 errors on Wed Mar  1 
03:34:56 2017

         NAME                          STATE     READ WRITE CKSUM
         data                          ONLINE       0     0     0
           raidz1-0                    ONLINE       0     0     0
             da0                       ONLINE       0     0     0
             da1                       ONLINE       0     0     0
             diskid/DISK-96JS100XTB4V  ONLINE       0     0     0
             diskid/DISK-96JS101TTB4V  ONLINE       0     0     0
             diskid/DISK-96JS100MTB4V  ONLINE       0     0     0
           raidz1-1                    ONLINE       0     0     0
             diskid/DISK-96JS1016TB4V  ONLINE       0     0     0
             diskid/DISK-96JS1002TB4V  ONLINE       0     0     0
             diskid/DISK-96LS1066TB4V  ONLINE       0     0     0
             diskid/DISK-96JS101JTB4V  ONLINE       0     0     0
             diskid/DISK-96LS106TTB4V  ONLINE       0     0     0

errors: No known data errors

So I guess poor vdev alignment is also not the case. I used to resize 
the zvol a couple of times, and I suspect this is the main reason, 
because I've read in another article that this is probably the case when 
"poor alignment" happens.

I will really appreciate if someone will explain this.


More information about the freebsd-fs mailing list