ZFS corruption due to lack of space?

Steven Hartland killing at multiplay.co.uk
Thu Nov 1 02:56:20 UTC 2012


----- Original Message ----- 
From: "Steven Hartland" <killing at multiplay.co.uk>
To: "Peter Jeremy" <peter at rulingia.com>
Cc: <freebsd-fs at freebsd.org>; <freebsd-stable at freebsd.org>
Sent: Thursday, November 01, 2012 12:19 AM
Subject: Re: ZFS corruption due to lack of space?


>
> ----- Original Message ----- 
> From: "Steven Hartland" <killing at multiplay.co.uk>
> To: "Peter Jeremy" <peter at rulingia.com>
> Cc: <freebsd-fs at freebsd.org>; <freebsd-stable at freebsd.org>
> Sent: Thursday, November 01, 2012 12:09 AM
> Subject: Re: ZFS corruption due to lack of space?
>
>
>> On 2012-Oct-31 17:25:09 -0000, Steven Hartland <steven at multiplay.co.uk> wrote:
>>>>Been running some tests on new hardware here to verify all
>>>>is good. One of the tests was to fill the zfs array which
>>>>seems like its totally corrupted the tank.
>>>
>>>I've accidently "filled" a pool, and had multiple processes try to
>>>write to the full pool, without either emptying the free space reserve
>>>(so I could still delete the offending files) or corrupting the pool.
>>
>> Same here but its the first time I've had ZIL in place at the time so
>> wondering if that may be playing a factor.
>>
>>> Had you tried to read/write the raw disks before you tried the
>>> ZFS testing?
>>
>> Yes, didn't see any issues but then it wasn't checksuming so tbh I
>> wouldn't have noticed if it was silently corrupting data.
>>
>>>Do you have compression and/or dedupe enabled on the pool?
>>
>> Nope bog standard raidz2 no additional settings
>>
>>>>1. Given the information it seems like the multiple writes filling
>>>>the disk may have caused metadata corruption?
>>>
>>> I don't recall seeing this reported before.
>>
>> Nore me and we've been using ZFS for years, but never filled a pool
>> with such known simultanious access + ZIL before
>>
>>>>2. Is there anyway to stop the scrub?
>>>
>>>Other than freeing up some space, I don't think so.  If this is a test
>>>pool that you don't need, you could try destroying it and re-creating
>>>it - that may be quicker and easier than recovering the existing pool.
>>
>> Artems trick of cat /dev/null > /tank2/<bigfile> worked and I've now
>> managed to stop the scrub :)
>>
>>>>3. Surely low space should never prevent stopping a scrub?
>>>
>>> As Artem noted, ZFS is a copy-on-write filesystem.  It is supposed to
>>> reserve some free space to allow metadata updates (stop scrubs, delete
>>> files, etc) even when it is "full" but I have seen reports of this not
>>> working correctly in the past.  A truncate-in-place may work.
>>
>> Yes it did thanks, but as you said if this metadata update was failing
>> due to out of space lends credability to the fact that the same lack of
>> space and hence failure to update metadata could have also caused the
>> corruption in the first place.
>>
>> Its interesting to note that the zpool is reporting pleanty of free space
>> even when the root zfs volume was showing 0, so you would expect there
>> to be pleanty of space for it be able to stop the scrub but it appears
>> not which is definitely interesting and could point to the underlying
>> cause?
>>
>> zpool list tank2
>> NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
>> tank2    19T  18.7T   304G    98%  1.00x  ONLINE  -
>>
>> zfs list tank2
>> NAME    USED  AVAIL  REFER  MOUNTPOINT
>> tank2  13.3T      0  13.3T  /tank2
>>
>> Current state is:-
>>  scan: scrub in progress since Wed Oct 31 16:13:53 2012
>>        1.64T scanned out of 18.7T at 62.8M/s, 79h12m to go
>>        280M repaired, 8.76% done
>>
>> Something else that was interesting is while the scrub was running
>> devd was using a good amount of CPU 40% of a 3.3Ghz core, which I've
>> never seen before. Any ideas why its usage would be so high?
>
>
> In case its useful here's the output from a zdb tank2 so far:-
> zdb tank2
>
> Cached configuration:
>        version: 28
>        name: 'tank2'
>        state: 0
>        txg: 39502
>        pool_guid: 15779146362913479443
>        hostid: 1751781486
>        vdev_children: 3
>        vdev_tree:
>            type: 'root'
>            id: 0
>            guid: 15779146362913479443
>            create_txg: 4
>            children[0]:
>                type: 'raidz'
>                id: 0
>                guid: 8518972900227438019
>                nparity: 2
>                metaslab_array: 33
>                metaslab_shift: 37
>                ashift: 9
>                asize: 21004116295680
>                is_log: 0
>                create_txg: 4
>                children[0]:
>                    type: 'disk'
>                    id: 0
>                    guid: 15577380450172137060
>                    path: '/dev/mfisyspd0'
>                    phys_path: '/dev/mfisyspd0'
>                    whole_disk: 1
>                    DTL: 236
>                    create_txg: 4
>                children[1]:
>                    type: 'disk'
>                    id: 1
>                    guid: 16940350228793267704
>                    path: '/dev/mfisyspd1'
>                    phys_path: '/dev/mfisyspd1'
>                    whole_disk: 1
>                    DTL: 235
>                    create_txg: 4
>                children[2]:
>                    type: 'disk'
>                    id: 2
>                    guid: 9264743178245473794
>                    path: '/dev/mfisyspd2'
>                    phys_path: '/dev/mfisyspd2'
>                    whole_disk: 1
>                    DTL: 234
>                    create_txg: 4
>                children[3]:
>                    type: 'disk'
>                    id: 3
>                    guid: 432716341673487166
>                    path: '/dev/mfisyspd3'
>                    phys_path: '/dev/mfisyspd3'
>                    whole_disk: 1
>                    DTL: 233
>                    create_txg: 4
>                children[4]:
>                    type: 'disk'
>                    id: 4
>                    guid: 18217760646550913544
>                    path: '/dev/mfisyspd4'
>                    phys_path: '/dev/mfisyspd4'
>                    whole_disk: 1
>                    DTL: 232
>                    create_txg: 4
>                children[5]:
>                    type: 'disk'
>                    id: 5
>                    guid: 6964614355298004256
>                    path: '/dev/mfisyspd5'
>                    phys_path: '/dev/mfisyspd5'
>                    whole_disk: 1
>                    DTL: 231
>                    create_txg: 4
>                children[6]:
>                    type: 'disk'
>                    id: 6
>                    guid: 4397961270160308034
>                    path: '/dev/mfisyspd6'
>                    phys_path: '/dev/mfisyspd6'
>                    whole_disk: 1
>                    DTL: 230
>                    create_txg: 4
>            children[1]:
>                type: 'disk'
>                id: 1
>                guid: 13757670211012452260
>                path: '/dev/mfisyspd7p3'
>                phys_path: '/dev/mfisyspd7p3'
>                whole_disk: 1
>                metaslab_array: 32
>                metaslab_shift: 27
>                ashift: 9
>                asize: 14125891584
>                is_log: 1
>                DTL: 237
>                create_txg: 4
>            children[2]:
>                type: 'disk'
>                id: 2
>                guid: 7315839509249482920
>                path: '/dev/mfisyspd8p3'
>                phys_path: '/dev/mfisyspd8p3'
>                whole_disk: 1
>                metaslab_array: 31
>                metaslab_shift: 27
>                ashift: 9
>                asize: 14125891584
>                is_log: 1
>                DTL: 229
>                create_txg: 4
>
> MOS Configuration:
>        version: 28
>        name: 'tank2'
>        state: 0
>        txg: 39502
>        pool_guid: 15779146362913479443
>        hostid: 1751781486
>        vdev_children: 3
>        vdev_tree:
>            type: 'root'
>            id: 0
>            guid: 15779146362913479443
>            create_txg: 4
>            children[0]:
>                type: 'raidz'
>                id: 0
>                guid: 8518972900227438019
>                nparity: 2
>                metaslab_array: 33
>                metaslab_shift: 37
>                ashift: 9
>                asize: 21004116295680
>                is_log: 0
>                create_txg: 4
>                children[0]:
>                    type: 'disk'
>                    id: 0
>                    guid: 15577380450172137060
>                    path: '/dev/mfisyspd0'
>                    phys_path: '/dev/mfisyspd0'
>                    whole_disk: 1
>                    DTL: 236
>                    create_txg: 4
>                children[1]:
>                    type: 'disk'
>                    id: 1
>                    guid: 16940350228793267704
>                    path: '/dev/mfisyspd1'
>                    phys_path: '/dev/mfisyspd1'
>                    whole_disk: 1
>                    DTL: 235
>                    create_txg: 4
>                children[2]:
>                    type: 'disk'
>                    id: 2
>                    guid: 9264743178245473794
>                    path: '/dev/mfisyspd2'
>                    phys_path: '/dev/mfisyspd2'
>                    whole_disk: 1
>                    DTL: 234
>                    create_txg: 4
>                children[3]:
>                    type: 'disk'
>                    id: 3
>                    guid: 432716341673487166
>                    path: '/dev/mfisyspd3'
>                    phys_path: '/dev/mfisyspd3'
>                    whole_disk: 1
>                    DTL: 233
>                    create_txg: 4
>                children[4]:
>                    type: 'disk'
>                    id: 4
>                    guid: 18217760646550913544
>                    path: '/dev/mfisyspd4'
>                    phys_path: '/dev/mfisyspd4'
>                    whole_disk: 1
>                    DTL: 232
>                    create_txg: 4
>                children[5]:
>                    type: 'disk'
>                    id: 5
>                    guid: 6964614355298004256
>                    path: '/dev/mfisyspd5'
>                    phys_path: '/dev/mfisyspd5'
>                    whole_disk: 1
>                    DTL: 231
>                    create_txg: 4
>                children[6]:
>                    type: 'disk'
>                    id: 6
>                    guid: 4397961270160308034
>                    path: '/dev/mfisyspd6'
>                    phys_path: '/dev/mfisyspd6'
>                    whole_disk: 1
>                    DTL: 230
>                    create_txg: 4
>            children[1]:
>                type: 'disk'
>                id: 1
>                guid: 13757670211012452260
>                path: '/dev/mfisyspd7p3'
>                phys_path: '/dev/mfisyspd7p3'
>                whole_disk: 1
>                metaslab_array: 32
>                metaslab_shift: 27
>                ashift: 9
>                asize: 14125891584
>                is_log: 1
>                DTL: 237
>                create_txg: 4
>            children[2]:
>                type: 'disk'
>                id: 2
>                guid: 7315839509249482920
>                path: '/dev/mfisyspd8p3'
>                phys_path: '/dev/mfisyspd8p3'
>                whole_disk: 1
>                metaslab_array: 31
>                metaslab_shift: 27
>                ashift: 9
>                asize: 14125891584
>                is_log: 1
>                DTL: 229
>                create_txg: 4
>
> Uberblock:
>        magic = 0000000000bab10c
>        version = 28
>        txg = 39547
>        guid_sum = 6486691012039134504
>        timestamp = 1351727718 UTC = Wed Oct 31 23:55:18 2012
>
> All DDTs are empty
>
> Metaslabs:
>        vdev          0
>        metaslabs   152   offset                spacemap 
>        free      ---------------   -------------------   ---------------   -------------
>        metaslab      0   offset            0   spacemap     36   free    1.41G
>                          segments       4170   maxsize   1.38G   freepct    1%
>        metaslab      1   offset   2000000000   spacemap     65   free    1.45G
>                          segments       4508   maxsize   1.40G   freepct    1%
>        metaslab      2   offset   4000000000   spacemap     70   free    1.44G
>                          segments       5723   maxsize   1.39G   freepct    1%
>        metaslab      3   offset   6000000000   spacemap     77   free    1.41G
>                          segments       6635   maxsize   1.35G   freepct    1%
>        metaslab      4   offset   8000000000   spacemap     80   free    1.46G
>                          segments       6408   maxsize   1.40G   freepct    1%
>        metaslab      5   offset   a000000000   spacemap     81   free    1.47G
>                          segments       6480   maxsize   1.42G   freepct    1%
>        metaslab      6   offset   c000000000   spacemap     82   free    1.48G
>                          segments       6684   maxsize   1.42G   freepct    1%
>        metaslab      7   offset   e000000000   spacemap     83   free    1.48G
>                          segments       6873   maxsize   1.42G   freepct    1%
>        metaslab      8   offset  10000000000   spacemap     84   free    1.48G
>                          segments       7298   maxsize   1.42G   freepct    1%
>        metaslab      9   offset  12000000000   spacemap     85   free    1.43G
>                          segments       6699   maxsize   1.37G   freepct    1%
>        metaslab     10   offset  14000000000   spacemap     88   free    1.50G
>                          segments       6781   maxsize   1.44G   freepct    1%
>        metaslab     11   offset  16000000000   spacemap     89   free    1.50G
>                          segments       6434   maxsize   1.44G   freepct    1%
>        metaslab     12   offset  18000000000   spacemap     90   free    1.51G
>                          segments       7188   maxsize   1.44G   freepct    1%
>        metaslab     13   offset  1a000000000   spacemap     91   free    1.49G
>                          segments       6712   maxsize   1.42G   freepct    1%
>        metaslab     14   offset  1c000000000   spacemap     92   free    1.52G
>                          segments       6810   maxsize   1.46G   freepct    1%
>        metaslab     15   offset  1e000000000   spacemap     93   free    1.52G
>                          segments       8306   maxsize   1.41G   freepct    1%
>        metaslab     16   offset  20000000000   spacemap     94   free    1.49G
>                          segments      21881   maxsize    660M   freepct    1%
>        metaslab     17   offset  22000000000   spacemap     97   free    1.50G
>                          segments       9590   maxsize   1.32G   freepct    1%
>        metaslab     18   offset  24000000000   spacemap     98   free    1.53G
>                          segments       6921   maxsize   1.47G   freepct    1%
>        metaslab     19   offset  26000000000   spacemap     99   free    1.53G
>                          segments       7982   maxsize   1.46G   freepct    1%
>        metaslab     20   offset  28000000000   spacemap    100   free    1.55G
>                          segments       7943   maxsize   1.48G   freepct    1%
>        metaslab     21   offset  2a000000000   spacemap    101   free    1.54G
>                          segments       8049   maxsize   1.47G   freepct    1%
>        metaslab     22   offset  2c000000000   spacemap    102   free    1.54G
>                          segments       8205   maxsize   1.46G   freepct    1%
>        metaslab     23   offset  2e000000000   spacemap    103   free    1.53G
>                          segments      11339   maxsize   1.37G   freepct    1%
>        metaslab     24   offset  30000000000   spacemap    104   free    1.55G
>                          segments      11536   maxsize   1.38G   freepct    1%
>        metaslab     25   offset  32000000000   spacemap    105   free    1.58G
>                          segments       7281   maxsize   1.50G   freepct    1%
>        metaslab     26   offset  34000000000   spacemap    108   free    1.57G
>                          segments       7917   maxsize   1.49G   freepct    1%
>        metaslab     27   offset  36000000000   spacemap    109   free    1.59G
>                          segments       8446   maxsize   1.51G   freepct    1%
>        metaslab     28   offset  38000000000   spacemap    110   free    1.59G
>                          segments       8437   maxsize   1.51G   freepct    1%
>        metaslab     29   offset  3a000000000   spacemap     35   free    1.60G
>                          segments      22991   maxsize   1.21G   freepct    1%
>        metaslab     30   offset  3c000000000   spacemap    111   free    1.57G
>                          segments       8358   maxsize   1.49G   freepct    1%
>        metaslab     31   offset  3e000000000   spacemap    112   free    1.62G
>                          segments       7724   maxsize   1.53G   freepct    1%
>        metaslab     32   offset  40000000000   spacemap    113   free    1.62G
>                          segments       8314   maxsize   1.53G   freepct    1%
>        metaslab     33   offset  42000000000   spacemap    114   free    1.61G
>                          segments       8294   maxsize   1.52G   freepct    1%
>        metaslab     34   offset  44000000000   spacemap    115   free    1.63G
>                          segments       8422   maxsize   1.54G   freepct    1%
>        metaslab     35   offset  46000000000   spacemap    116   free    1.60G
>                          segments       8125   maxsize   1.51G   freepct    1%
>        metaslab     36   offset  48000000000   spacemap    117   free    1.62G
>                          segments       8048   maxsize   1.53G   freepct    1%
>        metaslab     37   offset  4a000000000   spacemap    118   free    1.60G
>                          segments       8648   maxsize   1.51G   freepct    1%
>        metaslab     38   offset  4c000000000   spacemap    119   free    1.66G
>                          segments       8316   maxsize   1.57G   freepct    1%
>        metaslab     39   offset  4e000000000   spacemap     87   free    1.62G
>                          segments      27153   maxsize   1.05G   freepct    1%
>        metaslab     40   offset  50000000000   spacemap    121   free    1.66G
>                          segments       9314   maxsize   1.56G   freepct    1%
>        metaslab     41   offset  52000000000   spacemap    122   free    1.64G
>                          segments       9334   maxsize   1.56G   freepct    1%
>        metaslab     42   offset  54000000000   spacemap    123   free    1.67G
>                          segments      10391   maxsize   1.51G   freepct    1%
>        metaslab     43   offset  56000000000   spacemap    126   free    1.65G
>                          segments      12514   maxsize   1.49G   freepct    1%
>        metaslab     44   offset  58000000000   spacemap    127   free    1.67G
>                          segments      13441   maxsize   1.50G   freepct    1%
>        metaslab     45   offset  5a000000000   spacemap    129   free    1.70G
>                          segments      13288   maxsize   1.54G   freepct    1%
>        metaslab     46   offset  5c000000000   spacemap     96   free    1.71G
>                          segments      27184   maxsize   1.07G   freepct    1%
>        metaslab     47   offset  5e000000000   spacemap    130   free    1.69G
>                          segments      10019   maxsize   1.61G   freepct    1%
>        metaslab     48   offset  60000000000   spacemap    133   free    1.69G
>                          segments      13025   maxsize   1.53G   freepct    1%
>        metaslab     49   offset  62000000000   spacemap    135   free    1.71G
>                          segments      10562   maxsize   1.63G   freepct    1%
>        metaslab     50   offset  64000000000   spacemap    136   free    1.74G
>                          segments       9827   maxsize   1.66G   freepct    1%
>        metaslab     51   offset  66000000000   spacemap    137   free    1.73G
>                          segments      10206   maxsize   1.65G   freepct    1%
>        metaslab     52   offset  68000000000   spacemap    138   free    1.75G
>                          segments       9747   maxsize   1.67G   freepct    1%
>        metaslab     53   offset  6a000000000   spacemap    139   free    1.76G
>                          segments      14248   maxsize   1.57G   freepct    1%
>        metaslab     54   offset  6c000000000   spacemap    107   free    1.76G
>                          segments      29803   maxsize    987M   freepct    1%
>        metaslab     55   offset  6e000000000   spacemap    142   free    1.76G
>                          segments       9068   maxsize   1.68G   freepct    1%
>        metaslab     56   offset  70000000000   spacemap    143   free    1.76G
>                          segments      10561   maxsize   1.68G   freepct    1%
>        metaslab     57   offset  72000000000   spacemap    144   free    1.78G
>                          segments      10234   maxsize   1.70G   freepct    1%
>        metaslab     58   offset  74000000000   spacemap     34   free    1.79G
>                          segments      12737   maxsize   1.49G   freepct    1%
>        metaslab     59   offset  76000000000   spacemap    145   free    1.80G
>                          segments      10211   maxsize   1.71G   freepct    1%
>        metaslab     60   offset  78000000000   spacemap    146   free    1.77G
>                          segments      10696   maxsize   1.68G   freepct    1%
>        metaslab     61   offset  7a000000000   spacemap    147   free    1.81G
>                          segments      10934   maxsize   1.71G   freepct    1%
>        metaslab     62   offset  7c000000000   spacemap    148   free    1.81G
>                          segments       8698   maxsize   1.73G   freepct    1%
>        metaslab     63   offset  7e000000000   spacemap    149   free    1.82G
>                          segments       9165   maxsize   1.74G   freepct    1%
>        metaslab     64   offset  80000000000   spacemap    152   free    1.83G
>                          segments       9388   maxsize   1.74G   freepct    1%
>        metaslab     65   offset  82000000000   spacemap    154   free    1.84G
>                          segments      11321   maxsize   1.74G   freepct    1%
>        metaslab     66   offset  84000000000   spacemap    155   free    1.85G
>                          segments      10040   maxsize   1.76G   freepct    1%
>        metaslab     67   offset  86000000000   spacemap    156   free    1.86G
>                          segments      10531   maxsize   1.77G   freepct    1%
>        metaslab     68   offset  88000000000   spacemap     86   free    1.84G
>                          segments       8518   maxsize   1.73G   freepct    1%
>        metaslab     69   offset  8a000000000   spacemap    120   free    1.87G
>                          segments      18100   maxsize   1.51G   freepct    1%
>        metaslab     70   offset  8c000000000   spacemap    157   free    1.88G
>                          segments      12773   maxsize   1.70G   freepct    1%
>        metaslab     71   offset  8e000000000   spacemap    159   free    1.89G
>                          segments      11443   maxsize   1.79G   freepct    1%
>        metaslab     72   offset  90000000000   spacemap    125   free    1.90G
>                          segments      13633   maxsize   1.72G   freepct    1%
>        metaslab     73   offset  92000000000   spacemap    160   free    1.91G
>                          segments      10724   maxsize   1.81G   freepct    1%
>        metaslab     74   offset  94000000000   spacemap    161   free    1.92G
>                          segments      10550   maxsize   1.77G   freepct    1%
>        metaslab     75   offset  96000000000   spacemap    162   free    1.92G
>                          segments      10027   maxsize   1.83G   freepct    1%
>        metaslab     76   offset  98000000000   spacemap    132   free    1.93G
>                          segments      16007   maxsize   1.69G   freepct    1%
>        metaslab     77   offset  9a000000000   spacemap    164   free    1.94G
>                          segments       9721   maxsize   1.84G   freepct    1%
>        metaslab     78   offset  9c000000000   spacemap    134   free    1.95G
>                          segments      26262   maxsize   1.35G   freepct    1%
>        metaslab     79   offset  9e000000000   spacemap    165   free    1.95G
>                          segments       7968   maxsize   1.88G   freepct    1%
>        metaslab     80   offset  a0000000000   spacemap    166   free    1.97G
>                          segments       7757   maxsize   1.89G   freepct    1%
>        metaslab     81   offset  a2000000000   spacemap    167   free    1.97G
>                          segments       9206   maxsize   1.89G   freepct    1%
>        metaslab     82   offset  a4000000000   spacemap    168   free    1.98G
>                          segments       9225   maxsize   1.89G   freepct    1%
>        metaslab     83   offset  a6000000000   spacemap    106   free    1.99G
>                          segments      24197   maxsize   1.43G   freepct    1%
>        metaslab     84   offset  a8000000000   spacemap    169   free    2.00G
>                          segments       9637   maxsize   1.91G   freepct    1%
>        metaslab     85   offset  aa000000000   spacemap    170   free    2.01G
>                          segments      10167   maxsize   1.92G   freepct    1%
>        metaslab     86   offset  ac000000000   spacemap    171   free    2.02G
>                          segments      12180   maxsize   1.85G   freepct    1%
>        metaslab     87   offset  ae000000000   spacemap    174   free    2.02G
>                          segments       9716   maxsize   1.93G   freepct    1%
>        metaslab     88   offset  b0000000000   spacemap    175   free    2.04G
>                          segments      10583   maxsize   1.94G   freepct    1%
>        metaslab     89   offset  b2000000000   spacemap    176   free    2.05G
>                          segments       9935   maxsize   1.95G   freepct    1%
>        metaslab     90   offset  b4000000000   spacemap    177   free    2.06G
>                          segments      10459   maxsize   1.96G   freepct    1%
>        metaslab     91   offset  b6000000000   spacemap    178   free    2.07G
>                          segments       9396   maxsize   1.98G   freepct    1%
>        metaslab     92   offset  b8000000000   spacemap    179   free    2.07G
>                          segments       8301   maxsize   1.96G   freepct    1%
>        metaslab     93   offset  ba000000000   spacemap    151   free    2.08G
>                          segments      17800   maxsize   1.52G   freepct    1%
>        metaslab     94   offset  bc000000000   spacemap    181   free    2.10G
>                          segments      10951   maxsize   2.00G   freepct    1%
>        metaslab     95   offset  be000000000   spacemap    182   free    2.11G
>                          segments      11002   maxsize   2.01G   freepct    1%
>        metaslab     96   offset  c0000000000   spacemap    183   free    2.12G
>                          segments      10855   maxsize   2.01G   freepct    1%
>        metaslab     97   offset  c2000000000   spacemap     95   free    2.13G
>                          segments      13168   maxsize   1.96G   freepct    1%
>        metaslab     98   offset  c4000000000   spacemap    184   free    2.13G
>                          segments       9408   maxsize   2.04G   freepct    1%
>        metaslab     99   offset  c6000000000   spacemap    185   free    2.15G
>                          segments      10694   maxsize   2.04G   freepct    1%
>        metaslab    100   offset  c8000000000   spacemap    186   free    2.16G
>                          segments      10563   maxsize   2.05G   freepct    1%
>        metaslab    101   offset  ca000000000   spacemap    187   free    2.17G
>                          segments      11059   maxsize   2.06G   freepct    1%
>        metaslab    102   offset  cc000000000   spacemap    188   free    2.18G
>                          segments      11516   maxsize   2.07G   freepct    1%
>        metaslab    103   offset  ce000000000   spacemap    189   free    2.19G
>                          segments      10700   maxsize   2.08G   freepct    1%
>        metaslab    104   offset  d0000000000   spacemap    163   free    2.20G
>                          segments       7869   maxsize   1.77G   freepct    1%
>        metaslab    105   offset  d2000000000   spacemap    131   free    2.22G
>                          segments      11941   maxsize   2.08G   freepct    1%
>        metaslab    106   offset  d4000000000   spacemap    190   free    9.24G
>                          segments      11407   maxsize   2.11G   freepct    7%
>        metaslab    107   offset  d6000000000   spacemap    141   free    2.24G
>                          segments      13038   maxsize   2.00G   freepct    1%
>        metaslab    108   offset  d8000000000   spacemap    192   free     9.7G
>                          segments      10472   maxsize   2.13G   freepct    7%
>        metaslab    109   offset  da000000000   spacemap    193   free    8.61G
>                          segments      11389   maxsize   2.14G   freepct    6%
>        metaslab    110   offset  dc000000000   spacemap    194   free    6.34G
>                          segments      10513   maxsize   2.16G   freepct    4%
>        metaslab    111   offset  de000000000   spacemap    195   free    7.58G
>                          segments      10168   maxsize   2.17G   freepct    5%
>        metaslab    112   offset  e0000000000   spacemap    197   free    7.14G
>                          segments       6922   maxsize   2.22G   freepct    5%
>        metaslab    113   offset  e2000000000   spacemap    198   free    2.30G
>                          segments       6757   maxsize   2.24G   freepct    1%
>        metaslab    114   offset  e4000000000   spacemap    199   free    2.32G
>                          segments       7008   maxsize   2.26G   freepct    1%
>        metaslab    115   offset  e6000000000   spacemap    200   free    2.33G
>                          segments       6518   maxsize   2.27G   freepct    1%
>        metaslab    116   offset  e8000000000   spacemap    173   free    2.35G
>                          segments      13857   maxsize   1.92G   freepct    1%
>        metaslab    117   offset  ea000000000   spacemap    201   free    2.36G
>                          segments       7124   maxsize   2.30G   freepct    1%
>        metaslab    118   offset  ec000000000   spacemap    202   free    2.37G
>                          segments       6936   maxsize   2.31G   freepct    1%
>        metaslab    119   offset  ee000000000   spacemap    203   free    2.38G
>                          segments       6679   maxsize   2.32G   freepct    1%
>        metaslab    120   offset  f0000000000   spacemap    204   free    2.39G
>                          segments       6818   maxsize   2.34G   freepct    1%
>        metaslab    121   offset  f2000000000   spacemap    205   free    2.41G
>                          segments       7423   maxsize   2.34G   freepct    1%
>        metaslab    122   offset  f4000000000   spacemap    150   free    2.42G
>                          segments      15678   maxsize   2.14G   freepct    1%
>        metaslab    123   offset  f6000000000   spacemap    158   free    2.43G
>                          segments       9980   maxsize   2.28G   freepct    1%
>        metaslab    124   offset  f8000000000   spacemap    180   free    2.45G
>                          segments      11702   maxsize   1.71G   freepct    1%
>        metaslab    125   offset  fa000000000   spacemap    206   free    2.46G
>                          segments       7070   maxsize   2.40G   freepct    1%
>        metaslab    126   offset  fc000000000   spacemap    124   free    2.47G
>                          segments      11485   maxsize   2.31G   freepct    1%
>        metaslab    127   offset  fe000000000   spacemap    128   free    2.49G
>                          segments       2051   maxsize   2.42G   freepct    1%
>        metaslab    128   offset 100000000000   spacemap    207   free    2.50G
>                          segments       7309   maxsize   2.44G   freepct    1%
>        metaslab    129   offset 102000000000   spacemap    208   free    2.52G
>                          segments       7151   maxsize   2.46G   freepct    1%
>        metaslab    130   offset 104000000000   spacemap    209   free    2.52G
>                          segments       6041   maxsize   2.47G   freepct    1%
>        metaslab    131   offset 106000000000   spacemap    210   free    2.54G
>                          segments       6910   maxsize   2.47G   freepct    1%
>        metaslab    132   offset 108000000000   spacemap    211   free    2.56G
>                          segments       6816   maxsize   2.50G   freepct    1%
>        metaslab    133   offset 10a000000000   spacemap    212   free    2.57G
>                          segments       6182   maxsize   2.51G   freepct    2%
>        metaslab    134   offset 10c000000000   spacemap    213   free    2.59G
>                          segments       7541   maxsize   2.52G   freepct    2%
>        metaslab    135   offset 10e000000000   spacemap    214   free    2.61G
>                          segments       7810   maxsize   2.53G   freepct    2%
>        metaslab    136   offset 110000000000   spacemap    215   free    2.62G
>                          segments       6822   maxsize   2.56G   freepct    2%
>        metaslab    137   offset 112000000000   spacemap    191   free    9.39G
>                          segments      10891   maxsize   2.28G   freepct    7%
>        metaslab    138   offset 114000000000   spacemap    216   free    2.65G
>                          segments       7295   maxsize   2.58G   freepct    2%
>        metaslab    139   offset 116000000000   spacemap    217   free    2.67G
>                          segments       7435   maxsize   2.60G   freepct    2%
>        metaslab    140   offset 118000000000   spacemap    218   free    2.68G
>                          segments       6952   maxsize   2.62G   freepct    2%
>        metaslab    141   offset 11a000000000   spacemap    196   free    2.70G
>                          segments       5975   maxsize   2.37G   freepct    2%
>        metaslab    142   offset 11c000000000   spacemap    219   free    2.72G
>                          segments       7547   maxsize   2.65G   freepct    2%
>        metaslab    143   offset 11e000000000   spacemap    220   free    2.74G
>                          segments       7570   maxsize   2.67G   freepct    2%
>        metaslab    144   offset 120000000000   spacemap    221   free    2.75G
>                          segments       7321   maxsize   2.69G   freepct    2%
>        metaslab    145   offset 122000000000   spacemap    172   free    2.77G
>                          segments       3490   maxsize   2.68G   freepct    2%
>        metaslab    146   offset 124000000000   spacemap    222   free    2.79G
>                          segments       7448   maxsize   2.72G   freepct    2%
>        metaslab    147   offset 126000000000   spacemap    223   free    2.81G
>                          segments       7309   maxsize   2.74G   freepct    2%
>        metaslab    148   offset 128000000000   spacemap    224   free    2.82G
>                          segments       7367   maxsize   2.75G   freepct    2%
>        metaslab    149   offset 12a000000000   spacemap    225   free    2.84G
>                          segments       7249   maxsize   2.77G   freepct    2%
>        metaslab    150   offset 12c000000000   spacemap    226   free    2.86G
>                          segments       8091   maxsize   2.78G   freepct    2%
>        metaslab    151   offset 12e000000000   spacemap    153   free    2.88G
>                          segments       2771   maxsize   2.74G   freepct    2%
>
>        vdev          1
>        metaslabs   105   offset                spacemap 
>        free      ---------------   -------------------   ---------------   -------------
>        metaslab      0   offset            0   spacemap     38   free     128M
>                          segments          1   maxsize    128M   freepct   99%
>        metaslab      1   offset      8000000   spacemap     39   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      2   offset     10000000   spacemap     42   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      3   offset     18000000   spacemap     41   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      4   offset     20000000   spacemap     51   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      5   offset     28000000   spacemap     50   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      6   offset     30000000   spacemap     49   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      7   offset     38000000   spacemap     48   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      8   offset     40000000   spacemap     47   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      9   offset     48000000   spacemap     46   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     10   offset     50000000   spacemap     45   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     11   offset     58000000   spacemap     59   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     12   offset     60000000   spacemap     62   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     13   offset     68000000   spacemap     61   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     14   offset     70000000   spacemap     67   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     15   offset     78000000   spacemap     66   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     16   offset     80000000   spacemap     74   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     17   offset     88000000   spacemap     73   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     18   offset     90000000   spacemap     75   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     19   offset     98000000   spacemap     78   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     20   offset     a0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     21   offset     a8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     22   offset     b0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     23   offset     b8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     24   offset     c0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     25   offset     c8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     26   offset     d0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     27   offset     d8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     28   offset     e0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     29   offset     e8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     30   offset     f0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     31   offset     f8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     32   offset    100000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     33   offset    108000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     34   offset    110000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     35   offset    118000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     36   offset    120000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     37   offset    128000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     38   offset    130000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     39   offset    138000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     40   offset    140000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     41   offset    148000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     42   offset    150000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     43   offset    158000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     44   offset    160000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     45   offset    168000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     46   offset    170000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     47   offset    178000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     48   offset    180000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     49   offset    188000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     50   offset    190000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     51   offset    198000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     52   offset    1a0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     53   offset    1a8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     54   offset    1b0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     55   offset    1b8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     56   offset    1c0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     57   offset    1c8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     58   offset    1d0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     59   offset    1d8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     60   offset    1e0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     61   offset    1e8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     62   offset    1f0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     63   offset    1f8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     64   offset    200000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     65   offset    208000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     66   offset    210000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     67   offset    218000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     68   offset    220000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     69   offset    228000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     70   offset    230000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     71   offset    238000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     72   offset    240000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     73   offset    248000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     74   offset    250000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     75   offset    258000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     76   offset    260000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     77   offset    268000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     78   offset    270000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     79   offset    278000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     80   offset    280000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     81   offset    288000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     82   offset    290000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     83   offset    298000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     84   offset    2a0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     85   offset    2a8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     86   offset    2b0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     87   offset    2b8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     88   offset    2c0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     89   offset    2c8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     90   offset    2d0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     91   offset    2d8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     92   offset    2e0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     93   offset    2e8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     94   offset    2f0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     95   offset    2f8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     96   offset    300000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     97   offset    308000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     98   offset    310000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     99   offset    318000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    100   offset    320000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    101   offset    328000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    102   offset    330000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    103   offset    338000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    104   offset    340000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>
>        vdev          2
>        metaslabs   105   offset                spacemap 
>        free      ---------------   -------------------   ---------------   -------------
>        metaslab      0   offset            0   spacemap     37   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      1   offset      8000000   spacemap     40   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      2   offset     10000000   spacemap     44   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      3   offset     18000000   spacemap     43   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      4   offset     20000000   spacemap     58   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      5   offset     28000000   spacemap     57   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      6   offset     30000000   spacemap     56   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      7   offset     38000000   spacemap     55   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      8   offset     40000000   spacemap     54   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab      9   offset     48000000   spacemap     53   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     10   offset     50000000   spacemap     52   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     11   offset     58000000   spacemap     60   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     12   offset     60000000   spacemap     64   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     13   offset     68000000   spacemap     63   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     14   offset     70000000   spacemap     69   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     15   offset     78000000   spacemap     68   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     16   offset     80000000   spacemap     72   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     17   offset     88000000   spacemap     71   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     18   offset     90000000   spacemap     76   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     19   offset     98000000   spacemap     79   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     20   offset     a0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     21   offset     a8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     22   offset     b0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     23   offset     b8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     24   offset     c0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     25   offset     c8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     26   offset     d0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     27   offset     d8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     28   offset     e0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     29   offset     e8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     30   offset     f0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     31   offset     f8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     32   offset    100000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     33   offset    108000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     34   offset    110000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     35   offset    118000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     36   offset    120000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     37   offset    128000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     38   offset    130000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     39   offset    138000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     40   offset    140000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     41   offset    148000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     42   offset    150000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     43   offset    158000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     44   offset    160000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     45   offset    168000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     46   offset    170000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     47   offset    178000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     48   offset    180000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     49   offset    188000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     50   offset    190000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     51   offset    198000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     52   offset    1a0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     53   offset    1a8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     54   offset    1b0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     55   offset    1b8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     56   offset    1c0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     57   offset    1c8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     58   offset    1d0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     59   offset    1d8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     60   offset    1e0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     61   offset    1e8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     62   offset    1f0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     63   offset    1f8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     64   offset    200000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     65   offset    208000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     66   offset    210000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     67   offset    218000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     68   offset    220000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     69   offset    228000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     70   offset    230000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     71   offset    238000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     72   offset    240000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     73   offset    248000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     74   offset    250000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     75   offset    258000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     76   offset    260000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     77   offset    268000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     78   offset    270000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     79   offset    278000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     80   offset    280000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     81   offset    288000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     82   offset    290000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     83   offset    298000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     84   offset    2a0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     85   offset    2a8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     86   offset    2b0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     87   offset    2b8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     88   offset    2c0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     89   offset    2c8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     90   offset    2d0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     91   offset    2d8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     92   offset    2e0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     93   offset    2e8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     94   offset    2f0000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     95   offset    2f8000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     96   offset    300000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     97   offset    308000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     98   offset    310000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab     99   offset    318000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    100   offset    320000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    101   offset    328000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    102   offset    330000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    103   offset    338000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>        metaslab    104   offset    340000000   spacemap      0   free     128M
>                          segments          1   maxsize    128M   freepct  100%
>
> Dataset mos [META], ID 0, cr_txg 4, 459M, 235 objects
>
>    Object  lvl   iblk   dblk  dsize  lsize   %full  type
>         0    2    16K    16K   256K   128K   91.80  DMU dnode
>         1    1    16K    16K  19.0K    32K  100.00  object directory
>         2    1    16K    512      0    512    0.00  DSL directory
>         3    1    16K    512  3.00K    512  100.00  DSL props
>         4    1    16K    512  3.00K    512  100.00  DSL directory child map
>         5    1    16K    512      0    512    0.00  DSL directory
>         6    1    16K    512  3.00K    512  100.00  DSL props
>         7    1    16K    512  3.00K    512  100.00  DSL directory child map
>         8    1    16K    512      0    512    0.00  DSL directory
>         9    1    16K    512  3.00K    512  100.00  DSL props
>        10    1    16K    512  3.00K    512  100.00  DSL directory child map
>        11    1    16K   128K      0   128K    0.00  bpobj
>        12    1    16K    512      0    512    0.00  DSL directory
>        13    1    16K    512  3.00K    512  100.00  DSL props
>        14    1    16K    512  3.00K    512  100.00  DSL directory child map
>        15    1    16K    512      0    512    0.00  DSL dataset
>        16    1    16K    512  3.00K    512  100.00  DSL dataset snap map
>        17    1    16K    512  3.00K    512  100.00  DSL deadlist map
>        18    1    16K    512      0    512    0.00  DSL dataset
>        19    1    16K    512  3.00K    512  100.00  DSL deadlist map
>        20    1    16K   128K      0   128K    0.00  bpobj
>        21    1    16K    512      0    512    0.00  DSL dataset
>        22    1    16K    512  3.00K    512  100.00  DSL dataset snap map
>        23    1    16K    512  3.00K    512  100.00  DSL deadlist map
>        24    1    16K   128K      0   128K    0.00  bpobj
>        25    1    16K    512  3.00K    512  100.00  DSL dataset next clones
>        26    1    16K    512  3.00K    512  100.00  DSL dir clones
>        27    1    16K    16K  51.0K    16K  100.00  packed nvlist
>        28    1    16K    16K  51.0K    16K  100.00  bpobj (Z=uncompressed)
>        29    1    16K   128K    16K   128K  100.00  SPA history
>        30    1    16K    16K  6.50K    16K  100.00  packed nvlist
>        31    1    16K    512  3.00K    512  100.00  object array
>        32    1    16K    512  3.00K    512  100.00  object array
>        33    1    16K    512   9.5K  1.50K  100.00  object array
>        34    2    16K     4K   467K   176K  100.00  SPA space map
>        35    2    16K     4K   851K   296K  100.00  SPA space map
>        36    2    16K     4K   275K  88.0K  100.00  SPA space map
>        37    1    16K     4K      0     4K    0.00  SPA space map
>        38    1    16K     4K  3.00K     4K  100.00  SPA space map
>        39    1    16K     4K      0     4K    0.00  SPA space map
>        40    1    16K     4K      0     4K    0.00  SPA space map
>        41    1    16K     4K      0     4K    0.00  SPA space map
>        42    1    16K     4K      0     4K    0.00  SPA space map
>        43    1    16K     4K      0     4K    0.00  SPA space map
>        44    1    16K     4K      0     4K    0.00  SPA space map
>        45    1    16K     4K      0     4K    0.00  SPA space map
>        46    1    16K     4K      0     4K    0.00  SPA space map
>        47    1    16K     4K  3.00K     4K  100.00  SPA space map
>        48    1    16K     4K      0     4K    0.00  SPA space map
>        49    1    16K     4K      0     4K    0.00  SPA space map
>        50    1    16K     4K      0     4K    0.00  SPA space map
>        51    1    16K     4K      0     4K    0.00  SPA space map
>        52    1    16K     4K      0     4K    0.00  SPA space map
>        53    1    16K     4K      0     4K    0.00  SPA space map
>        54    1    16K     4K  3.00K     4K  100.00  SPA space map
>        55    1    16K     4K      0     4K    0.00  SPA space map
>        56    1    16K     4K      0     4K    0.00  SPA space map
>        57    1    16K     4K      0     4K    0.00  SPA space map
>        58    1    16K     4K      0     4K    0.00  SPA space map
>        59    1    16K     4K      0     4K    0.00  SPA space map
>        60    1    16K     4K      0     4K    0.00  SPA space map
>        61    1    16K     4K      0     4K    0.00  SPA space map
>        62    1    16K     4K      0     4K    0.00  SPA space map
>        63    1    16K     4K      0     4K    0.00  SPA space map
>        64    1    16K     4K      0     4K    0.00  SPA space map
>        65    2    16K     4K   291K  92.0K  100.00  SPA space map
>        66    1    16K     4K      0     4K    0.00  SPA space map
>        67    1    16K     4K      0     4K    0.00  SPA space map
>        68    1    16K     4K      0     4K    0.00  SPA space map
>        69    1    16K     4K      0     4K    0.00  SPA space map
>        70    2    16K     4K   311K   100K  100.00  SPA space map
>        71    1    16K     4K      0     4K    0.00  SPA space map
>        72    1    16K     4K      0     4K    0.00  SPA space map
>        73    1    16K     4K      0     4K    0.00  SPA space map
>        74    1    16K     4K      0     4K    0.00  SPA space map
>        75    1    16K     4K      0     4K    0.00  SPA space map
>        76    1    16K     4K      0     4K    0.00  SPA space map
>        77    2    16K     4K   336K   104K  100.00  SPA space map
>        78    1    16K     4K      0     4K    0.00  SPA space map
>        79    1    16K     4K      0     4K    0.00  SPA space map
>        80    2    16K     4K   330K   104K  100.00  SPA space map
>        81    2    16K     4K   333K   104K  100.00  SPA space map
>        82    2    16K     4K   339K   108K  100.00  SPA space map
>        83    2    16K     4K   343K   108K  100.00  SPA space map
>        84    2    16K     4K   368K   116K  100.00  SPA space map
>        85    2    16K     4K   339K   108K  100.00  SPA space map
>        86    2    16K     4K   384K   120K  100.00  SPA space map
>        87    2    16K     4K  1.15M   368K  100.00  SPA space map
>        88    2    16K     4K   346K   108K  100.00  SPA space map
>        89    2    16K     4K   339K   108K  100.00  SPA space map
>        90    2    16K     4K   349K   108K  100.00  SPA space map
>        91    2    16K     4K   339K   108K  100.00  SPA space map
>        92    2    16K     4K   365K   116K  100.00  SPA space map
>        93    2    16K     4K   391K   120K  100.00  SPA space map
>        94    2    16K     4K   867K   268K  100.00  SPA space map
>        95    2    16K     4K   506K   156K  100.00  SPA space map
>        96    2    16K     4K   858K   268K  100.00  SPA space map
>        97    2    16K     4K   490K   152K  100.00  SPA space map
>        98    2    16K     4K   346K   112K  100.00  SPA space map
>        99    2    16K     4K   371K   116K  100.00  SPA space map
>       100    2    16K     4K   362K   112K  100.00  SPA space map
>       101    2    16K     4K   384K   120K  100.00  SPA space map
>       102    2    16K     4K   400K   124K  100.00  SPA space map
>       103    2    16K     4K   522K   160K  100.00  SPA space map
>       104    2    16K     4K   477K   148K  100.00  SPA space map
>       105    2    16K     4K   346K   112K  100.00  SPA space map
>       106    2    16K     4K   774K   240K  100.00  SPA space map
>       107    2    16K     4K   906K   312K  100.00  SPA space map
>       108    2    16K     4K   359K   112K  100.00  SPA space map
>       109    2    16K     4K   387K   120K  100.00  SPA space map
>       110    2    16K     4K   387K   120K  100.00  SPA space map
>       111    2    16K     4K   413K   128K  100.00  SPA space map
>       112    2    16K     4K   375K   120K  100.00  SPA space map
>       113    2    16K     4K   378K   116K  100.00  SPA space map
>       114    2    16K     4K   410K   128K  100.00  SPA space map
>       115    2    16K     4K   400K   124K  100.00  SPA space map
>       116    2    16K     4K   381K   116K  100.00  SPA space map
>       117    2    16K     4K   387K   120K  100.00  SPA space map
>       118    2    16K     4K   397K   124K  100.00  SPA space map
>       119    2    16K     4K   394K   120K  100.00  SPA space map
>       120    2    16K     4K   826K   256K  100.00  SPA space map
>       121    2    16K     4K   407K   124K  100.00  SPA space map
>       122    2    16K     4K   429K   132K  100.00  SPA space map
>       123    2    16K     4K   451K   140K  100.00  SPA space map
>       124    2    16K     4K   461K   144K  100.00  SPA space map
>       125    2    16K     4K   519K   160K  100.00  SPA space map
>       126    2    16K     4K   477K   148K  100.00  SPA space map
>       127    2    16K     4K   605K   188K  100.00  SPA space map
>       128    2    16K     4K   227K   100K  100.00  SPA space map
>       129    2    16K     4K   528K   164K  100.00  SPA space map
>       130    2    16K     4K   426K   132K  100.00  SPA space map
>       131    2    16K     4K   528K   164K  100.00  SPA space map
>       132    2    16K     4K   589K   184K  100.00  SPA space map
>       133    2    16K     4K   605K   188K  100.00  SPA space map
>       134    2    16K     4K   790K   276K  100.00  SPA space map
>       135    2    16K     4K   439K   136K  100.00  SPA space map
>       136    2    16K     4K   423K   132K  100.00  SPA space map
>       137    2    16K     4K   455K   140K  100.00  SPA space map
>       138    2    16K     4K   410K   128K  100.00  SPA space map
>       139    2    16K     4K   528K   164K  100.00  SPA space map
>       141    2    16K     4K   531K   164K  100.00  SPA space map
>       142    2    16K     4K   397K   124K  100.00  SPA space map
>       143    2    16K     4K   439K   136K  100.00  SPA space map
>       144    2    16K     4K   423K   132K  100.00  SPA space map
>       145    2    16K     4K   419K   132K  100.00  SPA space map
>       146    2    16K     4K   477K   148K  100.00  SPA space map
>       147    2    16K     4K   461K   144K  100.00  SPA space map
>       148    2    16K     4K   384K   120K  100.00  SPA space map
>       149    2    16K     4K   403K   124K  100.00  SPA space map
>       150    2    16K     4K   560K   172K  100.00  SPA space map
>       151    2    16K     4K   586K   212K  100.00  SPA space map
>       152    2    16K     4K   407K   128K  100.00  SPA space map
>       153    2    16K     4K   183K  88.0K  100.00  SPA space map
>       154    2    16K     4K   506K   156K  100.00  SPA space map
>       155    2    16K     4K   506K   156K  100.00  SPA space map
>       156    2    16K     4K   432K   136K  100.00  SPA space map
>       157    2    16K     4K   499K   156K  100.00  SPA space map
>       158    2    16K     4K   442K   140K  100.00  SPA space map
>       159    2    16K     4K   448K   140K  100.00  SPA space map
>       160    2    16K     4K   435K   136K  100.00  SPA space map
>       161    2    16K     4K   471K   148K  100.00  SPA space map
>       162    2    16K     4K   435K   140K  100.00  SPA space map
>       163    2    16K     4K   343K   136K  100.00  SPA space map
>       164    2    16K     4K   426K   132K  100.00  SPA space map
>       165    2    16K     4K   365K   124K  100.00  SPA space map
>       166    2    16K     4K   371K   120K  100.00  SPA space map
>       167    2    16K     4K   416K   136K  100.00  SPA space map
>       168    2    16K     4K   426K   132K  100.00  SPA space map
>       169    2    16K     4K   419K   132K  100.00  SPA space map
>       170    2    16K     4K   461K   144K  100.00  SPA space map
>       171    2    16K     4K   541K   156K  100.00  SPA space map
>       172    2    16K     4K   237K   100K  100.00  SPA space map
>       173    2    16K     4K   531K   176K  100.00  SPA space map
>       174    2    16K     4K   426K   136K  100.00  SPA space map
>       175    2    16K     4K   445K   140K  100.00  SPA space map
>       176    2    16K     4K   426K   136K  100.00  SPA space map
>       177    2    16K     4K   451K   140K  100.00  SPA space map
>       178    2    16K     4K   423K   136K  100.00  SPA space map
>       179    2    16K     4K   413K   136K  100.00  SPA space map
>       180    2    16K     4K   439K   164K  100.00  SPA space map
>       181    2    16K     4K   471K   148K  100.00  SPA space map
>       182    2    16K     4K   458K   144K  100.00  SPA space map
>       183    2    16K     4K   471K   148K  100.00  SPA space map
>       184    2    16K     4K   419K   132K  100.00  SPA space map
>       185    2    16K     4K   490K   152K  100.00  SPA space map
>       186    2    16K     4K   445K   140K  100.00  SPA space map
>       187    2    16K     4K   464K   144K  100.00  SPA space map
>       188    2    16K     4K   467K   148K  100.00  SPA space map
>       189    2    16K     4K   451K   140K  100.00  SPA space map
>       190    2    16K     4K   522K   164K  100.00  SPA space map
>       191    2    16K     4K   503K   144K  100.00  SPA space map
>       192    2    16K     4K   544K   156K  100.00  SPA space map
>       193    2    16K     4K   535K   168K  100.00  SPA space map
>       194    2    16K     4K   487K   152K  100.00  SPA space map
>       195    2    16K     4K   429K   136K  100.00  SPA space map
>       196    2    16K     4K   291K   120K  100.00  SPA space map
>       197    2    16K     4K   339K   112K  100.00  SPA space map
>       198    2    16K     4K   359K   124K  100.00  SPA space map
>       199    2    16K     4K   311K   120K  100.00  SPA space map
>       200    2    16K     4K   314K   116K  100.00  SPA space map
>       201    2    16K     4K   336K   128K  100.00  SPA space map
>       202    2    16K     4K   317K   120K  100.00  SPA space map
>       203    2    16K     4K   327K   120K  100.00  SPA space map
>       204    2    16K     4K   346K   124K  100.00  SPA space map
>       205    2    16K     4K   323K   124K  100.00  SPA space map
>       206    2    16K     4K   339K   116K  100.00  SPA space map
>       207    2    16K     4K   349K   124K  100.00  SPA space map
>       208    2    16K     4K   333K   124K  100.00  SPA space map
>       209    2    16K     4K   291K   112K  100.00  SPA space map
>       210    2    16K     4K   336K   120K  100.00  SPA space map
>       211    2    16K     4K   320K   120K  100.00  SPA space map
>       212    2    16K     4K   291K   112K  100.00  SPA space map
>       213    2    16K     4K   349K   128K  100.00  SPA space map
>       214    2    16K     4K   397K   132K  100.00  SPA space map
>       215    2    16K     4K   330K   116K  100.00  SPA space map
>       216    2    16K     4K   320K   120K  100.00  SPA space map
>       217    2    16K     4K   394K   132K  100.00  SPA space map
>       218    2    16K     4K   336K   120K  100.00  SPA space map
>       219    2    16K     4K   327K   124K  100.00  SPA space map
>       220    2    16K     4K   349K   128K  100.00  SPA space map
>       221    2    16K     4K   391K   128K  100.00  SPA space map
>       222    2    16K     4K   339K   124K  100.00  SPA space map
>       223    2    16K     4K   339K   124K  100.00  SPA space map
>       224    2    16K     4K   365K   124K  100.00  SPA space map
>       225    2    16K     4K   346K   124K  100.00  SPA space map
>       226    2    16K     4K   400K   136K  100.00  SPA space map
>       228    3    16K    16K   395M   433M   99.95  persistent error log
>       229    1    16K     4K      0     4K    0.00  SPA space map
>       230    1    16K     4K      0     4K    0.00  SPA space map
>       231    1    16K     4K      0     4K    0.00  SPA space map
>       232    1    16K     4K      0     4K    0.00  SPA space map
>       233    1    16K     4K      0     4K    0.00  SPA space map
>       234    1    16K     4K      0     4K    0.00  SPA space map
>       235    1    16K     4K      0     4K    0.00  SPA space map
>       236    1    16K     4K      0     4K    0.00  SPA space map
>       237    1    16K     4K      0     4K    0.00  SPA space map
>
> Dataset tank2 [ZPL], ID 21, cr_txg 1, 13.3T, 37 objects
>
>    ZIL header: claim_txg 0, claim_blk_seq 0, claim_lr_seq 0 replay_seq 0, flags 0x0
>
>
>    Object  lvl   iblk   dblk  dsize  lsize   %full  type
>         0    7    16K    16K  40.5K    32K   57.81  DMU dnode
>        -1    1    16K    512     2K    512  100.00  ZFS user/group used
>        -2    1    16K    512     2K    512  100.00  ZFS user/group used
>         1    1    16K    512     2K    512  100.00  ZFS master node
>         2    1    16K    512     2K    512  100.00  SA master node
>         3    1    16K    512     2K    512  100.00  ZFS delete queue
>         4    1    16K  1.50K     2K  1.50K  100.00  ZFS directory
>         5    1    16K  1.50K     2K  1.50K  100.00  SA attr registration
>         6    1    16K    16K  10.5K    32K  100.00  SA attr layouts
>         7    1    16K    512     2K    512  100.00  ZFS directory
>         8    5    16K   128K   510G   510G  100.00  ZFS plain file
>         9    5    16K   128K   476G   476G  100.00  ZFS plain file
>        10    5    16K   128K   473G   473G  100.00  ZFS plain file
>        11    5    16K   128K   467G   467G  100.00  ZFS plain file
>        12    5    16K   128K   428G   428G  100.00  ZFS plain file
>        13    5    16K   128K   455G   455G  100.00  ZFS plain file
>        14    5    16K   128K   478G   478G  100.00  ZFS plain file
>        15    5    16K   128K   517G   517G  100.00  ZFS plain file
>        16    5    16K   128K   487G   487G  100.00  ZFS plain file
>        17    5    16K   128K   513G   513G  100.00  ZFS plain file
>        18    5    16K   128K   489G   489G  100.00  ZFS plain file
>        19    5    16K   128K   494G   493G  100.00  ZFS plain file
>        20    5    16K   128K   492G   492G  100.00  ZFS plain file
>        21    5    16K   128K   488G   487G  100.00  ZFS plain file
>        22    1    16K     1K     2K     1K  100.00  ZFS directory
>        23    4    16K   128K   107G   107G  100.00  ZFS plain file
>        24    4    16K   128K  92.4G  92.4G  100.00  ZFS plain file
>        25    4    16K   128K  97.2G  97.2G  100.00  ZFS plain file
>        26    4    16K   128K      0   128K    0.00  ZFS plain file
>        27    4    16K   128K   149G   149G  100.00  ZFS plain file
>        28    4    16K   128K   221G   221G  100.00  ZFS plain file
>        29    4    16K   128K  93.8G  93.8G  100.00  ZFS plain file
>        30    4    16K   128K  66.0G  66.0G  100.00  ZFS plain file
>        31    5    16K   128K  5.74T  5.74T  100.00  ZFS plain file
>        32    4    16K   128K  48.0G  48.0G  100.00  ZFS plain file
>        33    4    16K   128K  12.0G  12.0G  100.00  ZFS plain file
>        34    4    16K   128K  11.5G  11.5G  100.00  ZFS plain file
>        35    4    16K   128K  11.6G  11.5G  100.00  ZFS plain file
>        36    4    16K   128K  29.9G  29.8G  100.00  ZFS plain file
>        37    1    16K    512      0    512    0.00  ZFS plain file

After a long time waiting zdb started spaffing lots and lots of :-
zdb_blkptr_cb: Got error 122 reading <21, 8, 0, ea072> DVA[0]=<0:23c1e196400:2d000> [L0 ZFS plain file] fletcher4 uncompressed LE 
contiguous unique single size=20000L/20000P birth=4507L/4507P fill=1 
cksum=3f98d68a2453:fe4561392aca231:c327b800ff4c8ad5:25a834a4ff84cf0e -- skipping
zdb_blkptr_cb: Got error 122 reading <21, 8, 0, ea073> DVA[0]=<0:23c1e169400:2d000> [L0 ZFS plain file] fletcher4 uncompressed LE 
contiguous unique single size=20000L/20000P birth=4507L/4507P fill=1 
cksum=3ff53073c410:ff578ea3d9add37:56c79527f899fddd:c0f9a2751d3d4fe1 -- skipping
zdb_blkptr_cb: Got error 122 reading <21, 8, 0, ea074> DVA[0]=<0:23c1e1c3400:2d000> [L0 ZFS plain file] fletcher4 uncompressed LE 
contiguous unique single size=20000L/20000P birth=4507L/4507P fill=1 
cksum=403f20ed2647:100a131a2349650e:76e7f9a70f6d0b24:7329de5d1dfbd68b -- skipping
zdb_blkptr_cb: Got error 122 reading <21, 8, 0, ea075> DVA[0]=<0:23c1e1f0400:2d000> [L0 ZFS plain file] fletcher4 uncompressed LE 
contiguous unique single size=20000L/20000P birth=4507L/4507P fill=1 
cksum=3fa72784b6f2:fee8a882bc6849b:e90a52ef28fe1baa:8c84eb7b51c6d0ad -- skipping
zdb_blkptr_cb: Got error 122 reading <21, 8, 0, ea076> DVA[0]=<0:23c1e21d400:2d000> [L0 ZFS plain file] fletcher4 uncompressed LE 
contiguous unique single size=20000L/20000P birth=4507L/4507P fill=1 
cksum=4043be27b6cd:101d869585000e3c:52d438d1f3f14883:1f3e84a837d38701 -- skipping
zdb_blkptr_cb: Got error 122 reading <21, 8, 0, ea077> DVA[0]=<0:23c1e24a400:2d000> [L0 ZFS plain file] fletcher4 uncompressed LE 
contiguous unique single size=20000L/20000P birth=4507L/4507P fill=1 
cksum=402ba31a743e:10055a4e82d009fc:a6e97a0ebbaf4c80:6e60c0e410cf15e7 -- skipping

    Regards
    Steve 


================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.



More information about the freebsd-fs mailing list