panic: dva_get_dsize_sync(): bad DVA on 2016 11-STABLE.

Thu Aug 3 08:54:59 UTC 2017

Hi,

today I got the following panic on the December 2016 11-STABLE:

FreeBSD san02.playkey.net 11.0-STABLE FreeBSD 11.0-STABLE #0 r310734M: 
Thu Dec 29 19:22:30 UTC 2016 emz at san02:/usr/obj/usr/src/sys/GENERIC  amd64

panic: dva_get_dsize_sync(): bad DVA 4294967295:2086400

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: dva_get_dsize_sync(): bad DVA 4294967295:2086400
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff80b023a7 at kdb_backtrace+0x67
#1 0xffffffff80ab88e6 at vpanic+0x186
#2 0xffffffff80ab8753 at panic+0x43
#3 0xffffffff8226a148 at bp_get_dsize+0x128
#4 0xffffffff8222cc29 at dmu_tx_count_write+0x589
#5 0xffffffff8222c675 at dmu_tx_hold_write+0x35
#6 0xffffffff822c573e at zvol_strategy+0x21e
#7 0xffffffff809f6a10 at g_io_request+0x4a0
#8 0xffffffff8283a45f at ctl_be_block_dispatch_dev+0x20f
#9 0xffffffff8283bddc at ctl_be_block_worker+0x6c
#10 0xffffffff80b1484a at taskqueue_run_locked+0x14a
#11 0xffffffff80b15a38 at taskqueue_thread_loop+0xe8
#12 0xffffffff80a70785 at fork_exit+0x85
#13 0xffffffff80f55f2e at fork_trampoline+0xe
Uptime: 78d7h43m31s

My question is (since I din't find much on this) what does this "bad 
DVA" mean ? I've read that this may indicate the on-disk zfs corrupton, 
but I'm not suer about it. Is this fixable in any way ? Do I have to 
prepare to recreate the pool (btw I have three pools) from scratch, and 
how do I determine which one has the corruption.

Some [useless ?] zfs info:

# zpool status
   pool: data
  state: ONLINE
   scan: none requested
config:

         NAME        STATE     READ WRITE CKSUM
         data        ONLINE       0     0     0
           raidz1-0  ONLINE       0     0     0
             da2     ONLINE       0     0     0
             da3     ONLINE       0     0     0
             da4     ONLINE       0     0     0
             da5     ONLINE       0     0     0
           raidz1-1  ONLINE       0     0     0
             da6     ONLINE       0     0     0
             da7     ONLINE       0     0     0
             da8     ONLINE       0     0     0
             da9     ONLINE       0     0     0
           raidz1-2  ONLINE       0     0     0
             da10    ONLINE       0     0     0
             da11    ONLINE       0     0     0
             da12    ONLINE       0     0     0
             da13    ONLINE       0     0     0
           raidz1-3  ONLINE       0     0     0
             da14    ONLINE       0     0     0
             da15    ONLINE       0     0     0
             da16    ONLINE       0     0     0
             da17    ONLINE       0     0     0

errors: No known data errors

   pool: userdata
  state: ONLINE
   scan: none requested
config:

         NAME               STATE     READ WRITE CKSUM
         userdata           ONLINE       0     0     0
           mirror-0         ONLINE       0     0     0
             gpt/userdata0  ONLINE       0     0     0
             gpt/userdata1  ONLINE       0     0     0

errors: No known data errors

   pool: zroot
  state: ONLINE
   scan: none requested
config:

         NAME            STATE     READ WRITE CKSUM
         zroot           ONLINE       0     0     0
           mirror-0      ONLINE       0     0     0
             gpt/zroot0  ONLINE       0     0     0
             gpt/zroot1  ONLINE       0     0     0

errors: No known data errors

Hardware:

# camcontrol devlist
<AHCI SGPIO Enclosure 1.00 0001>   at scbus4 target 0 lun 0 (pass0,ses0)
<AHCI SGPIO Enclosure 1.00 0001>   at scbus11 target 0 lun 0 (pass1,ses1)
<ATA HGST HUS722T1TAL WA05>        at scbus12 target 4 lun 0 (pass2,da0)
<ATA HGST HUS722T1TAL WA05>        at scbus12 target 5 lun 0 (pass3,da1)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 8 lun 0 (pass4,da2)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 9 lun 0 (pass5,da3)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 10 lun 0 (pass6,da4)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 11 lun 0 (pass7,da5)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 12 lun 0 (pass8,da6)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 13 lun 0 (pass9,da7)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 14 lun 0 (pass10,da8)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 15 lun 0 (pass11,da9)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 16 lun 0 (pass12,da10)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 17 lun 0 (pass13,da11)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 18 lun 0 (pass14,da12)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 19 lun 0 (pass15,da13)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 20 lun 0 (pass16,da14)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 21 lun 0 (pass17,da15)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 22 lun 0 (pass18,da16)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 23 lun 0 (pass19,da17)
<ATA TOSHIBA THNSN81Q 6101>        at scbus12 target 24 lun 0 (pass20,da18)
<LSI SAS3x40 0601>                 at scbus12 target 32 lun 0 (pass21,ses2)

I also have the zdb -uuumdC for each pool, here they are in case someone 
needs them:

https://enaza.ru/stub-data/zdb-uuumdC-data.txt

https://enaza.ru/stub-data/zdb-uuumdC-userdata.txt

https://enaza.ru/stub-data/zdb-uuumdC-zroot.txt

Thanks.

Eugene.