kern/182570: ZFS panic in receive

[Keith White]

>Number:         182570
>Category:       kern
>Synopsis:       ZFS panic in receive
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 02 11:50:03 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Keith White
>Release:        current
>Organization:
EECS, University of Ottawa
>Environment:
FreeBSD freebsd10 10.0-ALPHA4 FreeBSD 10.0-ALPHA4 #0 r255937: Sun Sep 29 09:45:21 EDT 2013     kwhite at freebsd10:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
[previously sent to freebsd-current]

I get the following reproducible panic when doing a zfs send -R | zfs recv
of a well churned file system (snapshots of the ports tree before and
after libiconv update) running a recent current:

panic: solaris assert: dn->dn_maxblkid == 0 &&
(BP_IS_HOLE(&dn->dn_phys->dn_blkptr[0]) || dnode_block_freed(dn, 0)),
file:
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c,
line: 598

coredump available.


# zfs list -t snapshot -r tank/ports
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank/ports at 20130915  72.5M      -  3.64G  -
tank/ports at 20130917      0      -  3.65G  -
tank/ports at 20130918      0      -  3.65G  -
tank/ports at 20130921  88.6M      -  3.66G  -
tank/ports at 20130928   352K      -  3.66G  -

# zpool list -v
NAME             SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
m_tank          1.81T   831G  1.00T    44%  1.00x  ONLINE  -
  ada4.nop.eli  1.81T   831G  1.00T         -
tank             928G   831G  96.9G    89%  1.00x  ONLINE  -
  ada3.nop.eli   928G   831G  96.9G         -

# zfs send -vR tank/ports at 20130928 | zfs recv -vF m_tank/xports
.. panic eventually ...

# cd /boot/kernel; kgdb kernel /var/crash/vmcore.last
..
(kgdb) up
#5  0xffffffff81d09735 in dnode_reallocate (dn=0xfffff8006dde3000,
ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024, bonustype=DMU_OT_SA,
bonuslen=168, tx=0xfffff8006d7a2600)
    at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:596


596                     ASSERT(dn->dn_maxblkid == 0 &&

(kgdb) p dn->dn_maxblkid
$1 = 2

So, no question about why the ASSERT failed.

Sorry, debugging this is *way* beyond me.  Any hints, patches to try?


>How-To-Repeat:
on a zpool called "tank" with non-existent tank/nobj and tank/xobj:

#!/bin/sh

zfs create tank/nobj
zfs snapshot tank/nobj at 0
__MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6 buildkernel
zfs snapshot tank/nobj at 1
find /tank/nobj -name '.h' -print | xargs rm
zfs snapshot tank/nobj at 2
rm -rf /tank/nobj/*
zfs snapshot tank/nobj at 3
__MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6 buildkernel
zfs snapshot tank/nobj at 4
rm -rf /tank/nobj/*
zfs snapshot tank/nobj at 5

zfs send -vR tank/nobj at 5 | zfs recv -vF tank/xobj

>Fix:
This will just avoid the panic, the received filesystem is not complete.  The "zfs recv" now terminates with "internal error: File too large":


Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c
===================================================================
--- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c      (revision 255986)
+++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c      (working copy)
@@ -592,10 +592,16 @@
        rw_enter(&dn->dn_struct_rwlock, RW_WRITER);
        dnode_setdirty(dn, tx);
        if (dn->dn_datablksz != blocksize) {
+#if 1
+               if (dn->dn_maxblkid != 0) {
+                       printf("ZFS should panic here since dn->dn_maxblkid is %lu instead of 0\n...but we'll continue\n", dn->dn_maxblkid);
+               }
+#else
                /* change blocksize */
                ASSERT(dn->dn_maxblkid == 0 &&
                    (BP_IS_HOLE(&dn->dn_phys->dn_blkptr[0]) ||
                    dnode_block_freed(dn, 0)));
+#endif
                dnode_setdblksz(dn, blocksize);
                dn->dn_next_blksz[tx->tx_txg&TXG_MASK] = blocksize;
        }


>Release-Note:
>Audit-Trail:
>Unformatted: