solaris assert: avl_is_empty(&dn -> dn_dbufs) panic

[Don Lewis]

On 21 Aug, Don Lewis wrote:
> I just started getting this panic:
> 
> solaris assert: avl_is_empty(&dn -> dn_dbufs), file:
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c,
> line 495
> 
> System info:
> FreeBSD zipper.catspoiler.org 11.0-CURRENT FreeBSD 11.0-CURRENT #25 r286923: Wed Aug 19 09:28:53 PDT 2015     dl at zipper.catspoiler.org:/usr/obj/usr/src/sys/GENERIC  amd64
> 
> My zfs pool has one mirrored vdev.  Scrub doesn't find any problems.
> 
> %zpool status
>   pool: zroot
>  state: ONLINE
>   scan: scrub repaired 0 in 2h58m with 0 errors on Fri Aug 21 00:44:52 2015
> config:
> 
> 	NAME        STATE     READ WRITE CKSUM
> 	zroot       ONLINE       0     0     0
> 	  mirror-0  ONLINE       0     0     0
> 	    ada0p3  ONLINE       0     0     0
> 	    ada1p3  ONLINE       0     0     0
> 
> This panic is reproduceable and happens every time I use poudriere to
> build ports using my 9.3-RELEASE amd64 jail and occurs at the end of the
> poudriere run when it is unmounting filesystems.
> 
> [00:10:43] ====>> Stopping 4 builders
> 93amd64-default-job-01: removed
> 93amd64-default-job-01-n: removed
> 93amd64-default-job-02: removed
> 93amd64-default-job-02-n: removed
> 93amd64-default-job-03: removed
> 93amd64-default-job-03-n: removed
> 93amd64-default-job-04: removed
> 93amd64-default-job-04-n: removed
> [00:10:46] ====>> Creating pkgng repository
> Creating repository in /tmp/packages: 100%
> Packing files for repository: 100%
> [00:10:55] ====>> Committing packages to repository
> [00:10:55] ====>> Removing old packages
> [00:10:55] ====>> Built ports: devel/py-pymtbl net/sie-nmsg net/p5-Net-Nmsg net/axa
> [93amd64-default] [2015-08-21_00h47m41s] [committing:] Queued: 4  Built: 4  Failed: 0  Skipped: 0  Ignored: 0  Tobuild: 0   Time: 00:10:53
> [00:10:55] ====>> Logs: /var/poudriere/data/logs/bulk/93amd64-default/2015-08-21_00h47m41s
> [00:10:55] ====>> Cleaning up
> 93amd64-default: removed
> 93amd64-default-n: removed
> [00:10:55] ====>> Umounting file systems
> Write failed: Broken pipe
> 
> Prior to that, I ran poudriere a number of times with a 10.2-STABLE
> amd64 jail without incident.
> 
> I've kicked off a bunch of poudriere runs for other jails and
> will check on it in the morning.

Died the same way after building ports on the first jail,
10.1-RELEASE amd64.

Since there have been some zfs commits since r286923, I upgraded to
r286998 this morning and tried again with no better luck.  I got the
same panic again.

This machine has mirrored swap, and even though I've done what
gmirror(8) says to do in order to capture crash dumps, I've had no luck
with that.  The dump is getting written, but savecore is unable to find
it.