kern/180060: ZFS kernel panic, solaris assert on dsl_prop_unregister

[James TD Smith]

>Number:         180060
>Category:       kern
>Synopsis:       ZFS kernel panic, solaris assert on dsl_prop_unregister
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jun 28 15:20:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     James TD Smith
>Release:        9.1
>Organization:
>Environment:
FreeBSD masada.internal.mohorovi.cc 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #5 r251934: Tue Jun 18 15:41:34 BST 2013 root at masada.internal.mohorovi.cc:/usr/obj/usr/src/sys/MASADA  amd64
>Description:
I've had two ZFS-related kernel panics over the last few days. Both have happened since I upgraded to 9.1-RELEASE-4 on the 18th. PR 154447 has similar backtraces but was on 8.2, I can't find any other references to a similar problem.

It's a mixed UFS/ZFS system, root is on UFS with a single 4-disk RAIDZ1 pool used for backups, a media library, poudriere and a few other jails. 

I replaced one of the disks in the pool earlier this week which had been failing for a while and have been tuning the ZFS settings based on this guide: 
http://icesquare.com/wordpress/how-to-improve-zfs-performance/ after I upgraded to 4Gb RAM.

Jun 25 08:12:57 masada kernel: panic: solaris assert: dsl_prop_unregister(ds, "atime", atime_changed_cb, zfsvfs) == 0, file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, line: 1199
Jun 25 08:12:57 masada kernel: cpuid = 0
Jun 25 08:12:57 masada kernel: KDB: stack backtrace:
Jun 25 08:12:57 masada kernel: #0 0xffffffff80926c96 at kdb_backtrace+0x66
Jun 25 08:12:57 masada kernel: #1 0xffffffff808f0cae at panic+0x1ce
Jun 25 08:12:57 masada kernel: #2 0xffffffff816b5647 at zfs_unregister_callbacks+0x1b7
Jun 25 08:12:57 masada kernel: #3 0xffffffff816b58c5 at zfsvfs_teardown+0x175
Jun 25 08:12:57 masada kernel: #4 0xffffffff816b5a2b at zfs_suspend_fs+0x1b
Jun 25 08:12:57 masada kernel: #5 0xffffffff816aa229 at zfs_ioc_rollback+0xf9
Jun 25 08:12:57 masada kernel: #6 0xffffffff816abd46 at zfsdev_ioctl+0xe6
Jun 25 08:12:57 masada kernel: #7 0xffffffff807e077b at devfs_ioctl_f+0x7b
Jun 25 08:12:57 masada kernel: #8 0xffffffff80938715 at kern_ioctl+0x115
Jun 25 08:12:57 masada kernel: #9 0xffffffff8093894d at sys_ioctl+0xfd
Jun 25 08:12:57 masada kernel: #10 0xffffffff80be71d6 at amd64_syscall+0x546

Jun 28 04:08:02 masada kernel: panic: solaris assert: dsl_prop_unregister(ds, "snapdir", snapdir_changed_cb, zfsvfs) == 0, file: /usr/src/sys/modules/zfs/../..
/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, line: 1217
Jun 28 04:08:02 masada kernel: cpuid = 1
Jun 28 04:08:02 masada kernel: KDB: stack backtrace:
Jun 28 04:08:02 masada kernel: #0 0xffffffff80926c96 at kdb_backtrace+0x66
Jun 28 04:08:02 masada kernel: #1 0xffffffff808f0cae at panic+0x1ce
Jun 28 04:08:02 masada kernel: #2 0xffffffff816b570d at zfs_unregister_callbacks+0x27d
Jun 28 04:08:02 masada kernel: #3 0xffffffff816b58c5 at zfsvfs_teardown+0x175
Jun 28 04:08:02 masada kernel: #4 0xffffffff816b5a2b at zfs_suspend_fs+0x1b
Jun 28 04:08:02 masada kernel: #5 0xffffffff816aa229 at zfs_ioc_rollback+0xf9
Jun 28 04:08:02 masada kernel: #6 0xffffffff816abd46 at zfsdev_ioctl+0xe6
Jun 28 04:08:02 masada kernel: #7 0xffffffff807e077b at devfs_ioctl_f+0x7b
Jun 28 04:08:02 masada kernel: #8 0xffffffff80938715 at kern_ioctl+0x115
Jun 28 04:08:02 masada kernel: #9 0xffffffff8093894d at sys_ioctl+0xfd
Jun 28 04:08:02 masada kernel: #10 0xffffffff80be71d6 at amd64_syscall+0x546
>How-To-Repeat:
I think it's more likely to happen under heavy IO load. This machine has rsnaphot backups scheduled to run every 4 hours, both of the crashes have occurred shortly after a backup began. It was also building packages with poudriere at the same time in both instances.
>Fix:



>Release-Note:
>Audit-Trail:
>Unformatted: