kern/144402: [zfs] panic at zfs_znode_dmu_init: existing znode for dbuf

[Alex Bakhtin]

>Number:         144402
>Category:       kern
>Synopsis:       [zfs] panic at zfs_znode_dmu_init: existing znode for dbuf
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 01 22:20:05 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Alex Bakhtin
>Release:        8.0-STABLE
>Organization:
>Environment:
FreeBSD tarzan-new.private.flydrag.ru 8.0-STABLE FreeBSD 8.0-STABLE #1: Sun Feb 28 18:58:11 UTC 2010     bakhtin at tarzan-new.private.flydrag.ru:/zfs/obj/usr/src.old/sys/DEBUG  amd64
>Description:
In pre-production testing of FreeBSD 8.0 with ZFS/raidz I found a 100% reproducible panic in zfs. The problem was initially found after two hours of benchmarks/dbench testing on 8.0-RELEASE. The problem was reproduced on 8.0-STABLE csuped at Feb 28, 2010.

FreeBSD 8.0-RELEASE #0: Sun Feb 28 15:40:09 UTC 2010
    bakhtin at tarzan-new.private.flydrag.ru:/zfs/obj/usr/src.old/sys/DEBUG

CPU: Intel(R) Pentium(R)  CPU       E5200  @ 2.50GHz (2536.15-MHz K8-class CPU)
  real memory  = 8589934592 (8192 MB)

tarzan-new# zpool status
  pool: zfs
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on older software versions.
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zfs         ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad10    ONLINE       0     0     0
            ad12    ONLINE       0     0     0
            ad16    ONLINE       0     0     0

errors: No known data errors
tarzan-new#

tarzan-new# zdb -vvv
zfs
    version=13
    name='zfs'
    state=0
    txg=342
    pool_guid=14801748754090954299
    hostid=4266611921
    hostname='tarzan-new.private.flydrag.ru'
    vdev_tree
        type='root'
        id=0
        guid=14801748754090954299
        children[0]
                type='raidz'
                id=0
                guid=11719031541734505632
                nparity=1
                metaslab_array=23
                metaslab_shift=31
                ashift=9
                asize=240064659456
                is_log=0
                children[0]
                        type='disk'
                        id=0
                        guid=541462146913312867
                        path='/dev/ad10'
                        whole_disk=0
                children[1]
                        type='disk'
                        id=1
                        guid=14783639361535716946
                        path='/dev/ad12'
                        whole_disk=0
                children[2]
                        type='disk'
                        id=2
                        guid=8087457233125113893
                        path='/dev/ad16'
                        whole_disk=0
tarzan-new#


Crash data:

  60    405488    53.18 MB/sec  execute 400 sec
  60    405664    53.18 MB/sec  execute 401 sec
panic: existing znode 0xffffff0103514468 for dbuf 0xffffff00d494ea80^M
cpuid = 1^M
KDB: enter: panic^M
[thread pid 1113 tid 100202 ]
Stopped at      kdb_enter+0x3d: movq    $0,0x69a270(%rip)
db:0:kdb.enter.panic>  bt
Tracing pid 1113 tid 100202 td 0xffffff00196703a0
kdb_enter() at kdb_enter+0x3d
panic() at panic+0x17b
zfs_znode_dmu_init() at zfs_znode_dmu_init+0xb5
zfs_znode_alloc() at zfs_znode_alloc+0xa0
zfs_mknode() at zfs_mknode+0x204
zfs_freebsd_create() at zfs_freebsd_create+0x594
VOP_CREATE_APV() at VOP_CREATE_APV+0xb3
vn_open_cred() at vn_open_cred+0x473
kern_openat() at kern_openat+0x179
syscall() at syscall+0x118
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x80073075c, rsp = 0x7fffffffdd98, rbp = 0x800a05100 ---


Console logs:
http://flydrag.dyndns.org:9090/freebsd/zfs-panic/1/console.txt

crashinfo:
http://flydrag.dyndns.org:9090/freebsd/zfs-panic/1/core.txt

vmcore:
http://flydrag.dyndns.org:9090/freebsd/zfs-panic/1/vmcore.1.gz

>How-To-Repeat:

Install FreeBSD/amd64 (not tested on i386) 8.0-RELEASE, csup (optionally) to STABLE.

Create zpool (tested on raidz1).

Install /usr/ports/benchmarks/dbench

Run dbench -t 10000 -D /zfs/bench 60


>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: