Messages & dump with ZFS

[Michael Reifenberger]

Hi,
my setup using -current in an VMWare-session:
   FreeBSD guest 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Tue Jan  9 10:48:26 UTC 2007
   root at guest:/usr/obj.i386/usr/src/sys/GENERIC  i386

# dd if=/dev/zero of=/a.dd bs=1m count=100
# dd  if=/dev/zero of=/b.dd bs=1m count=100 
# mdconfig -a -tvnode -f /a.dd -u11
# mdconfig -a -tvnode -f /b.dd -u12 
# zpool create test -m /mnt/test /dev/md11 /dev/md12
# ( cd /mnt/test; iozone -s100m -r1m )
   #Here I get a panic kmem_malloc(16384): kmem_map too small: 83247104 total
    allocated if I have 128MB memory assigned to the VM
# zpool export test
# zpool import test

After all I have the attached messages on the screen.

What else can I debug?

BTW: Thanks for your work.
ZFS feels a lot like ADVFS in True64-Unix which I worked with gladly.

Bye/2
---
Michael Reifenberger
Michael at Reifenberger.com
http://www.Reifenberger.com
-------------- next part --------------
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT #0: Tue Jan  9 10:48:26 UTC 2007
    root at guest.reifenberger.com:/usr/obj.i386/usr/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) III Mobile CPU      1200MHz (1204.45-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x6b8  Stepping = 8
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 536870912 (512 MB)
avail memory = 511442944 (487 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <Intel 82443BX (440 BX) host to PCI bridge> on hostb0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1050-0x105f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
uhci0: <Intel 82371AB/EB (PIIX4) USB controller> port 0x1060-0x107f irq 19 at device 7.2 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 82371AB/EB (PIIX4) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> port 0x14c0-0x14cf mem 0xf0000000-0xf7ffffff,0xec000000-0xec7fffff at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xec800000-0xec800fff irq 17 at device 16.0 on pci0
mpt0: [GIANT-LOCKED]
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
le0: <AMD PCnet-PCI> port 0x1400-0x147f irq 18 at device 17.0 on pci0
le0: 16 receive buffers, 4 transmit buffers
le0: Ethernet address: 00:0c:29:2b:ad:6b
le0: [ITHREAD]
pci0: <multimedia, audio> at device 18.0 (no driver attached)
acpi_acad0: <AC Adapter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xdc000-0xdffff,0xe0000-0xe3fff pnpid ORM0000 on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1204453199 Hz quality 800
Timecounters tick every 1.000 msec
Waiting 5 seconds for SCSI devices to settle
(xpt0:mpt0:0:-1:-1): reset bus
ad0: 5120MB <VMware Virtual IDE Hard Drive 00000001> at ata0-master UDMA33
acd0: CDRW <VMware Virtual IDE CDROM Drive/00000001> at ata1-master UDMA33
Trying to mount root from ufs:/dev/ad0s1a
ZFS filesystem version 3
ZFS storage pool version 3
acquiring duplicate lock of same type: "zfs:dbuf"
 1st zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,d61ccaa4,c0a5c8d8,c0a5c8d8,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c34bfd5c,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(c34bfd5c,c321d083,2c,c3059c00,c3132000,...) at _sx_xlock+0x50
dbuf_rele(c34bfd20,c3059c00,c319f000,c3059c00,c3059c00,...) at dbuf_rele+0x19
dsl_dir_close(c3132000,c3059c00,c34bfbd0,d61ccb30,c31cc233,...) at dsl_dir_close+0x1f
dsl_dir_evict(c34bfbd0,c3059c00,0,d61ccb48,c31ce27b,...) at dsl_dir_evict+0x21
dbuf_evict_user(c34bfbd0,c2ee8400,c3059c00,c32bbe00,d61ccb68,...) at dbuf_evict_user+0x37
dbuf_rele(c34bfbd0,c2ee8400,c319f000,c2ee8400,c2ee8400,...) at dbuf_rele+0x93
dsl_dir_close(c3059c00,c2ee8400,c319f000,d61ccb90,c31e60f3,...) at dsl_dir_close+0x1f
dsl_pool_close(c2ee8400,c319f000,c32bb560,d61ccbb4,c31e7c00,...) at dsl_pool_close+0x16
spa_unload(c319f000,0,0,c30b7520,0,...) at spa_unload+0x5b
spa_tryimport(c32bb560,c32bb560,c314d000,48,0,...) at spa_tryimport+0x130
zfs_ioc_pool_tryimport(c314d000,6,0,ceec5a06,c2ffe900,...) at zfs_ioc_pool_tryimport+0x27
zfsdev_ioctl(c3174700,ceec5a06,c314d000,3,c305fa20,...) at zfsdev_ioctl+0xca
devfs_ioctl_f(c2ffe900,ceec5a06,c314d000,c3197580,c305fa20) at devfs_ioctl_f+0xaf
kern_ioctl(c305fa20,3,ceec5a06,c314d000) at kern_ioctl+0x296
ioctl(c305fa20,d61ccd00) at ioctl+0xf1
syscall(d61ccd38) at syscall+0x256
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x281991df, esp = 0xbfbf978c, ebp = 0xbfbf97c8 ---
acquiring duplicate lock of same type: "zfs:dbuf"
 1st zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,d625faa4,c0a5c9f0,c0a5c9f0,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c353518c,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(c353518c,c321d083,2c,c3579a00,c3579c00,...) at _sx_xlock+0x50
dbuf_rele(c3535150,c3579a00,c319c800,c3579a00,c3579a00,...) at dbuf_rele+0x19
dsl_dir_close(c3579c00,c3579a00,c3535000,d625fb30,c31cc233,...) at dsl_dir_close+0x1f
dsl_dir_evict(c3535000,c3579a00,0,d625fb48,c31ce27b,...) at dsl_dir_evict+0x21
dbuf_evict_user(c3535000,c3564400,c3579a00,c339e4e0,d625fb68,...) at dbuf_evict_user+0x37
dbuf_rele(c3535000,c3564400,c319c800,c3564400,c3564400,...) at dbuf_rele+0x93
dsl_dir_close(c3579a00,c3564400,c319c800,d625fb90,c31e60f3,...) at dsl_dir_close+0x1f
dsl_pool_close(c3564400,c319c800,c32bb2e0,d625fbb4,c31e7c00,...) at dsl_pool_close+0x16
spa_unload(c319c800,0,0,c3343d20,0,...) at spa_unload+0x5b
spa_tryimport(c32bb2e0,c32bb2e0,c32ff000,48,0,...) at spa_tryimport+0x130
zfs_ioc_pool_tryimport(c32ff000,6,0,ceec5a06,c2ffea68,...) at zfs_ioc_pool_tryimport+0x27
zfsdev_ioctl(c3174700,ceec5a06,c32ff000,3,c32df870,...) at zfsdev_ioctl+0xca
devfs_ioctl_f(c2ffea68,ceec5a06,c32ff000,c3197580,c32df870) at devfs_ioctl_f+0xaf
kern_ioctl(c32df870,3,ceec5a06,c32ff000) at kern_ioctl+0x296
ioctl(c32df870,d625fd00) at ioctl+0xf1
syscall(d625fd38) at syscall+0x256
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x281991df, esp = 0xbfbf977c, ebp = 0xbfbf97b8 ---
acquiring duplicate lock of same type: "zfs:dbuf"
 1st zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,d6232a70,c0a5c900,c0a5c900,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c3535624,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(c3535624,c321d083,2c,c32bb440,c3579c00,...) at _sx_xlock+0x50
dbuf_rele(c35355e8,c32bb440,c319e800,c32bb440,c32bb440,...) at dbuf_rele+0x19
dsl_dir_close(c3579c00,c32bb440,c32bb440,18,c2fcb370,...) at dsl_dir_close+0x1f
dsl_prop_unregister(c357a200,c32218f7,c31d3f54,c3579600,322107,...) at dsl_prop_unregister+0xd1
dmu_objset_evict(c357a200,c3579600,c35353f0,0,d6232b54,...) at dmu_objset_evict+0x3a
dsl_dataset_evict(c35353f0,c357a200,1,d6232b6c,c31ce27b,...) at dsl_dataset_evict+0x34
dbuf_evict_user(c35353f0,c3413800,c3579600,c3405000,d6232b88,...) at dbuf_evict_user+0x37
dbuf_rele(c35353f0,c357a200,0,c34a87ec,0,...) at dbuf_rele+0x93
killer(c3405000,c34aa604,c3579600,0,c34aa604,...) at killer+0xec
arc_write_done(c3405000,c3512480,c319e800,c3405000,c3405200,...) at arc_write_done+0x16c
zio_done(c3405000,c34051f4,d6232c00,c3204df7,c3405000,...) at zio_done+0x159
zio_next_stage(c3405000,c3405000,c3405000,0,d6232c14,...) at zio_next_stage+0x8e
zio_wait_for_children(c3405000,11,c34051f4,d6232c24,c320681a,...) at zio_wait_for_children+0x3f
zio_wait_children_done(c3405000,c3405000,d6232c5c,c32065ae,c3405000,...) at zio_wait_children_done+0x15
zio_next_stage(c3405000,0,c06b829b,c3224260,80,...) at zio_next_stage+0x8e
zio_vdev_io_assess(c3405000,c34137b8,d6232c94,c31f547e,c3405000,...) at zio_vdev_io_assess+0x19a
zio_next_stage(c3405000,c3405000,c0a4de88,0,3,...) at zio_next_stage+0x8e
vdev_mirror_io_done(c3405000,d6232d04,c31c5c61,c3405000,c32c2a48,...) at vdev_mirror_io_done+0xba
zio_vdev_io_done(c3405000,c32c2a48,c32c2a48,c32c2a48,29c8840,...) at zio_vdev_io_done+0x16
taskq_thread(c32c2a28,d6232d38) at taskq_thread+0x195
fork_exit(c31c5acc,c32c2a28,d6232d38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd6232d70, ebp = 0 ---
acquiring duplicate lock of same type: "zfs:dns"
 1st zfs:dns @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/rwlock.h:45
 2nd zfs:dns @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/rwlock.h:45
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,d630da60,c0a5c978,c0a5c978,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c3550480,9,c321d47a,2d) at witness_checkorder+0x586
_sx_xlock(c3550480,c321d47a,2d,c3535774,36,...) at _sx_xlock+0x50
dnode_new_blkid(c3550480,0,0,c3412e00,c09499a4,...) at dnode_new_blkid+0x57
dbuf_dirty(c3535738,c3412e00,c3533b40,5,c3132268,...) at dbuf_dirty+0x375
dnode_setdirty(c3533b40,c3412e00,c3533b40,0,0,...) at dnode_setdirty+0xf9
dnode_free_range(c3533b40,0,0,ffffffff,ffffffff,...) at dnode_free_range+0x5de
dmu_free_range(c3132218,12,0,0,0,...) at dmu_free_range+0x3f
space_map_truncate(c355904c,c3132218,c3412e00,11e,1,...) at space_map_truncate+0x1f
metaslab_sync(c3559000,5,0,c31a01f8,c319e800,...) at metaslab_sync+0x221
vdev_sync(c31a0000,5,0,aa,c2dbe4ac,...) at vdev_sync+0x98
spa_sync(c319e800,5,0,c2dbe4e0,c2dbe4e0,...) at spa_sync+0x23c
txg_sync_thread(c2dbe400,d630dd38) at txg_sync_thread+0x1b3
fork_exit(c31ef23c,c2dbe400,d630dd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd630dd70, ebp = 0 ---
lock order reversal:
 1st 0xc341d520 <mtxnone> (<mtxnone>) @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd 0xc35350e4 zfs:dbuf (zfs:dbuf) @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,ffffffff,c0a5c6f8,c0a5c900,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c35350e4,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(c35350e4,c321d083,2c,c35506c0,c341d700,...) at _sx_xlock+0x50
dbuf_sync_indirect(c341d700,c3413680,c341d520,c3404800,c341d500,...) at dbuf_sync_indirect+0x25
dbuf_sync_list(c341d554,c3413680,c35506c0,c341d500,c35507b8,...) at dbuf_sync_list+0x30
dbuf_sync_indirect(c341d500,c3413680,c34b3400,c35506c0,c3579800,...) at dbuf_sync_indirect+0xca
dbuf_sync_list(c35507b8,c3413680,c35507b8,2,40000,...) at dbuf_sync_list+0x30
dnode_sync(c35506c0,c3413680,c33be800,5,0,...) at dnode_sync+0x29b
dmu_objset_sync(c3579800,c3556c00,c3413680,c3579c00,c3413680,...) at dmu_objset_sync+0x112
dsl_dataset_sync(c3059e00,c3556c00,c3413680,c3132200,c3413680,...) at dsl_dataset_sync+0x22
dsl_pool_sync(c2dbe400,6,0,c319e800,6,...) at dsl_pool_sync+0x74
spa_sync(c319e800,6,0,c2dbe4e0,c2dbe4e0,...) at spa_sync+0x228
txg_sync_thread(c2dbe400,d630dd38) at txg_sync_thread+0x1b3
fork_exit(c31ef23c,c2dbe400,d630dd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd630dd70, ebp = 0 ---
acquiring duplicate lock of same type: "<mtxnone>"
 1st <mtxnone> @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd <mtxnone> @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,d630daf8,c0a5c6f8,c0a5c6f8,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c341d720,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(c341d720,c321d083,2c,c35506c0,c341d700,...) at _sx_xlock+0x50
dbuf_sync_indirect(c341d700,c3413680,c341d520,c3404800,c341d500,...) at dbuf_sync_indirect+0xbb
dbuf_sync_list(c341d554,c3413680,c35506c0,c341d500,c35507b8,...) at dbuf_sync_list+0x30
dbuf_sync_indirect(c341d500,c3413680,c34b3400,c35506c0,c3579800,...) at dbuf_sync_indirect+0xca
dbuf_sync_list(c35507b8,c3413680,c35507b8,2,40000,...) at dbuf_sync_list+0x30
dnode_sync(c35506c0,c3413680,c33be800,5,0,...) at dnode_sync+0x29b
dmu_objset_sync(c3579800,c3556c00,c3413680,c3579c00,c3413680,...) at dmu_objset_sync+0x112
dsl_dataset_sync(c3059e00,c3556c00,c3413680,c3132200,c3413680,...) at dsl_dataset_sync+0x22
dsl_pool_sync(c2dbe400,6,0,c319e800,6,...) at dsl_pool_sync+0x74
spa_sync(c319e800,6,0,c2dbe4e0,c2dbe4e0,...) at spa_sync+0x228
txg_sync_thread(c2dbe400,d630dd38) at txg_sync_thread+0x1b3
fork_exit(c31ef23c,c2dbe400,d630dd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd630dd70, ebp = 0 ---
lock order reversal:
 1st 0xc34c0e04 zfs:dbuf (zfs:dbuf) @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd 0xc34d6b10 zfs:zn (zfs:zn) @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,ffffffff,c0a5c900,c0a5c658,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c34d6b10,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(c34d6b10,c321d083,2c,d630da2c,c0a4de58,...) at _sx_xlock+0x50
znode_pageout_func(c34c0dc8,c34d6b00,c34c0dc8,d630da64,c31cd8a5,...) at znode_pageout_func+0xf0
dbuf_evict_user(c34c0dc8,0,c35421f8,c34c0dc8,c34beb40,...) at dbuf_evict_user+0x37
dbuf_clear(c34c0dc8,d630da90,d630db4c,c31db559,c34c0dc8,...) at dbuf_clear+0x25
dbuf_evict(c34c0dc8,c34bed14,c34bece0,0,0,...) at dbuf_evict+0xd
dnode_evict_dbufs(c34beb40,0,c34bec28,c3517800,c3413680,...) at dnode_evict_dbufs+0x209
dnode_sync_free(c34beb40,c3413680,c34bec28,1,c34beb40,...) at dnode_sync_free+0x2c
dnode_sync(c34beb40,c3413680,c35798ec,c34beb40,c3579810,...) at dnode_sync+0x26a
dmu_objset_sync_dnodes(c35798ec,c3413680,c35506c0,c3413680,c33cd800,...) at dmu_objset_sync_dnodes+0x30
dmu_objset_sync(c3579800,c3555c00,c3413680,c3579c00,c3413680,...) at dmu_objset_sync+0x12e
dsl_dataset_sync(c3059e00,c3555c00,c3413680,c3132200,c3413680,...) at dsl_dataset_sync+0x22
dsl_pool_sync(c2dbe400,9,0,c319e800,9,...) at dsl_pool_sync+0x74
spa_sync(c319e800,9,0,c2dbe4e0,c2dbe4e0,...) at spa_sync+0x228
txg_sync_thread(c2dbe400,d630dd38) at txg_sync_thread+0x1b3
fork_exit(c31ef23c,c2dbe400,d630dd38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd630dd70, ebp = 0 ---
acquiring duplicate lock of same type: "zfs:dbuf"
 1st zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,d6256aa4,c0a5c9a0,c0a5c9a0,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(ca0dccb4,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(ca0dccb4,c321d083,2c,c3579e00,c3579400,...) at _sx_xlock+0x50
dbuf_rele(ca0dcc78,c3579e00,c3025000,c3579e00,c3579e00,...) at dbuf_rele+0x19
dsl_dir_close(c3579400,c3579e00,ca0dcb28,d6256b30,c31cc233,...) at dsl_dir_close+0x1f
dsl_dir_evict(ca0dcb28,c3579e00,0,d6256b48,c31ce27b,...) at dsl_dir_evict+0x21
dbuf_evict_user(ca0dcb28,c3492400,c3579e00,c3437b20,d6256b68,...) at dbuf_evict_user+0x37
dbuf_rele(ca0dcb28,c3492400,c3025000,c3492400,c3492400,...) at dbuf_rele+0x93
dsl_dir_close(c3579e00,c3492400,c3025000,d6256b90,c31e60f3,...) at dsl_dir_close+0x1f
dsl_pool_close(c3492400,c3025000,c347bba0,d6256bb4,c31e7c00,...) at dsl_pool_close+0x16
spa_unload(c3025000,1,0,c30b7260,0,...) at spa_unload+0x5b
spa_tryimport(c347bba0,c347bba0,c3324000,48,0,...) at spa_tryimport+0x130
zfs_ioc_pool_tryimport(c3324000,6,0,ceec5a06,c2ffe900,...) at zfs_ioc_pool_tryimport+0x27
zfsdev_ioctl(c3174700,ceec5a06,c3324000,3,c32df360,...) at zfsdev_ioctl+0xca
devfs_ioctl_f(c2ffe900,ceec5a06,c3324000,c3414080,c32df360) at devfs_ioctl_f+0xaf
kern_ioctl(c32df360,3,ceec5a06,c3324000) at kern_ioctl+0x296
ioctl(c32df360,d6256d00) at ioctl+0xf1
syscall(d6256d38) at syscall+0x256
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x281991df, esp = 0xbfbf96fc, ebp = 0xbfbf9738 ---
acquiring duplicate lock of same type: "zfs:dbuf"
 1st zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd zfs:dbuf @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,d624aa70,c0a5ca90,c0a5ca90,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c501ee04,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(c501ee04,c321d083,2c,c32bb3e0,c4055e00,...) at _sx_xlock+0x50
dbuf_rele(c501edc8,c32bb3e0,c3079800,c32bb3e0,c32bb3e0,...) at dbuf_rele+0x19
dsl_dir_close(c4055e00,c32bb3e0,c32bb3e0,18,c40afc10,...) at dsl_dir_close+0x1f
dsl_prop_unregister(c3579200,c32218f7,c31d3f54,c405a400,c32d9b01,...) at dsl_prop_unregister+0xd1
dmu_objset_evict(c3579200,c405a400,c40c73f0,0,d624ab54,...) at dmu_objset_evict+0x3a
dsl_dataset_evict(c40c73f0,c3579200,1,d624ab6c,c31ce27b,...) at dsl_dataset_evict+0x34
dbuf_evict_user(c40c73f0,c4dc2980,c405a400,c348dc00,d624ab88,...) at dbuf_evict_user+0x37
dbuf_rele(c40c73f0,c3579200,0,c5d48138,0,...) at dbuf_rele+0x93
killer(c348dc00,c34aa280,c405a400,400,c34aa280,...) at killer+0xec
arc_write_done(c348dc00,ca0b0480,c3079800,c348dc00,c348de00,...) at arc_write_done+0x16c
zio_done(c348dc00,c348ddf4,d624ac00,c3204df7,c348dc00,...) at zio_done+0x159
zio_next_stage(c348dc00,c348dc00,c348dc00,0,d624ac14,...) at zio_next_stage+0x8e
zio_wait_for_children(c348dc00,11,c348ddf4,d624ac24,c320681a,...) at zio_wait_for_children+0x3f
zio_wait_children_done(c348dc00,c348dc00,d624ac5c,c32065ae,c348dc00,...) at zio_wait_children_done+0x15
zio_next_stage(c348dc00,0,c06b829b,c3224260,80,...) at zio_next_stage+0x8e
zio_vdev_io_assess(c348dc00,c40942b8,d624ac94,c31f547e,c348dc00,...) at zio_vdev_io_assess+0x19a
zio_next_stage(c348dc00,c348dc00,c0a4dba0,0,3,...) at zio_next_stage+0x8e
vdev_mirror_io_done(c348dc00,d624ad04,c31c5c61,c348dc00,c32c2e58,...) at vdev_mirror_io_done+0xba
zio_vdev_io_done(c348dc00,c32c2e58,c32c2e58,c32c2e58,3d7a155e,...) at zio_vdev_io_done+0x16
taskq_thread(c32c2e38,d624ad38) at taskq_thread+0x195
fork_exit(c31c5acc,c32c2e38,d624ad38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd624ad70, ebp = 0 ---
lock order reversal:
 1st 0xc40a38c4 zfs:dbuf (zfs:dbuf) @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
 2nd 0xc34d6170 zfs:zn (zfs:zn) @ /usr/src/sys/modules/zfs/../../compat/opensolaris/sys/mutex.h:44
KDB: stack backtrace:
db_trace_self_wrapper(c094d39c) at db_trace_self_wrapper+0x25
kdb_backtrace(0,ffffffff,c0a5ca90,c0a5c608,c09f7bac,...) at kdb_backtrace+0x29
witness_checkorder(c34d6170,9,c321d083,2c) at witness_checkorder+0x586
_sx_xlock(c34d6170,c321d083,2c,c0a9b6f0,c0a4dfa8,...) at _sx_xlock+0x50
znode_pageout_func(c40a3888,c34d6160,c40a3888,d6322a78,c31cd8a5,...) at znode_pageout_func+0xf0
dbuf_evict_user(c40a3888,0,c40a3930,c40a3888,c4040268,...) at dbuf_evict_user+0x37
dbuf_clear(c40a3888,c404031c,d6322aa0,c31d8ca8,c40a3888,...) at dbuf_clear+0x25
dbuf_evict(c40a3888,3,20,c3413880,d6322ab8,...) at dbuf_evict+0xd
dnode_destroy(ca117000,c40a3930,ca0d9b40,c4f9be70,d6322acc,...) at dnode_destroy+0xa0
dnode_buf_pageout(c40a3930,c3413880,c40a3930,d6322aec,c31cd8a5,...) at dnode_buf_pageout+0x38
dbuf_evict_user(c40a3930,0,0,c40a3930,c40a396c,...) at dbuf_evict_user+0x37
dbuf_clear(c40a3930,ca0d9d14,ca0d9ce0,0,1,...) at dbuf_clear+0x25
dnode_evict_dbufs(ca0d9b40,1,c4040268,c4040200,0,...) at dnode_evict_dbufs+0xf0
dmu_objset_evict_dbufs(c403fc40,1,c403fc40,0,c2fd929c,...) at dmu_objset_evict_dbufs+0x10f
zfs_objset_close(c3425000,c3414080,c3416000,0,0,...) at zfs_objset_close+0x15e
zfs_umount(c2fd929c,0,c3342bd0) at zfs_umount+0x266
dounmount(c2fd929c,0,c3342bd0,c06e9872,c3342bd0,...) at dounmount+0x389
unmount(c3342bd0,d6322d00) at unmount+0x22d
syscall(d6322d38) at syscall+0x256
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (22, FreeBSD ELF32, unmount), eip = 0x2811242f, esp = 0xbfbfab6c, ebp = 0xbfbfab98 ---