CAM hangs in 9-STABLE? [Was: NFS/ZFS hangs after upgrading from 9.0-RELEASE to -STABLE]
Reed A. Cartwright
cartwright at asu.edu
Wed Jan 16 00:07:40 UTC 2013
I don't know if this is relevant or not, but I deadlock was recently
fixed in the VFS code:
http://svnweb.freebsd.org/base?view=revision&revision=244795
On Tue, Jan 15, 2013 at 12:55 PM, olivier <olivier777a7 at gmail.com> wrote:
> Dear All,
> Still experiencing the same hangs I reported earlier with 9.1. I've been
> running a kernel with WITNESS enabled to provide more information.
>
> During an occurrence of the hang, running show alllocks gave
>
> Process 25777 (sysctl) thread 0xfffffe014c5b2920 (102567)
> exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff811e34c0) locked @
> /usr/src/sys/dev/usb/usb_transfer.c:3171
> Process 25750 (sshd) thread 0xfffffe015a688000 (104313)
> exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe0204e0bb98) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> Process 24922 (cnid_dbd) thread 0xfffffe0187ac4920 (103597)
> shared lockmgr zfs (zfs) r = 0 (0xfffffe0973062488) locked @
> /usr/src/sys/kern/vfs_syscalls.c:3591
> Process 24117 (sshd) thread 0xfffffe07bd914490 (104195)
> exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xfffffe0204e0a8f0) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> Process 1243 (java) thread 0xfffffe01ca85d000 (102704)
> exclusive sleep mutex pmap (pmap) r = 0 (0xfffffe015aec1440) locked @
> /usr/src/sys/amd64/amd64/pmap.c:4840
> exclusive rw pmap pv global (pmap pv global) r = 0 (0xffffffff81409780)
> locked @ /usr/src/sys/amd64/amd64/pmap.c:4802
> exclusive sleep mutex vm page (vm page) r = 0 (0xffffffff813f0a80) locked @
> /usr/src/sys/vm/vm_object.c:1128
> exclusive sleep mutex vm object (standard object) r = 0
> (0xfffffe01458e43a0) locked @ /usr/src/sys/vm/vm_object.c:1076
> shared sx vm map (user) (vm map (user)) r = 0 (0xfffffe015aec1388) locked @
> /usr/src/sys/vm/vm_map.c:2045
> Process 994 (nfsd) thread 0xfffffe015a0df000 (102426)
> shared lockmgr zfs (zfs) r = 0 (0xfffffe0c3b505878) locked @
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1760
> Process 994 (nfsd) thread 0xfffffe015a0f8490 (102422)
> exclusive lockmgr zfs (zfs) r = 0 (0xfffffe02db3b3e60) locked @
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1760
> Process 931 (syslogd) thread 0xfffffe015af18920 (102365)
> shared lockmgr zfs (zfs) r = 0 (0xfffffe0141dd6680) locked @
> /usr/src/sys/kern/vfs_syscalls.c:3591
> Process 22 (syncer) thread 0xfffffe0125077000 (100279)
> exclusive lockmgr syncer (syncer) r = 0 (0xfffffe015a2ff680) locked @
> /usr/src/sys/kern/vfs_subr.c:1809
>
> I don't have full "show lockedvnods" output because the output does not get
> captured by ddb after using "capture on", it doesn't fit on a single
> screen, and doesn't get piped into a "more" equivalent. What I did manage
> to get (copied by hand, typos possible) is:
>
> 0xfffffe0c3b5057e0: 0xfffffe0c3b5057e0: tag zfs, type VREG
> tag zfs, type VREG
> usecount 1, writecount 0, refcount 1 mountedhere 0
> usecount 1, writecount 0, refcount 1 mountedhere 0
> flags (VI_ACTIVE)
> flags (VI_ACTIVE)
> v_object 0xfffffe089bc1b828 ref 0 pages 0
> v_object 0xfffffe089bc1b828 ref 0 pages 0
> lock type zfs: SHARED (count 1)
> lock type zfs: SHARED (count 1)
>
> 0xfffffe02db3b3dc8: 0xfffffe02db3b3dc8: tag zfs, type VREG
> tag zfs, type VREG
> usecount 6, writecount 0, refcount 6 mountedhere 0
> usecount 6, writecount 0, refcount 6 mountedhere 0
> flags (VI_ACTIVE)
> flags (VI_ACTIVE)
> v_object 0xfffffe0b79583ae0 ref 0 pages 0
> v_object 0xfffffe0b79583ae0 ref 0 pages 0
> lock type zfs: EXCL by thread 0xfffffe015a0f8490 (pid 994)
> lock type zfs: EXCL by thread 0xfffffe015a0f8490 (pid 994)
> with exclusive waiters pending
> with exclusive waiters pending
>
> The output of show witness is at http://pastebin.com/eSRb3FEu
>
> The output of alltrace is at http://pastebin.com/X1LruNrf (a number of
> threads are stuck in zio_wait, none I can find in zio_interrupt, and
> according to gstat and disks eventually going to sleep all disk IO seems to
> be stuck for good; I think Andriy explained earlier that these criteria
> might indicate this is a ZFS hang).
>
> The output of show geom is at http://pastebin.com/6nwQbKr4
>
> The output of vmstat -i is at http://pastebin.com/9LcZ7Mi0 Interrupts are
> occurring at a normal rate during the hang, as far as I can tell.
>
> Any help would be greatly appreciated.
> Thanks
> Olivier
> PS: my kernel was compiled from 9-STABLE from December, with CAM and ahci
> from 9.0 (in the hope it would fix the hangs I was experiencing in plain
> 9-STABLE; obviously the hangs are still occurring). The rest of my
> configuration is the same as posted earlier.
>
> On Mon, Dec 24, 2012 at 9:42 PM, olivier <olivier777a7 at gmail.com> wrote:
>
>> Dear All
>> It turns out that reverting to an older version of the mps driver did not
>> fix the ZFS hangs I've been struggling with in 9.1 and 9-STABLE after all
>> (they just took a bit longer to occur again, possibly just by chance). I
>> followed steps along lines suggested by Andriy to collect more information
>> when the problem occurs. Hopefully this will help figure out what's going
>> on.
>>
>> As far as I can tell, what happens is that at some point IO operations to
>> a bunch of drives that belong to different pools get stuck. For these
>> drives, gstat shows no activity but 1 pending operation, as such:
>>
>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
>> ms/d %busy Name
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da1
>>
>> I've been running gstat in a loop (every 100s) to monitor the machine.
>> Just before the hang occurs, everything seems fine (see full gstat output
>> below). Right after the hang occurs a number of drives seem stuck (see full
>> gstat output below). Notably, some stuck drives are seen through the mps
>> driver and others through the mpt driver. So the problem doesn't seem to be
>> driver-specific. I have had the problem occur (at a lower frequency) on
>> similar machines that don't use the mpt driver (and only have 1 disk
>> provided through mps), so the problem doesn't seem to be caused by the mpt
>> driver (and is likely not caused by defective hardware). Since based on the
>> information I provided earlier Andriy thinks the problem might not
>> originate in ZFS, perhaps that means that the problem is in the CAM layer?
>>
>> camcontrol tags -v (as suggested by Andriy) in the hung state shows for
>> example
>>
>> (pass56:mpt1:0:8:20): dev_openings 254
>> (pass56:mpt1:0:8:20): dev_active 1
>> (pass56:mpt1:0:8:20): devq_openings 254
>> (pass56:mpt1:0:8:20): devq_queued 0
>> (pass56:mpt1:0:8:20): held 0
>> (pass56:mpt1:0:8:20): mintags 2
>> (pass56:mpt1:0:8:20): maxtags 255
>> (I'm not providing full camcontrol tags output below because I couldn't
>> get it to run during the specific hang I documented most thoroughly; the
>> example above is from a different occurrence of the hang).
>>
>> The buses don't seem completely frozen: if I manually remove drives while
>> the machine is hanging, that's picked up by the mpt driver, which prints
>> out corresponding messages to the console. But camcontrol reset all or
>> rescan all don't seem to do anything.
>>
>> I've tried reducing vfs.zfs.vdev.min_pending and vfs.zfs.vdev.max_pending
>> to 1, to no avail.
>>
>> Any suggestions to resolve this problem, work around it, or further
>> investigate it would be greatly appreciated!
>> Thanks a lot
>> Olivier
>>
>> Detailed information:
>>
>> Output of procstat -a -kk when the machine is hanging is available at
>> http://pastebin.com/7D2KtT35 (not putting it here because it's pretty
>> long)
>>
>> dmesg is available at http://pastebin.com/9zJQwWJG . Note that I'm using
>> LUN masking, so the "illegal requests" reported aren't really errors. Maybe
>> one day if I get my problems sorted out I'll use geom multipathing instead.
>>
>> My kernel config is
>> include GENERIC
>> ident MYKERNEL
>>
>> options IPSEC
>> device crypto
>>
>> options OFED # Infiniband protocol
>>
>> device mlx4ib # ConnectX Infiniband support
>> device mlxen # ConnectX Ethernet support
>> device mthca # Infinihost cards
>> device ipoib # IP over IB devices
>>
>> options ATA_CAM # Handle legacy controllers with CAM
>> options ATA_STATIC_ID # Static device numbering
>>
>> options KDB
>> options DDB
>>
>>
>>
>> Full output of gstat just before the hang (at most 100s before the hang):
>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
>> ms/d %busy Name
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da0
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da2/da2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da0/da0
>> 1 85 48 79 4.7 35 84 0.5 0 0
>> 0.0 24.3 da1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da1/da1
>> 1 83 47 77 4.3 34 79 0.5 0 0
>> 0.0 22.1 da4
>> 1 1324 1303 21433 0.6 19 42 0.7 0 0
>> 0.0 79.8 da3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da5
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da6
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da7
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da8
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da9
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da10
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da11
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da12
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da13
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da14
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da15
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da16
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da17
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da18
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da19
>> 0 97 57 93 3.5 38 84 0.3 0 0
>> 0.0 21.3 da20
>> 0 85 47 69 3.3 36 86 0.4 0 0
>> 0.0 16.8 da21
>> 0 1666 1641 18992 0.3 23 43 0.4 0 0
>> 0.0 57.9 da22
>> 0 93 55 98 3.5 36 87 0.4 0 0
>> 0.0 20.6 da23
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da24
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da25
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da26
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da27
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da28
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da29
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da30
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da31
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da32
>> 0 1200 0 0 0.0 1198 11751 0.6 0 0
>> 0.0 67.3 da33
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da34
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da35
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da36
>> 0 81 44 67 2.0 35 84 0.3 0 0
>> 0.0 10.1 da37
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da38
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da39
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da40
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da41
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da42
>> 1 1020 999 22028 0.8 19 42 0.7 0 0
>> 0.0 84.8 da43
>> 0 1050 1029 23479 0.8 19 47 0.7 0 0
>> 0.0 83.3 da44
>> 1 1006 984 22758 0.8 21 46 0.6 0 0
>> 0.0 84.8 da45
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da46
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da47
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da48
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da49
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da50
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 cd0
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da4/da4
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da3/da3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da5/da5
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da6/da6
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da7/da7
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da8/da8
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da9/da9
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da10/da10
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da11/da11
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da12/da12
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da13/da13
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da14/da14
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da15/da15
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da16/da16
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da17/da17
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da18/da18
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da19/da19
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da20/da20
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da21/da21
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da22/da22
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da23/da23
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da24/da24
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da25/da25
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da26/da26
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 PART/da26/da26
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da26p1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da26p2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da26p3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da27/da27
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da28/da28
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da29/da29
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da30/da30
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da31/da31
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da32/da32
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da33/da33
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da34/da34
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da35/da35
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da36/da36
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da37/da37
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da38/da38
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da39/da39
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da40/da40
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da41/da41
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da42/da42
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da43/da43
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da44/da44
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da45/da45
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da46/da46
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da47/da47
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da48/da48
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da49/da49
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da50/da50
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/cd0/cd0
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da26p1/da26p1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da26p2/da26p2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 LABEL/da26p1/da26p1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 gptid/84d4487b-34e3-11e2-b773-00259058949a
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da26p3/da26p3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 LABEL/da26p2/da26p2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 gptid/b4255780-34e3-11e2-b773-00259058949a
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0
>> DEV/gptid/84d4487b-34e3-11e2-b773-00259058949a/gptid/84d4487b-34e3-11e2-b773-00259058949a
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da25
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0
>> DEV/gptid/b4255780-34e3-11e2-b773-00259058949a/gptid/b4255780-34e3-11e2-b773-00259058949a
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da40
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da41
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da26p3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da29
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da30
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da24
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da6
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da7
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da16
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da17
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da20
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da21
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da37
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da23
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da4
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da43
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da44
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da22
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da33
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da45
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da3
>>
>>
>> Full output of gstat just after the hang (at most 100s after the hang):
>> L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps
>> ms/d %busy Name
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da0
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da2/da2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da0/da0
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da1/da1
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da4
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da5
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da6
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da7
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da8
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da9
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da10
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da11
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da12
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da13
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da14
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da15
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da16
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da17
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da18
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da19
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da20
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da21
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da22
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da23
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da24
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da25
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da26
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da27
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da28
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da29
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da30
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da31
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da32
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da33
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da34
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da35
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da36
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da37
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da38
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da39
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da40
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da41
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da42
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da43
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da44
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da45
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da46
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da47
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da48
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da49
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da50
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 cd0
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da4/da4
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da3/da3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da5/da5
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da6/da6
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da7/da7
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da8/da8
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da9/da9
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da10/da10
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da11/da11
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da12/da12
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da13/da13
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da14/da14
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da15/da15
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da16/da16
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da17/da17
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da18/da18
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da19/da19
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da20/da20
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da21/da21
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da22/da22
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da23/da23
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da24/da24
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da25/da25
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da26/da26
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 PART/da26/da26
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da26p1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da26p2
>> 1 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 da26p3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da27/da27
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da28/da28
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da29/da29
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da30/da30
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da31/da31
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da32/da32
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da33/da33
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da34/da34
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da35/da35
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da36/da36
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da37/da37
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da38/da38
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da39/da39
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da40/da40
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da41/da41
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da42/da42
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da43/da43
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da44/da44
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da45/da45
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da46/da46
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da47/da47
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da48/da48
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da49/da49
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da50/da50
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/cd0/cd0
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da26p1/da26p1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da26p2/da26p2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 LABEL/da26p1/da26p1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 gptid/84d4487b-34e3-11e2-b773-00259058949a
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 DEV/da26p3/da26p3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 LABEL/da26p2/da26p2
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 gptid/b4255780-34e3-11e2-b773-00259058949a
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0
>> DEV/gptid/84d4487b-34e3-11e2-b773-00259058949a/gptid/84d4487b-34e3-11e2-b773-00259058949a
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da25
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0
>> DEV/gptid/b4255780-34e3-11e2-b773-00259058949a/gptid/b4255780-34e3-11e2-b773-00259058949a
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da40
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da41
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da26p3
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da29
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da30
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da24
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da6
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da7
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da16
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da17
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da20
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da21
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da37
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da23
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da1
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da4
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da43
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da44
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da22
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da33
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da45
>> 0 0 0 0 0.0 0 0 0.0 0 0
>> 0.0 0.0 ZFS::VDEV/zfs::vdev/da3
>>
>>
>> On Thu, Dec 13, 2012 at 10:14 PM, olivier <olivier777a7 at gmail.com> wrote:
>>
>>> For what it's worth, I think I might have solved my problem by reverting
>>> to an older version of the mps driver. I checked out a recent version of
>>> 9-STABLE and reversed the changes in
>>> http://svnweb.freebsd.org/base?view=revision&revision=230592 (perhaps
>>> there was a simpler way of reverting to the older mps driver). So far so
>>> good, no hang even when hammering the file system.
>>>
>>> This does not conclusively prove that the new LSI mps driver is at fault,
>>> but that seems to be a likely explanation.
>>>
>>> Thanks to everybody who pointed me in the right direction. Hope this
>>> helps others who run into similar problems with 9.1
>>> Olivier
>>>
>>>
>>> On Thu, Dec 13, 2012 at 10:14 AM, olivier <olivier777a7 at gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Dec 13, 2012 at 9:54 AM, Andriy Gapon <avg at freebsd.org> wrote:
>>>>
>>>>> Google for "zfs deadman". This is already committed upstream and I
>>>>> think that it
>>>>> is imported into FreeBSD, but I am not sure... Maybe it's imported
>>>>> just into the
>>>>> vendor area and is not merged yet.
>>>>>
>>>>
>>>> Yes, that's exactly what I had in mind. The logic for panicking makes
>>>> sense.
>>>> As far as I can tell you're correct that deadman is in the vendor area
>>>> but not merged. Any idea when it might make it into 9-STABLE?
>>>> Thanks
>>>> Olivier
>>>>
>>>>
>>>>
>>>>
>>>>> So, when enabled this logic would panic a system as a way of letting
>>>>> know that
>>>>> something is wrong. You can read in the links why panic was selected
>>>>> for this job.
>>>>>
>>>>> And speaking FreeBSD-centric - I think that our CAM layer would be a
>>>>> perfect place
>>>>> to detect such issues in non-ZFS-specific way.
>>>>>
>>>>> --
>>>>> Andriy Gapon
>>>>>
>>>>
>>>>
>>>
>>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
--
Reed A. Cartwright, PhD
Assistant Professor of Genomics, Evolution, and Bioinformatics
School of Life Sciences
Center for Evolutionary Medicine and Informatics
The Biodesign Institute
Arizona State University
More information about the freebsd-stable
mailing list