AOC-USAS2-L8i zfs panics and SCSI errors in messages

Wed Oct 26 09:48:17 UTC 2011

Hi all,

I tracked down what causes the panics!

I got a tip from aragon and phoenix at the forum about
/etc/periodic/security/100.chksetuid

And to put:
daily_status_security_chksetuid_enable="NO"
into /etc/periodic.conf

I can now run periodic daily without any panics. I´m still wondering about the cause of this, the explanation from the forum was that that phase is too demanding for multi TB systems. But I have several multi TB servers with FreeBSD and ZFS, and none of them has ever behaved this way. Besides, the panic is instantaneous, not degenerative. I imagine that a run like that would start out OK and then just get worse and worse, getting gradually slower and slower until it just wouldn´t cope any more and hang. This feels more like hitting a wall. As if it found something that is couldn´t deal with and has no choice but to panic immediately.

I´m hoping this can be resolved without having to know beforehand about putting stuff into periodic.conf that you couldn´t have anticipated?

@Ken
The hard drives are connected with two breakout cables from each controller to the caddies with CABMS2FN05 from:
http://www.promise.com/single_page_session/page.aspx?region=en-US&m=575&rsn=114

>From controller 1 -> channel 1 -> ports 1,2,3 -> ports 1,2,3 in caddie 1
>From controller 1 -> channel 2 -> ports 1,2 -> ports 4,5 in caddie 1

>From controller 2 -> channel 1 -> ports 1,2,3 -> ports 1,2,3 in caddie 2
>From controller 2 -> channel 2 -> ports 1,2 -> ports 4,5 in caddie 2

Is there any problem with that type of cabling?

These timeouts happens with all harddrives at one time or another, would that mean that all cables are bad? Or of a worse quality perhaps? Regarding the firmware, they are all running version 1AQ10001. I´m going to search for known problems with that, and if you know something, your are welcome to share;)

Best Regards
Karli Sjöberg

25 okt 2011 kl. 21.33 skrev Kenneth D. Merry:

On Thu, Oct 20, 2011 at 13:28:17 +0200, Karli Sj?berg wrote:
Hi,

I?m in the process of vacating a Sun/Oracle system to a another Supermicro/FreeBSD system, doing zfs send/recv between. Two times now, the system has panicked while not doing anything at all, and it?s throwing alot of SCSI/CAM-related errors while doing IO-intensive operations, like send/recv, resilver, and zpool has sometimes reported read/write errors on the hard drives. Best part is that the errors in messages are about all hard drives at one time or another, and they are connected with separate cables, controllers and caddies. Specs:

HW:
1x  Supermicro X8SIL-F
2x  Supermicro AOC-USAS2-L8i
2x  Supermicro CSE-M35T-1B
1x  Intel Core i5 650 3,2GHz
4x  2GB 1333MHZ DDR3 ECC UDIMM
10x SAMSUNG HD204UI (in a raidz2 zpool)
1x  OCZ Vertex 3 240GB (L2ARC)

SW:
# uname -a
FreeBSD server 8.2-STABLE FreeBSD 8.2-STABLE #0: Mon Oct 10 09:12:25 UTC 2011     root at server:/usr/obj/usr/src/sys/GENERIC  amd64
# zpool get version pool1
NAME   PROPERTY  VALUE    SOURCE
pool1  version   28       default[/CODE]

I got the panic from the IPMI KVM:
http://i55.tinypic.com/synpzk.png

In looking at the panic, this is a ZFS panic.  Nothing the disks do should
be able to cause ZFS to panic.  ZFS is panicing in avl_add():

/*
* This is unfortunate.  We want to call panic() here, even for
* non-DEBUG kernels.  In userland, however, we can't depend on anything
* in libc or else the rtld build process gets confused.  So, all we can
* do in userland is resort to a normal ASSERT().
*/
if (avl_find(tree, new_node, &where) != NULL)
#ifdef _KERNEL
panic("avl_find() succeeded inside avl_add()");
#else
ASSERT(0);
#endif

There are certainly timeouts and two terminated IOCs in the log below.  That
does suggest a hardware or driver problem, but it isn't very obvious what
it might be.

I have seen bad behavior with SATA drives behind 3Gb Maxim expanders
talking to 6GB LSI controllers, but your particular configuration does not
involve any expanders, and therefore is not that particular STP issue.

My best guess, and it is a guess, is that either the drives are misbehaving
(i.e. firmware type problem) or you've got a cabling issue.

If you have more hardware available, you might try swapping out the cables
and/or drives to see if you can reproduce the drive errors with a
different setup.  If you swap the drives, I would use a different brand if
you've got them available.

I'm CCing the fs list, perhaps someone there can look at the stack trace
above and figure out what ZFS might be doing.

Again, ZFS should survive any errors from the drives, and the panic above
looks like ZFS is flagging a logic bug somewhere.

And an extract from /var/log/messages:
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): WRITE(10). CDB: 2a 0 6 13 66 f 0 0 f 0
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): CAM status: SCSI Status Error
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): SCSI status: Check Condition
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): WRITE(6). CDB: a 0 1 b2 2 0
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): CAM status: SCSI Status Error
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): SCSI status: Check Condition
Oct 19 17:37:19 fs2-7 kernel: (da6:mps1:0:0:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 859
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 495
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 725
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 722
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI command timeout on device handle 0x000c SMID 438
Oct 19 17:40:38 fs2-7 kernel: mps1: (1:4:0) terminated ioc 804b scsi 0 state c xfer 0
Oct 19 17:40:38 fs2-7 last message repeated 3 times
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 859 complete
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_complete_tm_request: sending deferred task management request for handle 0x0c SMID 495
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 495 complete
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_complete_tm_request: sending deferred task management request for handle 0x0c SMID 725
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 725 complete
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_complete_tm_request: sending deferred task management request for handle 0x0c SMID 722
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 722 complete
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_complete_tm_request: sending deferred task management request for handle 0x0c SMID 438
Oct 19 17:40:38 fs2-7 kernel: mps1: mpssas_abort_complete: abort request on handle 0x0c SMID 438 complete
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): WRITE(10). CDB: 2a 0 6 25 4f 75 0 0 b 0
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): CAM status: SCSI Status Error
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI status: Check Condition
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): WRITE(10). CDB: 2a 0 2d a5 10 ca 0 0 80 0
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): CAM status: SCSI Status Error
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI status: Check Condition
Oct 19 17:40:38 fs2-7 kernel: (da9:mps1:0:4:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:45:40 fs2-7 kernel: (da1:mps0:0:1:0): SCSI command timeout on device handle 0x000a SMID 976
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI command timeout on device handle 0x000a SMID 636
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI command timeout on device handle 0x000a SMID 888
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI command timeout on device handle 0x000a SMID 983
Oct 19 17:45:41 fs2-7 kernel: mps0: (0:1:0) terminated ioc 804b scsi 0 state c xfer 0
Oct 19 17:45:41 fs2-7 last message repeated 2 times
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_abort_complete: abort request on handle 0x0a SMID 976 complete
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0a SMID 636
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_abort_complete: abort request on handle 0x0a SMID 636 complete
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0a SMID 888
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_abort_complete: abort request on handle 0x0a SMID 888 complete
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0a SMID 983
Oct 19 17:45:41 fs2-7 kernel: mps0: mpssas_abort_complete: abort request on handle 0x0a SMID 983 complete
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): WRITE(10). CDB: 2a 0 6 40 a7 2 0 0 3 0
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): CAM status: SCSI Status Error
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI status: Check Condition
Oct 19 17:45:41 fs2-7 kernel: (da1:mps0:0:1:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Oct 19 17:45:42 fs2-7 kernel: (da1:mps0:0:1:0): WRITE(10). CDB: 2a 0 6 40 b0 9 0 0 9 0
Oct 19 17:45:42 fs2-7 kernel: (da1:mps0:0:1:0): CAM status: SCSI Status Error
Oct 19 17:45:42 fs2-7 kernel: (da1:mps0:0:1:0): SCSI status: Check Condition
Oct 19 17:45:42 fs2-7 kernel: (da1:mps0:0:1:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)

What?s going on?

Regards
Karli Sj?berg_______________________________________________
freebsd-scsi at freebsd.org<mailto:freebsd-scsi at freebsd.org> mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org<mailto:freebsd-scsi-unsubscribe at freebsd.org>"

Ken
--
Kenneth Merry
ken at FreeBSD.ORG<mailto:ken at FreeBSD.ORG>

Med Vänliga Hälsningar
-------------------------------------------------------------------------------
Karli Sjöberg
Swedish University of Agricultural Sciences
Box 7079 (Visiting Address Kronåsvägen 8)
S-750 07 Uppsala, Sweden
Phone:  +46-(0)18-67 15 66
karli.sjoberg at slu.se<mailto:karli.sjoberg at adm.slu.se>