gmultipath, ses and shared disks / cant seem to share between local nodes

[Teske, Devin]

On Apr 17, 2013, at 3:26 PM, Outback Dingo wrote:

> Ok, maybe im at a loss here in the way my brain is viewing this
> 
> we have a box, its got 2 nodes in the chassis, and 32 sata drives
> attached to a SATA/SAS backplane via 4 (2 per node) LSI MPT SAS2 cards
> should i not logically be seeing 4 controllers X #drive count ??
> 
> camcontrol devlist shows 32 devices, daX,passX and sesX,passX
> 
> <SEAGATE ST33000650SS 0004>        at scbus0 target 9 lun 0 (da0,pass0)
> <STORBRICK-3 1400>        at scbus0 target 10 lun 0 (ses0,pass1)
> <SEAGATE ST33000650SS 0004>        at scbus0 target 11 lun 0 (da1,pass2)
> <STORBRICK-1 1400>        at scbus0 target 12 lun 0 (ses1,pass3)
> <SEAGATE ST33000650SS 0004>        at scbus0 target 13 lun 0 (da2,pass4)
> <STORBRICK-2 1400>        at scbus0 target 14 lun 0 (ses2,pass5)
> <SEAGATE ST33000650SS 0004>        at scbus0 target 15 lun 0 (da3,pass6)
> <STORBRICK-4 1400>        at scbus0 target 16 lun 0 (ses3,pass7)
> <SEAGATE ST33000650SS 0004>        at scbus0 target 17 lun 0 (da4,pass8)
> <STORBRICK-6 1400>        at scbus0 target 18 lun 0 (ses4,pass9)
> <SEAGATE ST33000650SS 0004>        at scbus0 target 19 lun 0 (da5,pass10)
> <STORBRICK-0 1400>        at scbus0 target 20 lun 0 (ses5,pass11)
> <SEAGATE ST33000650SS 0004>        at scbus0 target 21 lun 0 (da6,pass12)
> <STORBRICK-7 1400>        at scbus0 target 22 lun 0 (ses6,pass13)
> <SEAGATE ST33000650SS 0004>        at scbus0 target 23 lun 0 (da7,pass14)
> <STORBRICK-5 1400>        at scbus0 target 24 lun 0 (ses7,pass15)
> <SEAGATE ST9300605SS 0004>         at scbus1 target 0 lun 0 (da8,pass16)
> <SEAGATE ST9300605SS 0004>         at scbus1 target 1 lun 0 (da9,pass17)
> <STORBRICK-3 1400>        at scbus8 target 10 lun 0 (ses8,pass19)
> <SEAGATE ST33000650SS 0004>        at scbus8 target 11 lun 0 (da11,pass20)
> <STORBRICK-1 1400>        at scbus8 target 12 lun 0 (ses9,pass21)
> <SEAGATE ST33000650SS 0004>        at scbus8 target 13 lun 0 (da12,pass22)
> <STORBRICK-2 1400>        at scbus8 target 14 lun 0 (ses10,pass23)
> <SEAGATE ST33000650SS 0004>        at scbus8 target 15 lun 0 (da13,pass24)
> <STORBRICK-4 1400>        at scbus8 target 16 lun 0 (ses11,pass25)
> <SEAGATE ST33000650SS 0004>        at scbus8 target 17 lun 0 (da14,pass26)
> <STORBRICK-6 1400>        at scbus8 target 18 lun 0 (ses12,pass27)
> <SEAGATE ST33000650SS 0004>        at scbus8 target 19 lun 0 (da15,pass28)
> <STORBRICK-0 1400>        at scbus8 target 20 lun 0 (ses13,pass29)
> <SEAGATE ST33000650SS 0004>        at scbus8 target 21 lun 0 (da16,pass30)
> <STORBRICK-7 1400>        at scbus8 target 22 lun 0 (ses14,pass31)
> <SEAGATE ST33000650SS 0004>        at scbus8 target 23 lun 0 (da17,pass32)
> <STORBRICK-5 1400>        at scbus8 target 24 lun 0 (ses15,pass33)
> <USB 2.0 Flash Drive 8.07>         at scbus9 target 0 lun 0 (da18,pass34)
> 
> 
> we would like to create a zpool from all the devices, that in theory if
> nodeA failed
> then nodeB could force import the pool,

gmultipath (which you mention in the subject) is the appropriate tool for this, but there's no need for an import of the pool if you build the pool out of multipath devices. In our experience, we can pull a cable and zfs continues working just fine.

In other words, don't build the pool out of the devices, put a gmultipath label on each device and then use /dev/multipath/LABEL for the zpool devices.


> nodeA and NodeB are attached through
> dual LSI controllers, to the SATA/SAS backplane. but i cant seem to create
> a zpool from sesX or passX devices, i can however create a 16 drive zp0ol
> on either node, from any daX device. what did i miss? ive looked at
> gmirror, and also ses documents. Any insight is appreciated, thanks in
> advance.

gmirror is the wrong tool, gmultipath is what you want. The basic task is to use "gmultipath label FOO da#" to write a cookie on the disk (used to identify new/existing paths during GOEM "taste" events for example).

After you've labeled the da# devices with gmultipath you say "gmultipath status" to see the components of each label and you use "multipath/LABEL" as your disk name when creating the zpool (these correspond directly to /dev/multipath/LABEL, but "zpool create …" or "zpool add …" allow you to omit the leading "/dev").
-- 
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.