multi-homed drive SATA drive affiliations

[David Ford]

Hello,

I suspect I'm missing something obvious here, but I can't quite think what. I have a set of SAS disk shelves which are each dual homed to a pair of FreeBSD 10.1 systems. These shelves contain a bunch of SAS drives, which as expected work fine in this configuration, and can be accessed from whichever host wishes to use them - the end workload is ZFS, so each disk will be assigned to a single host, and the second host is for failover purposes.

The problem is that for SATA drives (the idea is the use SSDs for L2ARC/ZIL) SATA affiliations don't appear to be having the desired effect, the disks get assigned to one host when they are attached, and whilst I can clear the affiliations on that host, there seems no way to make the second host see the drive once the affiliations have been cleared.

On the host where the drive appears:

[root at backup-san1 ~]# camcontrol devlist
<IBM-ESXS HUS723030ALS64 J210>     at scbus0 target 9 lun 0 (pass0,da0)
<IBM-ESXS HUS723030ALS64 J3K7>     at scbus0 target 10 lun 0 (pass1,da1)
<IBM-ESXS ST33000650SS BC36>       at scbus0 target 11 lun 0 (pass2,da2)
<IBM-ESXS EXP3512 0366>            at scbus0 target 12 lun 0 (pass3,ses0)
<ATA ST1000DM003-1ER1 CC45>        at scbus0 target 13 lun 0 (da33,pass36)

The last line is the ATA drive in question,

On the other host:

root at backup-san-02:~ # camcontrol devlist
<IBM-ESXS EXP3512 0366>            at scbus0 target 9 lun 0 (pass0,ses0)
<IBM-ESXS ST33000650SS BC36>       at scbus0 target 10 lun 0 (pass1,da0)
<IBM-ESXS HUS723030ALS64 J210>     at scbus0 target 11 lun 0 (pass2,da1)
<IBM-ESXS HUS723030ALS64 J3K7>     at scbus0 target 12 lun 0 (pass3,da2)

I have tried using both:

camcontrol --phy=2 --op=ca /dev/ses0
smp_phy_control --phy=2 --op=ca /dev/ses0

on the working host, to clear the affiliation, and smp_rep_phy_sata implies this has had the desired effect:

[root at backup-san1 ~]# smp_rep_phy_sata --phy=2 /dev/ses0
Report phy SATA response:
  expander change count: 71
  phy identifier: 2
  STP I_T nexus loss occurred: 0
  affiliations supported: 1
  affiliation valid: 0
  STP SAS address: 0x50080e53c2b8f002
  register device to host FIS:
    34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
  affiliated STP initiator SAS address: 0x0
  STP I_T nexus loss SAS address: 0x0
  affiliation context: 0
  current affiliation contexts: 0

However, nothing I've tried on the second host then allows it to see it:

root at backup-san-02:~ # smp_phy_control --phy=2 --op=tspss /dev/ses0
root at backup-san-02:~ # smp_rep_phy_sata --phy=2 /dev/ses0
Report phy SATA result: Phy does not support SATA
root at backup-san-02:~ # smp_phy_control --phy=2 --op=lr /dev/ses0
root at backup-san-02:~ # camcontrol reset
root at backup-san-02:~ # camcontrol rescan
root at backup-san-02:~ # smp_rep_phy_sata --phy=2 /dev/ses0
Report phy SATA result: Phy does not support SATA

I've even tried disabling the device from the first host, but it still behaves in an identical fashion.

Does anyone happen to know what I've done wrong?

Thanks

David
-- 
David Ford
IT Manager, School of Geography and the Environment
For general IT Support queries please contact itsupport at ouce.ox.ac.uk
Telephone: +44 1865 285089