ZFS and Glabel

[Jeremy Chadwick]

On Sun, Jun 09, 2013 at 01:25:47PM -0700, Reed A. Cartwright wrote:
> Thanks, it makes sense now.
> 
> Would it make sense to have a script that processes the output of "cam
> devlist -v" to produce such an example output?
> 
> On Sat, Jun 8, 2013 at 11:54 PM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> > On Sat, Jun 08, 2013 at 09:46:18PM -0700, Reed A. Cartwright wrote:
> >> I'm looking at my dmesg.boot to figure out what settings I need to
> >> wire down my HDDs.  I read the cam(4) documentation but I'm not sure I
> >> know what I'm doing.  Any advice would be helpful.
> >>
> >> Let's assume that I want to wire everything down to their current
> >> positions, what should I put in loader.conf?  I'll paste below some of
> >> my hardware configuration and lines from dmesg.boot that I think I
> >> need to look at.
> >>
> >> I have 4 LSI cards in the system: mps0, mps1, mps2, mps3.
> >>
> >> mps0: <LSI SAS2008> port 0xd000-0xd0ff mem
> >> 0xdff3c000-0xdff3ffff,0xdff40000-0xdff7ffff irq 24 at device 0.0 on
> >> pci5
> >> mps1: <LSI SAS2008> port 0xc000-0xc0ff mem
> >> 0xdfe3c000-0xdfe3ffff,0xdfe40000-0xdfe7ffff irq 44 at device 0.0 on
> >> pci4
> >> mps2: <LSI SAS2008> port 0xb000-0xb0ff mem
> >> 0xdfd3c000-0xdfd3ffff,0xdfd40000-0xdfd7ffff irq 32 at device 0.0 on
> >> pci3
> >> mps3: <LSI SAS2008> port 0xe000-0xe0ff mem
> >> 0xdbf3c000-0xdbf3ffff,0xdbf40000-0xdbf7ffff irq 56 at device 0.0 on
> >> pci65
> >>
> >> I have drives attached to two of those cards:
> >>
> >> da0 at mps0 bus 0 scbus0 target 0 lun 0
> >> da1 at mps0 bus 0 scbus0 target 1 lun 0
> >> da2 at mps0 bus 0 scbus0 target 2 lun 0
> >> da3 at mps0 bus 0 scbus0 target 3 lun 0
> >> da4 at mps0 bus 0 scbus0 target 4 lun 0
> >> da5 at mps0 bus 0 scbus0 target 5 lun 0
> >> da6 at mps0 bus 0 scbus0 target 6 lun 0
> >> da7 at mps0 bus 0 scbus0 target 7 lun 0
> >
> >> da8 at mps3 bus 0 scbus9 target 0 lun 0
> >> da9 at mps3 bus 0 scbus9 target 1 lun 0
> >> da10 at mps3 bus 0 scbus9 target 2 lun 0
> >> da11 at mps3 bus 0 scbus9 target 3 lun 0
> >> da12 at mps3 bus 0 scbus9 target 4 lun 0
> >>
> >> {snip}
> >
> > As usual, the situation is insane because you have so many controllers
> > on the system (more than just mps(4)) -- specifically 11 separate
> > controllers or systems using CAM (hence scbus0 to scbus10).
> >
> > Below is for mps(4).  If you want to wire down ahci(4), things are
> > a bit different, but you can read this post of mine:
> >
> > http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071851.html
> >
> > Enjoy:
> >
> > hint.scbus.0.at="mps0"
> > hint.scbus.1.at="mps1"
> > hint.scbus.2.at="mps2"
> > hint.scbus.9.at="mps3"
> > hint.da.0.at="scbus0"
> > hint.da.1.at="scbus0"
> > hint.da.2.at="scbus0"
> > hint.da.3.at="scbus0"
> > hint.da.4.at="scbus0"
> > hint.da.5.at="scbus0"
> > hint.da.6.at="scbus0"
> > hint.da.7.at="scbus0"
> > hint.da.8.at="scbus9"
> > hint.da.9.at="scbus9"
> > hint.da.10.at="scbus9"
> > hint.da.11.at="scbus9"
> > hint.da.12.at="scbus9"
> > hint.da.13.at="scbus9"
> > hint.da.14.at="scbus9"
> > hint.da.15.at="scbus9"
> > hint.da.16.at="scbus1"
> > hint.da.17.at="scbus1"
> > hint.da.18.at="scbus1"
> > hint.da.19.at="scbus1"
> > hint.da.20.at="scbus1"
> > hint.da.21.at="scbus1"
> > hint.da.22.at="scbus1"
> > hint.da.23.at="scbus1"
> > hint.da.24.at="scbus2"
> > hint.da.25.at="scbus2"
> > hint.da.26.at="scbus2"
> > hint.da.27.at="scbus2"
> > hint.da.28.at="scbus2"
> > hint.da.29.at="scbus2"
> > hint.da.30.at="scbus2"
> > hint.da.31.at="scbus2"
> > hint.da.0.target="0"
> > hint.da.1.target="1"
> > hint.da.2.target="2"
> > hint.da.3.target="3"
> > hint.da.4.target="4"
> > hint.da.5.target="5"
> > hint.da.6.target="6"
> > hint.da.7.target="7"
> > hint.da.8.target="0"
> > hint.da.9.target="1"
> > hint.da.10.target="2"
> > hint.da.11.target="3"
> > hint.da.12.target="4"
> > hint.da.13.target="5"
> > hint.da.14.target="6"
> > hint.da.15.target="7"
> > hint.da.16.target="0"
> > hint.da.17.target="1"
> > hint.da.18.target="2"
> > hint.da.19.target="3"
> > hint.da.20.target="4"
> > hint.da.21.target="5"
> > hint.da.22.target="6"
> > hint.da.23.target="7"
> > hint.da.24.target="0"
> > hint.da.25.target="1"
> > hint.da.26.target="2"
> > hint.da.27.target="3"
> > hint.da.28.target="4"
> > hint.da.29.target="5"
> > hint.da.30.target="6"
> > hint.da.31.target="7"

The script would be ugly and require one-offs per driver.  For example,
look at your "camcontrol devlist -v" output with regards to mps1 and
mps2.  There's no indication of what the bus #, target #, or lun #
should be because there are no disks on the controller.  I made an
educated guess based off of mps0/mps3 and previous familiarity (on the
lists -- I've never used one of these controllers) with mps(4).

Had you shown me "camcontrol devlist -v" output with only 1 controller
and 1 disk, I would have had to go purely off of what I've seen in the
past.

The behaviour could change as well, depending on firmware upgrades or
driver changes (many of the storage drivers in FreeBSD in the past 3-4
years have gone through massive changes), or even operational mode (RAID
vs. non-RAID), where the target then is always 0 but the lun #
increases, or maybe the bus number, or maybe a combination.

If someone really wants to take a stab at writing some script that does
this, be my guest, but I definitely don't.  :-)  There's just too many
one-offs or assumptions that have to be made which a human mind +
experience can do more reliably, IMO.

Because remember: the last thing you want to do is modify loader.conf
for wiring down and botch it/break it.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |