MPS

[Karl Denninger]

On 9/29/2014 7:17 PM, Kevin Oberman wrote:
> On Mon, Sep 29, 2014 at 1:25 PM, Steven Hartland <killing at multiplay.co.uk>
> wrote:
>
>> I seem to remember detection being made parallel for speed, as there
>> can be lots of drives. The Avago guys may be able to shed more light.
>>
>>    Regards
>>    Steve
>>
>> ----- Original Message ----- From: "Karl Denninger" <karl at denninger.net>
>> To: <freebsd-stable at freebsd.org>
>> Sent: Monday, September 29, 2014 9:04 PM
>> Subject: MPS
>>
>>
>>
>> Has anyone found a set of configuration settings for the mps driver (LSI
>> SAS series cards) that gets the following situation under control?
>>
>> I have a machine here with one of these cards and a SAS expander.  The
>> expander has 16 ports, the card has 8 -- one connector attached to the
>> expander.  So I have 20 potential drives associated with this adapter.
>>
>> The issue is that the driver makes utterly no sense when it comes to
>> assigning drive designations.  In theory it should enumerate in series;
>> that is, target 0 on the card should be da0, target 1 da1, etc.  This
>> assumes there are no gaps in the attached drives; if there are then
>> things may move around, which is why some people wire drives.
>>
>> Ok so far, but it doesn't behave that way!
>>
>> This is the excerpt of what it returns:
>>
>> root at Dbms-10:/boot # dmesg|grep mps
>> mps0: <LSI SAS2008> port 0xc000-0xc0ff mem
>> 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3
>> mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd
>> mps0: IOCCapabilities:
>> 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
>> ses0 at mps0 bus 0 scbus0 target 8 lun 0
>> da0 at mps0 bus 0 scbus0 target 0 lun 0
>> da1 at mps0 bus 0 scbus0 target 1 lun 0
>> da2 at mps0 bus 0 scbus0 target 2 lun 0
>> da3 at mps0 bus 0 scbus0 target 3 lun 0
>> *da4 at mps0 bus 0 scbus0 target 6 lun 0**
>> **da5 at mps0 bus 0 scbus0 target 7 lun 0*
>>
>> Those last two are the drives the card identifies as being on targets 0
>> and 1 when looked at in the BIOS.  The other four drives (0-3) are on
>> the SAS expander -- and there's a two drive "gap" in the targets
>> identified as well which has zippo for logic associated with it.
>>
>> Even more-oddly if I swap the plugs -- that is, plug the two boot disks
>> into the OTHER connector (putatively targets 4-7) and put the SAS
>> expander on the 0-3 port connector the BIOS *does* show the target shift
>> but the machine, when it boots, still places those two drives on targets
>> 6 and 7!
>>
>> There is nothing in /boot/loader.conf or /boot/device.hints that relates
>> to these devices at all.
>>
>> I would like to wire down the various ports so Drive Bay #x corresponds
>> to da[x] but how can you when you don't know what order they will come
>> up in predicated on either the physical port they're occupying or, for
>> that matter, what the card's BIOS shows them as?
>>
>> I looked in the driver's man page and saw nothing related to this; I get
>> around it by labeling the drives and then using "glabel list" to see
>> what da[x] number it grabbed if I need to pull a disk while the machine
>> is up, but this is a rather-unfriendly way to handle that.
>>
>> --
>> Karl Denninger
>> karl at denninger.net <mailto:karl at denninger.net>
>> /The Market Ticker/
>>
> Karl,
>
> This problem seems to crop up with  some controllers, especially when port
> expanders are used.
>
> If at all possible, I really recommend labeling them and using the labels
> in lieu of device names. I stopped using device names some time ago just
> because of this. I do recommend using the labeling for the file system
> involved as opposed to glabel, if it has one. Certainly UFS does and I
> thought ZFS did, as well, but I am less sure of that as ZFS started to
> stabilize about the time I stopped dealing with big systems. I still find
> UFS quite adequate for my baby server and laptop. I do use glabel for swap.
>
> In any case, using labeling will give you stable names to put into things
> like your fstab and not make you worry about the actual device number. And,
> of course, talking to the manufacturer is also a good idea, espacially when
> they don't say "FreeBSD? We don't support that.".
> --
> R. Kevin Oberman, Network Engineer, Retired
> E-mail: rkoberman at gmail.com
>
Yeah, that's been my answer along with lettering the drive carriers but
it pisses me off.  Ah well if that's how it is then that's how it is. :-)

Glabel works fine across the board provided you put it on a slice rather
than a disk, which I do anyway so as to maintain "nice-nice" with AF
(4k) disks along with leaving a small amount of space at the end
unallocated, so if by some chance a different brand replacement drive
isn't quite the same number of sectors it can be dropped into the pool
as a replacement.

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2711 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140929/bd72afb6/attachment.bin>