tying down adaX to physical interfaces

[Jeremy Chadwick]

On Fri, Mar 25, 2011 at 05:57:10PM -0700, Michael DeMan wrote:
> Yeah - this is what I was looking for.  I can morph it for mvsch.
> 
> In regards to prior post...
> The glabel solution works too, but again gets complicated when a regular staff person just needs to swap out a drive, has the disk drive in hand (which could either not have a glabel on it or maybe has one from somewhere else by accident), and just needs to replace a drive.
> 
> Thanks!
> 
> - mike
> 
> On Mar 25, 2011, at 5:38 PM, Kostik Belousov wrote:
> 
> > On Fri, Mar 25, 2011 at 05:21:50PM -0700, Michael DeMan wrote:
> >> Hi All,
> >> 
> >> I seem to recall that there is a way to do this, but can no longer google it.
> >> 
> >> Basically, for NCQ support with SATA devices we are using the 'ada' driver, which of course has SCSI like behavior.
> >> 
> >> With two 8-port disk controllers in the system, we end up in the uncomfortable situation where if the box is rebooted with an 'earlier' drive in the list, the system boots up with different drives mapped to adaX.  This becomes more of a problem with multiple drive controllers and not all the ports populated with drives at the start (i.e, add more drives later).
> >> 
> >> In our case, we have two Marvell controllers, and are doing ZFS mirroring with drives off each controller.
> >> 
> >> We end up with mvsch.0 through mvsch.15 showing up (16 controller interfaces), but with only 10 drives right now, we end up with ada0 through ada9.  If we lose a drive and reboot, then we end up with the upper ones each being ada(X-1), etc.  If we add more drives, the situation becomes even more confusing.
> >> 
> >> Is there a way to force at boot time a mapping from the mvsch interface to the specific ada disk that the operating system sees?  Pretty sure I saw something about this last summer but can't seem to find it now.
> >> 
> >> What we would like to have, with 10 disks, is them showing up as:
> >> ada0, ada1, ada2, ada3, ada4
> >> ada8, ada9, ada10, ada11, ada12
> >> 
> >> If we add another pair of disks in the future, we would know where to physically insert them for ada5 and ada13, etc.
> > 
> > I use the following stanza in /boot/device.hints for machine with intel
> > on-board ahci and siis in pcie:
> > hint.scbus.0.at="ahcich0"
> > hint.ada.0.at="scbus0"
> > hint.scbus.1.at="ahcich1"
> > hint.ada.1.at="scbus1"
> > hint.scbus.2.at="ahcich2"
> > hint.ada.2.at="scbus2"
> > hint.scbus.3.at="ahcich3"
> > hint.ada.3.at="scbus3"
> > hint.scbus.4.at="ahcich4"
> > hint.ada.4.at="scbus4"
> > hint.scbus.5.at="siisch0"
> > hint.ada.5.at="scbus5"
> > hint.scbus.6.at="siisch1"
> > hint.ada.6.at="scbus6"
> > 
> > You should get an idea from this.

Why would the disk number change on a drive swap?  This doesn't happen
on my Intel ICH9R controller on *any* Supermicro X7xxx or Supermicro
PDxxx board I have access to.

The ONLY TIME I have seen this happen is in the following situation:

1) State:
physical SATA port 0 <--> hot-swap backplane port 0 <--> disk ada0
physical SATA port 1 <--> hot-swap backplane port 1 <--> disk ada1
physical SATA port 2 <--> hot-swap backplane port 2 <--> empty
physical SATA port 3 <--> hot-swap backplane port 3 <--> disk ada2
physical SATA port 4 <--> hot-swap backplane port 4 <--> disk ada3
physical SATA port 5 <--> hot-swap backplane port 5 <--> disk ada4

2) Administrator adds disk to backplane port 2.  Disk appears as ada5.

3) Machine gets shut off, rebooted, whatever.

4) Kernel starts.  State then becomes:
physical SATA port 0 <--> hot-swap backplane port 0 <--> disk ada0
physical SATA port 1 <--> hot-swap backplane port 1 <--> disk ada1
physical SATA port 2 <--> hot-swap backplane port 2 <--> disk ada2
physical SATA port 3 <--> hot-swap backplane port 3 <--> disk ada3
physical SATA port 4 <--> hot-swap backplane port 4 <--> disk ada4
physical SATA port 5 <--> hot-swap backplane port 5 <--> disk ada5

I should note that in my experience (on multiple systems/boards/chassis)
hot-swapping a disk *does not* change the disk number.  E.g. yanking
ada3 results in the SATA channel seeing a disconnection, and an
insertion results in the SATA channel seeing an insertion of ada3.  Two
weeks ago I did this a good 15-16 times while I zeroed out all the old
SATA disks I had laying around my flat.

Anyway, the way I've addressed the problem described is: leave your
empty disks at the "end of the SATA port chain" (I'm using the word
chain incorrectly here; I am well aware SATA is 1:1 and not chained).

If you *have* to move disks around/empty bays out which were previously
filled, you should boot into single-user and fix things there before
actually bringing up multi-user mode.

I've mentioned this before -- and I'm certain others will point out that
said problem was fixed, etc. etc. (done so before, and I've simply
forgotten) -- but there are complexities introduced into the fray when
using glabel(8) and similar utilities.  There have been complaints on
the lists about it.  I've had a very, very long week and can't be
bothered to dig up the evidence; sorry.

This is one great thing about ZFS and disk tasting -- it doesn't matter
if the disks all "shift up by one" (number-wise), it figures it out
and does the Right Thing(tm) with pools as a result.  This is why I tend
to use ZFS on non-OS disks, that way if the above situation happens,
I'm not fighting with the system after a disk replacement.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |