FreeBSD & Hot pluggable disks (SATA?)

[Jeremy Chadwick]

On Thu, Jul 26, 2007 at 06:53:37PM +0100, Thomas Hurst wrote:
> * Dag-Erling Sm?rgrav (des at des.no) wrote:
> 
> > It is a function of the disk controller and driver.  AFAIK, the ata
> > driver supports hot-plug as long as the SATA controller does.  I
> > believe most Promise and Intel ICH SATA controllers do, not sure about
> > Sil or nVidia MCP.
> 
> Both SiI and nForce's should support hot-swap, though if ata(4) actually
> handles them correctly is another matter.
> 
> I have an nForce 4 Pro board (a Tyan K8WE, CK804 chipset) which in
> FreeBSD gracefully handles hot removal, but needs a reboot to correctly
> recognise hot-plugged devices.  I thought I'd filed a PR about it, but I
> can't seem to find it...
> 
> I also have an 8 port Marvell SATA controller (88SX6081) which needs an
> atacontrol reinit to pick up new devices, but otherwise seems to work
> fine.

I recently had the joy of needing to hot-swap a disk in one of our
running servers (Supermicro SuperServer 5015M-T, Intel ICH7 SATA
controller).  The server does have a hot-swap SATA backplane, and AHCI
is enabled on the box (and reported by FreeBSD too).

I moved all data off of the disk and umount'd it.  Once our datacenter
guy got there, I did an atacontrol detach ad6, which resulted in:

subdisk6: detached
ad6: detached

I had our DC guy pull the carriage/disk, put the replacement, and
re-insert it.  I then did an atacontrol attach ad6, which resulted in:

ad6: 238475MB <WDC WD2500KS-00MJB0 02.01C03> at ata3-master SATA300

I was immediately able to see the drive, began newfs, mount, moved data
back, and all was well.

I like telling this incredibly boring story because both our DC guy and
myself were amazed at the fact that it *actually worked*.  He and I both
deal with hardware and software "insanities" on a daily basis (at our
day jobs), and we were expecting either the kernel to panic or the drive
to simply not show up until a reinit or even a reboot.  It's really nice
when things *just work*, you know what I mean?

Comparatively, at my home I have an Asus A8N-E (nForce CK804 SATA
controller) with numerous disks in it.  One recently went bad, and for
"fun" I decided to go through the same procedure as above.

It went well until the physical portion -- I disconnected the SATA cable
without issue, but the instant I disconnected the SATA power cable, I
heard all of the drives on the bus reset.  I looked up at the console
and found the machine had panic'd.  I plugged in the new disk (SATA
cable first, power 2nd) and again heard all the drives on the bus reset.
After rebooting the box, the BIOS found the new disk as well as all the
old ones, and of course FreeBSD did as well.

I'm not sure why all the devices on the bus reset.  Was this due to
the lack of SATA hot-swap backplane?  I really have no idea what those
backplanes do/provide on purely an electronic (and electrical) level,
so maybe what happened was normal for the scenario.  If someone could
clue me in here, I'd really appreciate it.

I'm pretty sure FreeBSD panic'd because the existing (and mounted) disks
which got reset disappeared from the bus; there's another -stable thread
about what FreeBSD does when you pull hardware out from underneath it
and it tries to access a device with a mounted filesystem.  :-)

An interesting/educational experience nonetheless.

-- 
| Jeremy Chadwick                                    jdc at parodius.com |
| Parodius Networking                           http://www.parodius.com/ |
| UNIX Systems Administrator                      Mountain View, CA, USA |
| Making life hard for others since 1977.                  PGP: 4BD6C0CB |