ZFS hanging at kernel boot now, but didn't before... (Re: ZFS MFC heads up)

[Joe Karthauser]

Hmm, I've had a bit of a miserable afternoon trying to fight my RELENG_7 
server, which now doesn't boot. :(.

So, it's a ZRAID2 pool with a ufs/gmirror root partition split over 5 
disks (gmirror on 500Mb partition on each of five disks, and zraid2 over 
the rest of each drive).

What I did was to update the userland, and then reboot. I didn't upgrade 
the kernel (but I've subsequently done that and have the same problem).

What happens is that the kernel hangs booting just after displaying a 
LABEL message or ZFS pool/spool message. I _can_ get it to boot if I 
boot single user with acpi switched off. When I do that I can manually 
start zfs, and mount all the partitions. However, one of the disks is 
missing.... more on that next.

The machine is running a gigabyte motherboard (domestic gamer P35 board, 
similar to this 
http://www.gigabyte.com.tw/Products/Motherboard/Products_Overview.aspx?ProductID=2533, 
although it might be a DS4 variant).  I've got 5 of the 6 sata ports 
wired to a 5 unit SATA hot swap bay (5 drives vertially mounted into 3 
5-1/4" bays kind of thing).

Now, because of the gmirror I can boot the system on any disk, or 
combination of plugged in disks. I should be able to succeed with the
kernel probe up to the attempt to mount the root filesystem irrespective 
of any zfs pool, etc. And, indeed, this has been working fine for about 
two years.

But, now it hangs in the same place no matter what disk I boot on (I've 
tried every bay).

But, without ACPI enabled it does appear to boot ok... what's going on 
here? Is it possible that the machine has developed a hardware fault?

Ok, finally, if I boot with ACPI disabled then one of the disks is 
missing. If I unplug it I get a disconnect message from the ata device, 
and a reconnect and reinit attempt when I plug it back in, but no device 
appears on the bus. Usually I can do a 'atacontrol detach sata4; sleep 
1; atacontrol attach sata4' and the device reappears. This happens on 
the other buses, but not on the last one. It's not the disk, because if 
I swap it into another bay, it comes up and appears on the bus. On the 
other hand it doesn't appear to be that controller or slow in the drive 
bay because if I unplug all the over disks the system will boot that 
disk and get as far as the hang.... hmm.

Is this a consequence of disabling the ACPI?

Does anyone have a clue what might be going on?

Joe