Booting Supermicro 6015V-M3 on SAS drive with LSI 1068E

[T. Middleton]

I have this machine and I'm just trying to get it to boot from an SAS 
drive.... that's all. I'm not trying to use the LSI 1068E for RAID.

See here if you're curious about specs:
http://www.supermicro.com/products/system/1U/6015/SYS-6015V-M3.cfm

Booting from the FreeBSD 6.2 install CD is not a problem; and the install 
kernel is, interestingly, able to see the drives attached to the LSI 1068E 
controller and use them. 

The problem is that the machine's BIOS can not seem to see the drives plugged 
into the LSI 1068E (raid controller), and thus those drives can not be 
selected in the BIOS for booting. "operating system not found". 

It seems the only way to get drives bootable from the BIOS that are plugged 
into the LSI is to put them into a raid. Once a logical disk is configured in 
the RAID bios one can see in the system bios "PCI SCSI: Software Raid". 
However, even if i did want to use this RAID device, FreeBSD doesn't 
see/support it (or, it doesn't support FreeBSD).

As an experiment I configured the LSI raid to contain one drive (RAID0) 
(stripe being the size of the drive). I then installed FreeBSD on this drive,
and, wonderfully enough, it worked. The BIOS was set to use the "PCI SCSI: 
Software raid", and the MBR was picked up from the single drive, and FreeBSD 
took over from there. I don't know how safe this is, but it seemed to still 
work after multiple reboots, and installing a bunch of software. Problems 
arose however when i installed a gmirror label onto the drive. Suddenly there 
was no more logical drive in the RAID bios on my next reboot; so i was back 
to "operating system not found": so it seems the LSI RAID also uses the same 
area as gmirror to store metadata. We want the gmirror on the SAS drives but 
not the LSI RAID... 

Other interesting ideas I had included trying to create a boot CD-ROM that 
would instruct the boot loader to do whatever it needs to do to load gmirror 
and then continue booting (using vfs.root.mountfrom?). Another idea was using 
a USB flash key for this. (Not really sure how to create a bootable CDRom to 
do this, but i'm sure it's possible. Any pointers here might be helpful.)

Another idea is it seems we could possibly switch a couple of the cables 
currently plugged into the LSI ports to the ESB2 SATA ports (this board has 
both the LSI 1068E controller supporting SAS/SATA, and an ESB2 controler 
supporting SATA) and use gmirrored SATA drives for booting. This is just a 
bit of a  pain because the existing cables probably aren't long enough... and 
everything is very tightly packed. Also it means we have to use a couple of 
SATA drives. And it makes my boss nervous to start moving cables... (though 
perhaps not as nervous as using a USB flash to boot).

Anyhow, anyone played with this hardware? Any recommendations on how to handle 
this seemingly weird situation?