mfi - setting up disks

Thu Sep 23 12:55:08 UTC 2010

Hi guys,
In the SuperMicro system where I had problems with the mpt controller, I switched it for a mfi-based controller. I had it set up with 36x RAID0 volumes with each their own disk (no way to access the disk otherwise I found), and added them to a ZFS system. The numbering became a bit weird, so I pulled the disks out one by one and put them back to figure out and note down what disk number was in what slot. Only test data on my ZFS volume, so I didn't mind that crashing.

Now that all disks have been taken out and put back in one by one, I do:

# mfiutil show volumes
mfi0 Volumes:
  Id     Size    Level   Stripe State  Cache   Name

Whoops, no volumes? That can't be good. I check up on the disks, they're all there:

 mfiutil show drives
mfi0 Physical Drives:
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721953> SATA enclosure 1, slot 2
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722143> SATA enclosure 1, slot 3
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722152> SATA enclosure 1, slot 4
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722133> SATA enclosure 1, slot 14
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722139> SATA enclosure 1, slot 15
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721963> SATA enclosure 1, slot 17
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722160> SATA enclosure 1, slot 23
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722127> SATA enclosure 2, slot 1
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722155> SATA enclosure 2, slot 4
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721968> SATA enclosure 2, slot 5
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721956> SATA enclosure 2, slot 6
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722137> SATA enclosure 2, slot 7
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721969> SATA enclosure 2, slot 8
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722122> SATA enclosure 2, slot 10
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722149> SATA enclosure 2, slot 11
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721951> SATA enclosure 1, slot 0
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722128> SATA enclosure 1, slot 5
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721966> SATA enclosure 1, slot 6
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721952> SATA enclosure 1, slot 7
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722145> SATA enclosure 1, slot 8
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722131> SATA enclosure 1, slot 9
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721961> SATA enclosure 1, slot 10
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721954> SATA enclosure 1, slot 11
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722158> SATA enclosure 1, slot 12
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721967> SATA enclosure 1, slot 13
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721971> SATA enclosure 1, slot 16
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722157> SATA enclosure 1, slot 19
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721970> SATA enclosure 1, slot 21
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722140> SATA enclosure 1, slot 22
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722125> SATA enclosure 2, slot 2
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722146> SATA enclosure 2, slot 3
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722134> SATA enclosure 2, slot 9
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721965> SATA enclosure 1, slot 1
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ722151> SATA enclosure 1, slot 20
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721964> SATA enclosure 2, slot 0
( 1863G) UNCONFIGURED GOOD <SAMSUNG HD203WI 0003 serial=S1UYJDWZ721955> SATA enclosure 1, slot 18

Well, that's an excelent way of adding the volumes in the sequence they appear physically, so I start with the first one from top left:
# mfiutil create raid0 -v E01:S05
Adding drive 26 to array 0
Adding array 0 to volume 0
mfiutil: Command failed: Wrong firmware or drive state
mfiutil: Failed to add volume: Input/output error

Firmware error? Invalid drive state?

# mfiutil good E01:S05
mfiutil: Drive 26 is already in the desired state

Seems to be good... do I have a firmware issue? Check dmesg:

mfi0: <LSI MegaSAS Gen2> port 0xc000-0xc0ff mem 0xfad7c000-0xfad7ffff,0xfadc0000-0xfadfffff irq 16 at device 0.0 on pci5
mfi0: Megaraid SAS driver Ver 3.00 
mfi0: 1966 (338565595s/0x0020/info) - Shutdown command received from host
mfi0: 1967 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9261/1000)
mfi0: 1968 (boot + 3s/0x0020/info) - Firmware version 2.0.03-0673
mfi0: 1969 (boot + 4s/0x0020/info) - Board Revision 
mfi0: 1970 (boot + 24s/0x0004/info) - Enclosure (SES) discovered on PD 08(c Port 0 - 3/p1)
mfi0: 1971 (boot + 24s/0x0004/info) - Enclosure (SES) discovered on PD 09(c Port 0 - 3/p2)
mfi0: 1972 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) communication restored
mfi0: 1973 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 1 speed changed
mfi0: 1974 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 2 speed changed
mfi0: 1975 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 3 speed changed
mfi0: 1976 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) communication restored
mfi0: 1977 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 1 speed changed
mfi0: 1978 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 2 speed changed
mfi0: 1979 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 3 speed changed
mfi0: 1980 (boot + 24s/0x0002/info) - Inserted: Encl PD 08
mfi0: 1981 (boot + 24s/0x0002/info) - Inserted: PD 08(c Port 0 - 3/p1) Info: enclPd=08, scsiType=d, portMap=00, sasAddr=50030480008fb0fd,0000000000000000
mfi0: 1982 (boot + 24s/0x0002/info) - Inserted: Encl PD 09
mfi0: 1983 (boot + 24s/0x0002/info) - Inserted: PD 09(c Port 0 - 3/p2) Info: enclPd=09, scsiType=d, portMap=00, sasAddr=50030480008e7b7d,0000000000000000
mfi0: 1984 (boot + 24s/0x0002/info) - Inserted: PD 0a(e0x08/s2)

and then lots of disks.... looks fine, right?

What am I missing? I badly want mfid0-mfid35 back so that I can recreate my ZFS and get to work :-)

On a side note, the ZFS I'll make is this, any comments on the configuration?

zpool create tank \
raidz2 /dev/mfid0 /dev/mfid1 /dev/mfid2 /dev/mfid3 /dev/mfid4 /dev/mfid5 /dev/mfid30 \
raidz2 /dev/mfid6 /dev/mfid7 /dev/mfid8 /dev/mfid9 /dev/mfid10 /dev/mfid11 /dev/mfid31 \
raidz2 /dev/mfid12 /dev/mfid13 /dev/mfid14 /dev/mfid15 /dev/mfid16 /dev/mfid17 /dev/mfid32 \
raidz2 /dev/mfid18 /dev/mfid19 /dev/mfid20 /dev/mfid21 /dev/mfid22 /dev/mfid23 /dev/mfid33 \
raidz2 /dev/mfid24 /dev/mfid25 /dev/mfid26 /dev/mfid27 /dev/mfid28 /dev/mfid29 /dev/mfid34 \
spare /dev/mfid35

Cheers

	Nik