mfi - setting up disks

Thu Sep 23 13:56:04 UTC 2010

Hello,

I already tinkered with that kind of controller too (it seems we faced 
the same problems in the same order too...). For this kind of purposes, 
you really want a controller that does real JBOD and tells you about 
what is going on with the disk. :/

I bumped in exactly the same problem as you, meaning that removing the 
drive will destroy the RAID-0 volume, and frankly, even if the tools 
were working for recreating the volumes, having to do that to see the 
disk again is a "dirty hack"(tm) that is bound to blow up in your face 
at some point, or to be really hard to maintain for any upcoming upgrade.

Sorry for not providing any additional help with your problem. :/

On 09/23/2010 09:55 PM, Niklas Saers wrote:
> Hi guys,
> In the SuperMicro system where I had problems with the mpt controller, I switched it for a mfi-based controller. I had it set up with 36x RAID0 volumes with each their own disk (no way to access the disk otherwise I found), and added them to a ZFS system. The numbering became a bit weird, so I pulled the disks out one by one and put them back to figure out and note down what disk number was in what slot. Only test data on my ZFS volume, so I didn't mind that crashing.
>
> Now that all disks have been taken out and put back in one by one, I do:
>
> # mfiutil show volumes
> mfi0 Volumes:
>    Id     Size    Level   Stripe State  Cache   Name
>
> Whoops, no volumes? That can't be good. I check up on the disks, they're all there:
>
>   mfiutil show drives
> mfi0 Physical Drives:
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721953>  SATA enclosure 1, slot 2
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722143>  SATA enclosure 1, slot 3
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722152>  SATA enclosure 1, slot 4
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722133>  SATA enclosure 1, slot 14
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722139>  SATA enclosure 1, slot 15
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721963>  SATA enclosure 1, slot 17
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722160>  SATA enclosure 1, slot 23
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722127>  SATA enclosure 2, slot 1
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722155>  SATA enclosure 2, slot 4
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721968>  SATA enclosure 2, slot 5
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721956>  SATA enclosure 2, slot 6
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722137>  SATA enclosure 2, slot 7
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721969>  SATA enclosure 2, slot 8
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722122>  SATA enclosure 2, slot 10
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722149>  SATA enclosure 2, slot 11
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721951>  SATA enclosure 1, slot 0
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722128>  SATA enclosure 1, slot 5
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721966>  SATA enclosure 1, slot 6
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721952>  SATA enclosure 1, slot 7
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722145>  SATA enclosure 1, slot 8
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722131>  SATA enclosure 1, slot 9
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721961>  SATA enclosure 1, slot 10
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721954>  SATA enclosure 1, slot 11
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722158>  SATA enclosure 1, slot 12
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721967>  SATA enclosure 1, slot 13
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721971>  SATA enclosure 1, slot 16
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722157>  SATA enclosure 1, slot 19
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721970>  SATA enclosure 1, slot 21
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722140>  SATA enclosure 1, slot 22
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722125>  SATA enclosure 2, slot 2
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722146>  SATA enclosure 2, slot 3
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722134>  SATA enclosure 2, slot 9
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721965>  SATA enclosure 1, slot 1
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ722151>  SATA enclosure 1, slot 20
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721964>  SATA enclosure 2, slot 0
> ( 1863G) UNCONFIGURED GOOD<SAMSUNG HD203WI 0003 serial=S1UYJDWZ721955>  SATA enclosure 1, slot 18
>
> Well, that's an excelent way of adding the volumes in the sequence they appear physically, so I start with the first one from top left:
> # mfiutil create raid0 -v E01:S05
> Adding drive 26 to array 0
> Adding array 0 to volume 0
> mfiutil: Command failed: Wrong firmware or drive state
> mfiutil: Failed to add volume: Input/output error
>
> Firmware error? Invalid drive state?
>
> # mfiutil good E01:S05
> mfiutil: Drive 26 is already in the desired state
>
> Seems to be good... do I have a firmware issue? Check dmesg:
>
> mfi0:<LSI MegaSAS Gen2>  port 0xc000-0xc0ff mem 0xfad7c000-0xfad7ffff,0xfadc0000-0xfadfffff irq 16 at device 0.0 on pci5
> mfi0: Megaraid SAS driver Ver 3.00
> mfi0: 1966 (338565595s/0x0020/info) - Shutdown command received from host
> mfi0: 1967 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9261/1000)
> mfi0: 1968 (boot + 3s/0x0020/info) - Firmware version 2.0.03-0673
> mfi0: 1969 (boot + 4s/0x0020/info) - Board Revision
> mfi0: 1970 (boot + 24s/0x0004/info) - Enclosure (SES) discovered on PD 08(c Port 0 - 3/p1)
> mfi0: 1971 (boot + 24s/0x0004/info) - Enclosure (SES) discovered on PD 09(c Port 0 - 3/p2)
> mfi0: 1972 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) communication restored
> mfi0: 1973 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 1 speed changed
> mfi0: 1974 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 2 speed changed
> mfi0: 1975 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 3 speed changed
> mfi0: 1976 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) communication restored
> mfi0: 1977 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 1 speed changed
> mfi0: 1978 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 2 speed changed
> mfi0: 1979 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 3 speed changed
> mfi0: 1980 (boot + 24s/0x0002/info) - Inserted: Encl PD 08
> mfi0: 1981 (boot + 24s/0x0002/info) - Inserted: PD 08(c Port 0 - 3/p1) Info: enclPd=08, scsiType=d, portMap=00, sasAddr=50030480008fb0fd,0000000000000000
> mfi0: 1982 (boot + 24s/0x0002/info) - Inserted: Encl PD 09
> mfi0: 1983 (boot + 24s/0x0002/info) - Inserted: PD 09(c Port 0 - 3/p2) Info: enclPd=09, scsiType=d, portMap=00, sasAddr=50030480008e7b7d,0000000000000000
> mfi0: 1984 (boot + 24s/0x0002/info) - Inserted: PD 0a(e0x08/s2)
>
> and then lots of disks.... looks fine, right?
>
> What am I missing? I badly want mfid0-mfid35 back so that I can recreate my ZFS and get to work :-)
>
> On a side note, the ZFS I'll make is this, any comments on the configuration?
>
> zpool create tank \
> raidz2 /dev/mfid0 /dev/mfid1 /dev/mfid2 /dev/mfid3 /dev/mfid4 /dev/mfid5 /dev/mfid30 \
> raidz2 /dev/mfid6 /dev/mfid7 /dev/mfid8 /dev/mfid9 /dev/mfid10 /dev/mfid11 /dev/mfid31 \
> raidz2 /dev/mfid12 /dev/mfid13 /dev/mfid14 /dev/mfid15 /dev/mfid16 /dev/mfid17 /dev/mfid32 \
> raidz2 /dev/mfid18 /dev/mfid19 /dev/mfid20 /dev/mfid21 /dev/mfid22 /dev/mfid23 /dev/mfid33 \
> raidz2 /dev/mfid24 /dev/mfid25 /dev/mfid26 /dev/mfid27 /dev/mfid28 /dev/mfid29 /dev/mfid34 \
> spare /dev/mfid35
>
>
> Cheers
>
> 	Nik_______________________________________________
> freebsd-scsi at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"

-- 
Stephane LAPIE, EPITA SRS, Promo 2005
"Even when they have digital readouts, I can't understand them."
--MegaTokyo