The bloody RAID/JBOD virus

[Borja Marcos]

Hello,

We are again evaluating hardware to use FreeBSD with ZFS as a storage server. And, of course, we are again banging our heads
against the bloody "intelligent" controllers.

This machine has two controllers. The first one is recognized by the mps driver,
mps0: <LSI SAS2004> port 0x3f00-0x3fff mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on pci17
mps0: Firmware: 15.00.00.00, Driver: 16.00.00.00-fbsd
mps0: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>


and the second one (this is what I don't like at all) by the mfi driver.
mfi0: <Invader> port 0x4f00-0x4fff mem 0x913f0000-0x913fffff,0x91400000-0x914fffff irq 34 at device 0.0 on pci22
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23 
mfi0: FW MaxCmds = 240, limiting to 128
mfi0: MaxCmd = 240, Drv MaxCmd = 128, MaxSgl = 70, state = 0xb73c00f0


So, again, we have the typical scenario of the RAID card between ZFS and the disks.

# camcontrol devlist
<LSI Logical Volume 3000>          at scbus0 target 1 lun 0 (pass0,da0)


The machine has actually 24 disks. 

# zpool status
  pool: clientes
 state: ONLINE
  scan: none requested
config:

	NAME            STATE     READ WRITE CKSUM
	clientes        ONLINE       0     0     0
	  raidz2-0      ONLINE       0     0     0
	    mfisyspd0   ONLINE       0     0     0
	    mfisyspd1   ONLINE       0     0     0
	    mfisyspd2   ONLINE       0     0     0
	    mfisyspd3   ONLINE       0     0     0
	    mfisyspd4   ONLINE       0     0     0
	    mfisyspd5   ONLINE       0     0     0
	    mfisyspd6   ONLINE       0     0     0
	    mfisyspd7   ONLINE       0     0     0
	    mfisyspd8   ONLINE       0     0     0
	    mfisyspd9   ONLINE       0     0     0
	    mfisyspd10  ONLINE       0     0     0
	  raidz2-1      ONLINE       0     0     0
	    mfisyspd11  ONLINE       0     0     0
	    mfisyspd12  ONLINE       0     0     0
	    mfisyspd13  ONLINE       0     0     0
	    mfisyspd14  ONLINE       0     0     0
	    mfisyspd15  ONLINE       0     0     0
	    mfisyspd16  ONLINE       0     0     0
	    mfisyspd17  ONLINE       0     0     0
	    mfisyspd18  ONLINE       0     0     0
	    mfisyspd19  ONLINE       0     0     0
	    mfisyspd20  ONLINE       0     0     0
	    mfisyspd21  ONLINE       0     0     0
	spares
	  mfisyspd22    AVAIL   

errors: No known data errors



But, again, defined as "jbod", which I don't like at all. At least in the past this formula has been a proven disaster. In the past (similar situations) I've been unable to hot swap disks without voodoo macumba procedures (I consider needing a "mfiutil" voodoo macumba) and, of course, the real CAM subsystem has no visibility at all.

# mfiutil show adapter
mfi0 Adapter:
    Product Name: ServeRAID M5210e
   Serial Number: 3CJ0SG          
        Firmware: 24.0.2-0013
     RAID Levels: JBOD, RAID0, RAID1, RAID10
  Battery Backup: not present
           NVRAM: 32K
  Onboard Memory: 0M
  Minimum Stripe: 64K
  Maximum Stripe: 64K


Actually, in several machines I use a patched driver that ignores all the raid crap and presents the disks as real SAS disks. But I am using it on machines in which a failure is not a tragedy. Been working for years without incidents, but OS updates are always risky.

So, I would like to know:

1) Is this "mfisyspd" REALLY a disk? Won't I notice any differences? So far, I've attached SSD disks and ZFS has created a pool with 512 byte blocks. Note the difference between a "more or less like a disk JBOD" (which I definitely do not want) and a real disk.

2) Is there a way to bypass all that or should I look for a replacement HBA instead? Seems it's impossible to get manufacturers to ship simple HBAs without that "intelligent" RAID thing.

Of course I may be wrong, and this card might be what I really want, with no interference from the RAID functionality. But, so far, every time I've seen so-called JBODs defined with RAID cards, they were actually 1 disk RAID0 logical volumes, which I don't want.

At least loading the mfip driver gives me access to the pass devices, which is some progress.  But I'm still not sure.


Sorry for the blunt message, but wherever I look I see all these cards we should not use with ZFS.



Thanks!







Borja.