Dell branded storage controller for ZFS file-server | Advice requested

[Ahmed Kamal]

comments down below

On Fri, Jan 23, 2015 at 1:57 PM, David Gwynne <david at gwynne.id.au> wrote:

>
> > On 23 Jan 2015, at 9:41 pm, Ahmed Kamal <email.ahmedkamal at googlemail.com>
> wrote:
> >
> > Thanks a lot for your reply .. I've got a couple of questions inline
> >
> > On Fri, Jan 23, 2015 at 12:21 PM, David Gwynne <david at gwynne.id.au>
> wrote:
> >
> > > On 23 Jan 2015, at 7:16 pm, Ahmed Kamal <
> email.ahmedkamal at googlemail.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > I'm looking to build a ZFS file-server, unfortunately I'm locked into
> Dell
> > > branded hardware only. I know Dell sometimes ships LSI branded cards,
> but
> > > that too probably won't work for me, as I live outside the US and the
> local
> > > Dell rep is clueless (so whatever I want to buy, has to be on Dell's
> > > website). I am considering Dell R720xd or R730xd. I will use Intel i350
> > > NICs, and ECC RAM, but the biggest questions are around the SAS HBA
> > >
> > > Per my research, I have the following notes and questions. Appreciating
> > > your comments on them
> > >
> > > * H310 Card supports syspd (jbod) mode per
> > > http://svnweb.freebsd.org/base?view=revision&revision=254906 so it is
> > > probably the best card for what I want to do, right?
> >
> > no.
> >
> > > * H310 is a low end card, it has very limited queue depth (25 I
> believe) ..
> > > Is this actually a problem when using the disks (20 x SAS 10k disks) in
> > > syspd/jbod mode? what about if I attach a couple of SSD disks?
> >
> > there are two queue limits here, one for the controller and one for each
> disk attached to it.
> >
> > h310s have a controller queue depth of 31, ie, they can only support 31
> commands at a time. if you have 24 disks and distribute the controllers
> command slots between them, thats about 1 command per disk.
> >
> > My initial line of thought was that in pass-through (jbod) mode, the HBA
> does *not* buffer commands and proxy disk responses internally. Which meant
> that in jbod mode, the HBA basically becomes a communications channel
> connecting the OS to disks, making its internal buffering limitations
> not-applicable. Is this not correct ?
>
> the h310 physically doesnt have cache, so it cant buffer your I/O even if
> it wanted to.
>
> so yes, in your use case it would just be a hba, but a very constrained
> one. even most sata controllers (eg, ahci or the silicon image ones) can
> keep more commands up in the air since they can have up 32 command slots
> per port instead of 31 for the whole controller.
>
> im also not sure if dell will let you order an r720xd with a h310. i dont
> think they consider that a supported config.
>
> >
> >
> > > * H310 can be flashed to IT mode (LSI firmware), and is supported by
> > > Illumos kernel (if I'll ever need that) .. so overall it looks like a
> good
> > > option. Any "watch-outs" I should be aware of ?
> >
> > yes. if you flash the h310 to become an sas hba, the pci product/vendor
> and subproduct/subvendor ids on the controller change. the dell bios will
> refuse to boot if it detects an unsupported controller in its storage slot.
> you'll brick the machine until you remove the h310. because it is on a non
> standard mezzanine connector you will not be able to flash it back to a
> h310, and you'll have a useless chunk of silicon to put on your desk as a
> monument to how annoying vendors are.
> >
> > pff nightmare scenario .. thanks for pointing this out. I stumbled
> across web posts where people successfully flashed h310, but meh, now I'm
> scared to buy that configuration
>
> there are h310s that are normal pci-e cards that go into dells lower end
> servers. the ones in the r720s sit on the custom mezzanine connector.
>
> > > * H710/p have no way of exposing JBOD/syspd in any way .. If using
> those
> > > cards, the best I can do, is to create a raid-0 per disk, and put zfs
> on
> > > top of that. How bad is this? Is the only problem that I'm locked into
> H710
> > > firmware? Would the nvram and bbu on H710 improve performance
> significantly
> > > vs H310?
> >
> > i wouldnt run zfs on that.
> >
> > > * H330 and H730, seem supported by mrsas driver. Is this driver able to
> > > expose jbod/syspd mode? If yes, what's the overall advise, should I go
> with
> > > one of those, or the "trusty" H310 ?
> >
> > thats a workable option. h330 should be fine if all you want to do is
> jbod.
> >
> > Now that you're pointing me to H330 being my best option given the
> situation, got some more questions:
> > * "if all you want is jbod" .. with zfs, that's all we need right ? just
> wondering if I'm missing something important
>
> nope.
>
> > * Any idea if jbod configuration on h330 is done on bios level, or using
> some freebsd tool (how?)
>
> you can do it in the bios.
>
> > * mrsas man page, mentions the driver appeared in 10.1 .. Does that mean
> if I decide to use FreeNAS (9.x based) it won't be there ?
>
> i have no idea :)
>
> > * If I go the h330 path, it'll be a r730xd server. Worst case if the
> h330 gives me trouble, can I throw it out and plug a LSI 9207-8i in its
> place (same pci slot, cables ..etc) ?
>
> the h330 sits on a mezzanine connector. you probably can get that card and
> drive the disks with it, but i havent done it with an r730 or 730xd so i
> cant say for sure. i have done that with an r720 though, and it mostly
> works.
>
> we had to order those systems with a h310 and add the lsi card ourselves,
> and then we discovered that the disks they shipped in the system do not
> spin up when theyre powered on. the raid controller (h310) explicitly spins
> them up before issuing io to them.
>
> if you attach those disks to a straight sas hba or use them via syspd on
> the h310, you cant boot off them because the machines bios (not the raid
> controller) blindly issues io to them without spinning them up first.
>

scary stuff .. so just to be clear:

- LSI SAS hba => disks DONT spin up
- h310 in jbod mode => disks DONT spin up
- h330 in jbod mode => disks work fine ?

Note that currently dell's web configurator, only sells h330 with r730(xd)
not r720(xd) .. The r720 comes with h310 or h710 afaict

Note 2, Illumos people are able to pass a driver parameter to spin up
crappy dell disks .. check
http://www.listbox.com/member/archive/182179/2013/06/sort/from/page/4/entry/15:430/20130617155143:52F0F7EA-D787-11E2-84A9-87C6F7D228D8/


>
> fun times.
>
> dlg
>
>
>
> >
> >
> > Thanks!
> >
> >
> > >
> > > Your help and advice is most appreciated
> > > _______________________________________________
> > > freebsd-scsi at freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org
> "
> >
> >
>
>