[Fwd: Re: Large ZFS arrays?]

[Rich]

On Sun, Jun 15, 2014 at 12:00 PM, Kevin Day <toasty at dragondata.com> wrote:
>
> On Jun 15, 2014, at 10:43 AM, Dennis Glatting <dg at pki2.com> wrote:
>>
>> Total. I am looking at three pieces in total:
>>
>> * Two 1PT storage "blocks" providing load sharing and
>>  mirroring for failover.
>>
>> * One 5PB storage block for on-line archives (3-5 years).
>>
>> The 1PB nodes will divided into something that makes sense, such as
>> multiple SuperMicro 847 chassis with 3TB disks providing some number of
>> volumes. Division is a function of application, such as a 100TB RAIDz2
>> volumes for bulk storage whereas smaller 8TB volumes for active data,
>> such as iSCSI, databases, and home directories.
>>
>> Thanks.
>

> 2) You definitely want a JBOD controller instead of a smart RAID controller. The LSI 9207 works pretty well, but when you exceed 192 drives it complains on boot up of running out of heap space and makes you press a key to continue, which then works fine. There is a very recently released firmware update for the card that seems to fix this, but we haven't completed testing yet. You'll also want to increase hw.mps.max_chains. The driver warns you when you need to, but you need to reboot to change this, and you're probably only going to discover this under heavy load.

You should be able to disable the boot ROM on the HBAs to avoid this
problem? I had this happen too in various configs, but only when the
controllers were configured to add the devices attached as boot
options...

- Rich