Cannot replace broken hard drive with LSI HBA

[Karli Sjöberg]

Den 28 sep 2015 4:27 em skrev Rich <rincebrain at gmail.com>:
>
> Hi Karli,
> Which mps-supported HBA? Your firmware version indicates it's
> something in the 92xx family, but there's a number of variants on that
> flavor.

Several different models. We have 9200, 9201, 9211 and several SuperMicro AOC-USAS2-L8i (LSI2008)

>
> Have you played with any of the drive timeout settings in the HBA
> firmware/OS/drives themselves (the dark vendor-specific magic known
> variously as TLER, CCTL, ERC...)?

OS timeout is:
dev.mps.0.spinup_wait_time: 5

Nothing else. Defaults should be there for a reason, at least until I know of a better one:)

>
> What models are the servers?

Several different models/makers. There are a couple of different SuperMicro motherboards, X9SRH-F, X8SIL-F, X9DBU-F, an HP DL180 G6, and a Sun Fire X4140. Most have one or several SuperMicro SC837E26-RJBOD1 but the HP just have the internal bays and the X4140 has two standard Sun JBOD's, can't remember their names right now. So very varying in nature, except for the HBA's, that are all of the same maker.

>
> There are a number of possible complicating factors here - whether the
> drives are SAS or SATA (and any "quirks" of the drives), whether the
> backplanes are passive or have SAS expanders, what version of SAS/SATA
> these backplanes are capable of handling, any firmware strangeness on
> the passive or otherwise backplane...

SATA drives only, different models and makers. All of them have different expanders, different versions. The HP and Sun should have SAS1, while the SM JBOD's should be SAS2

>
> How does the machine misbehave once you re-insert the drive?

The expected result of inserting a new drive would be to have it show up in the OS.

>
> Does the machine misbehave if you keep the drive removed?

Nothing in particular, except for _not_ being able to resilver the zpool:)

One interesting thing to mention is that I've tried inserting a SAS drive in that bay and it successfully showed up. But any other SATA drive in the same bay is "blocked" until after a reboot.

>
> One final quirk I'll mention is that a number of SAS expander
> backplanes I've encountered sometimes will not notice a drive is
> physically pulled until a new drive is inserted, and sometimes the
> best way to convince it to see a drive after pulling one that was
> misbehaving is:
> - seat a "new" (not otherwise in the machine) drive
> - unseat said drive after a few seconds
> - seat whatever drive you intended to seat in the first place, be it
> "new" or the original drive

The drives have been newly ordered as replacements and so have never previously been in the machine.

/K

>
> Good luck,
>
> - Rich
>
> On Mon, Sep 28, 2015 at 9:36 AM, Karli Sjöberg <karli.sjoberg at slu.se> wrote:
> > Hey all!
> >
> > I´m just giving a shout out here to see if anyone else have had similar
> > experiences working with LSI/Avago HBA's in FreeBSD.
> >
> > For some time now, about a year or so, we´ve had several times were hard
> > drives have dropped out, you pull it out, pop a new back in, but it
> > never shows up in the OS. When inserted, nothing prints in the logs, and
> > physically, it just blinks for a half a second, then nothing. The entire
> > server then needs to be rebooted to get the drive back.
> >
> > As for the hardware, we have several SuperMicro servers, an HP, and an
> > old SUN server that all have this problem. It´s happened with both old
> > and new drives from different manufacturers and sizes. The only thing in
> > common has been the LSI/Avago HBA.
> >
> > The software is FreeBSD-10.1-STABLE as per this[*] bug, very close to
> > 10.2-RELEASE, mps driver version 20 and the firmware has been flashed to
> > 19. Also tried firmware version 20 but ZFS went nuts, displaying
> > checksum errors on just about every disk in the pool.
> >
> > I´ts gotten to the point I´m fed up and have to ask if someone else
> > could think of a fix, since neither software nor firmware upgrade seems
> > to make a difference. Or to suggest another HBA instead?
> >
> > Thanks in advance!
> >
> > /K
> >
> > [*]: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191348
> >
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"