MegaRAID lockups under 5.5 PRELEASE

[Tony Byrne]

Folks,

We have a pair of servers, each with an of Intel SCRU42X branded
MegaRAID controller installed. The cards both have battery backed
cache. The servers form a 2-node Slony cluster for PostgreSQL, with
each node having a single RAID5 array consisting of 3 live disks with
a hot standby disk.

About a week ago we upgraded FreeBSD on both boxes from a point on
RELENG_5 (1 July 2005) to 5.5-PRERELEASE and now one of the boxes has
started behaving badly.

In the space of two days we've had about half a dozen occurrences of
random processes blocking and rendering the machine unusable.  'top'
shows the stricken processes in state 'ffsfsn' and they cannot be killed.

The affected machine is a Slony subscriber and so isn't directly used
by customers, but is still an important component in our system.
PostgreSQL seems to be most likely to block, but sshd has also blocked
preventing logins.

The interesting thing is that the main box, which has an almost
identical configuration and a greater work load, has remained
unaffected, so far. The only difference between the two machine
configurations is that the misbehaving machine only has 128Mb of
on-board battery backed cache, while the main machine has 256Mb. The
firmware on both machines is Intel's version 413Y.

This is not the first time that we've had problems like this with
FreeBSD and this model of controller.  For background see:

http://docs.freebsd.org/cgi/getmsg.cgi?fetch=172108+0+archive/2004/freebsd-stable/20041231.freebsd-stable

http://docs.freebsd.org/cgi/getmsg.cgi?fetch=126427+0+archive/2004/freebsd-stable/20041121.freebsd-stable

The current problem is very reminiscent of the latter issue, which had
been causing headaches for us over a year ago, but which disappeared
after some driver improvements by Scott Long (many thanks Scott :-).

I see from a diff of the driver code from RELENG_5 on 1 July 2005 and
the latest RELENG_5 version that some further changes have been made
to the driver. I would have been moderately happy if I could have
pinned the reemergence of the problem on these changes, because then I
would have had a specific cause. However, as a test, I reverted to the
*previous* kernel from 1 July 2005 and the box blocked in sshd within
a few hours, preventing login.

At this stage I'm looking at upgrading the firmware of the RAID card
to the latest and greatest and if that doesn't resolve it, I plan to
make a jump to FreeBSD 6, which appears to have substantial changes to
the amr driver and which might solve the problem.

Before I leap though, I'd be interested in hearing if anyone is
familiar with the behavior that I've described and can point me in the
right direction. Alternatively, if someone can say "hey we use FreeBSD
6.x with the SCRU42X and it works great" then I'd be obliged.

Our woes may well be caused be some faulty hardware, especially since
the other box has remained stable after the same upgrade, but I remain
suspicious because the the problem arose so soon after an upgrade of
FreeBSD and after many months of uptime.

Many thanks,

Regards,

Tony.

-- 
Tony Byrne