Dual AMD64, APCI woes, 5.3 RC2, Help!

[John Baldwin]

On Sunday 07 November 2004 08:33 pm, Tom Krouper wrote:
> This e-mail may seem very familiar. It was posted back in September by
> a co-worker, but I don't think he ever responded to the responses he
> got. Here is the entire e-mail that he has sent. The only difference
> from the original is that we tried installing 5.3 RC2 instead of 5.2.1
> Release. I've also added the version of BIOS as requested by someone
> on the list. In addition, I have not been able to get the ACPI
> disabled install to work. (Though I'll admit I didn't take much time
> trying to get it to work.)
>
> \/ Original message. \/
>
> I'm trying to get another FreeBSD system up at our office for a
> mission critical app. Unfortunately if I can't figure this problem
> out, FreeBSD will be tossed and Linux used instead and I really don't
> want that. We've always been a FreeBSD/i386 house, this is our first
> foray into Opteron. Here's the particulars....
>
> Rhapsody HDAMA-F (Version 1.89) mainboard
> Dual bge 10/100/1000 nics
> 8gb ram
> dual Opteron 2.0ghz cpus
> LSI megaraid amr 320-2X raid controller w/128mb cache
> Four ultra-320 15Krpm scsi drives, 35gb each
>
> The drives are set up raid 10. A striped set of 70gb, then a 70gb mirror.
>
> I'm using the amd64 ISO for 5.3 RC2
>
> My problem seems to center around APCI. If I disable APCI in the bios
> and boot off the install CD and select the boot without APCI, the
> system comes up and installs just fine. However, it seems to only
> recognize one cpu. At least, the sysctl variables I looked at appeared
> to indicate just one cpu and the top display didn't have a cpu column.
>
> If I leave APCI enabled, the system boots off the cd fine but when the
> installed starts to probe the devices, it gets stuck just after the
> line for the CD drive (acd0). I am pretty certain that that is exactly
> the point where it's probing the scsi drives, because if APCI is
> turned off - right after the acd0 probe is where geom starts and the
> disk is recognized. The lsi controller is definitely recognized ok
> further up in the probe display.
>
> I'd just disable APCI to get around this, but if FreeBSD will only use
> one cpu.... management will switch to Suse (which purportedly runs
> this configuration perfectly) and I'd really like to stay FreeBSD.
> Normally I'd just love to dive into the new and unknown (to me)
> opteron/5.X/APCI stuff. But I'm completely out of time on this project
> to get the machine up. Could anyone possibly offer me some quick
> pointers or help on this? I've googled lots and so far not come up
> with anything concrete, other than "disable APCI" which seems to leave
> me with just one cpu.
>
> /\ End of original message /\

You'll need to install the box with ACPI disabled for now, then build a test 
kernel with DDB, KDB, INVARIANTS, etc. in it.  Boot the test kernel, when it 
hangs, use Ctrl-Alt-Esc to break into the debugger and 'show intrcnt' to dump 
the interrupt counts.  The problem is likely some sort of interrupt storm due 
to PCI interrupt routing being busted or some such.  Also, I would look for a 
BIOS upgrade just in case.

-- 
John Baldwin <jhb at FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve"  =  http://www.FreeBSD.org