data corruption with ahc driver and 4GB of memory using a FBSD-8 64-bit installation?

[Scott Long]

Gary Jennejohn wrote:
> On Thu, 31 Jan 2008 10:51:34 -0700
> Scott Long <scottl at samsco.org> wrote:
> 
>> Gary Jennejohn wrote:
>>> I think $(subject) says it all.
>>>
>>> I have a Gigabyte AM2 mobo with an AMD64 X2 CPU installed.
>>>
>>> The SCIS set up looks lie this:
>>>
>>> 29160N <--> da0 <--> da1
>>>
>>> da0 has a 32-bit installation of FBSD-8
>>> da1 has a 64-bit installtion of FBSD-8
>>>
>>> If I install 4GB of memory (4 DIMMs) in the system I see the following
>>> behavior:
>>>
>>> a) booting from da0 works just fine and I can access both disks without
>>> any problem.
>>>
>>> b) booting from da1 results in (apparent) data errors such that /bin/sh
>>> dies with SIGILL and /rescue/sh dies with SIGSEGV.
>>>
>>> c) trying to do an installation of a snapshot of FBSD-8 to a SCSI disk
>>> results in various problems, among others a kernel panic in ffs_balloc
>>> during newfs.
>>>
>>> d) an installation to a SATA disk succeeds and the system runs just
>>> fine.
>>>
>>> With only 3GB of memory everything works. Of course, I'd really like to
>>> be able to use the entire 4GB. And I want to keep my SCSI disks.
>>>
>>> I suspect that ahc has some sort of problem in 64-bit mode. However,
>>> I'm not certain whether the 32-bit installation works with 4GB simply
>>> because the ahc driver uses bounce buffers.
>>>
>>> Has anybody else seen this? Can anyone confirm that ahc does indeed
>>> have an error with 4GB and a 64-bit installation?
>>>
>>> BTW please put me in the Cc because I'm not subscribed to this ML.
>>>
>> The ahc driver should work with >4GB, and in fact that was a standard
>> part of the test suite back when it was still in active development.  If
>> it doesn't work now then it's due to some sort of bitrot.  I can try to
>> test and debug it myself, but I'm pretty overcommitted so I can't
>> guarantee that I'll get to it.  If others would like to help test and
>> debug, I'm happy to answer questions and provide assistance.
>>
> 
> Well, the thing which I find confusing is that the 32-bit installation
> works without any problems.  I would expect the 64-bit installation to
> work with 4GB, if anything does.
> 
> Does the 32-bit version use bounce buffers?
> 
> I'll try poking around some, but the 64-bit install is my (money earning)
> work station and I can't afford to trash it, so I'm sort of unwilling to
> risk it.  And I don't have any spare SCSI disks to use for testing.  Well,
> I suppose I could sacrifice the 32-bit disk.  It's not so important.
> 
> Any hints on where to start looking?
> 

Even with only 4GB of physical RAM, the motherboard will remap part of
that RAM to be above the 4GB barrier.  So running in 64-bit mode with
only 4GB of RAM will still exercise the >4GB bugs.  Let me think on
where to point you for debugging...  The ahc cards don't normally use
bounce buffers; they are designed to handle up to 2^39 bytes of RAM
directly.

Scott