Accessing SCSI-Devices >2TB

[Raphael H. Becker]

On Fri, Jun 10, 2005 at 09:07:18AM -0600, Kenneth D. Merry wrote:

> > and here the relevant diffs:
> > http://rabe.uugrn.org/temp/FreeBSD/bigraid/dmesg.knoppix_diff.txt
> 
> This is quite interesting:
[....]
> Linux notices that the device returned 0xffffffff as the capacity in
> response to a READ CAPACITY(10) command, so it tries a READ CAPACITY(16)
> command, which *fails*.
> 
> So even under Linux you aren't getting the full capacity of your device,
> you're only getting 2TB.

The support told me, SuSE Linux is known to work with >2TB in one device,
means they might have some patches to work around. I will try a SuSE
live system next days just to get sure it works. But the System won't
be SuSE in future.
 
> > Second I rebooted FreeBSD with CAMDEBUG in kernel and enabled it via
> > "camcontrol debug ..." and did a "camcontrol rescan 1" then:
> > http://rabe.uugrn.org/temp/FreeBSD/bigraid/freebsd54_camdebug.txt
> 
> camcontrol debug -I isn't quite what we need in this situation.  Instead,
> you should try 'camcontrol debug -c'.

# camcontrol debug -c 1:0
# camcontrol rescan 1
Re-scan of bus 1 was successful

in /var/log/messages:

kernel: (probe0:ahc1:0:0:0): TEST UNIT READY.  CDB: 0 0 0 0 0 0
kernel: (probe0:ahc1:0:0:0): INQUIRY. CDB: 12 0 0 0 24 0
kernel: (probe0:ahc1:0:0:0): INQUIRY. CDB: 12 0 0 0 fc 0
kernel: (probe0:ahc1:0:0:0): MODE SENSE(06).  CDB: 1a 0 a 0 14 0
kernel: (probe0:ahc1:0:0:0): INQUIRY. CDB: 12 1 80 0 ff 0
kernel: (probe0:ahc1:0:0:1): INQUIRY. CDB: 12 20 0 0 24 0
kernel: (probe0:ahc1:0:0:2): INQUIRY. CDB: 12 40 0 0 24 0
kernel: (probe0:ahc1:0:0:3): INQUIRY. CDB: 12 60 0 0 24 0
kernel: (probe0:ahc1:0:0:4): INQUIRY. CDB: 12 80 0 0 24 0
kernel: (probe0:ahc1:0:0:5): INQUIRY. CDB: 12 a0 0 0 24 0
kernel: (probe0:ahc1:0:0:6): INQUIRY. CDB: 12 c0 0 0 24 0
kernel: (probe0:ahc1:0:0:7): INQUIRY. CDB: 12 e0 0 0 24 0

Does not say anything to me.

 
> > Any idea, whats wrong with it?
> 
> >From what I can see, it's likely the device is misbehaving.  The fact that
> the 16 byte read capacity fails under Linux is telling.  If you've got a
> device that supports a LUN size greater than 2TB, it must support the 16
> byte read capacity and read/write commands.

So you would say this is a misbehaviour of the RAID's firmware/controller?
 
> Here are some more things you can try.  Does your system boot?  
Well, that RAID is just one of 3 RAIDs, the system is on the internal PERC-RAID.

> If so, we
> can try sending a few commands to the device via the pass(4) driver and see
> what happens.
 
> First, run 'camcontrol devlist' and see if the array is there and whether
> there is a pass device attached.  If so, try this:
> 
> camcontrol cmd passX -v -c "25 0 0 0 0 0 0 0 0 0" -i 8 "i4 i4"

<IFT A12U-G2421 342D>              at scbus1 target 0 lun 0 (pass3)
# camcontrol cmd pass3 -v -c "25 0 0 0 0 0 0 0 0 0" -i 8 "i4 i4"
-1 512 

 
> That will send a standard 10 byte read capacity command to the device.
> Next, try a 16 byte read capacity.  This is where things are likely failing
> in the da(4) driver attach, and apparantly where things are failing under
> Linux:
> 
> camcontrol cmd passX -v -c "9e 10 0 0 0 0 0 0 0 0 0 0 0 c 0 0" -i 12 "i4 i4 i4"

# camcontrol cmd pass3 -v -c "9e 10 0 0 0 0 0 0 0 0 0 0 0 c 0 0" -i 12 "i4 i4 i4"
camcontrol: error sending command
(pass3:ahc1:0:0:0): SERVICE ACTION IN(16). CDB: 9e 10 0 0 0 0 0 0 0 0 0 0 0 c 0 0 
(pass3:ahc1:0:0:0): CAM Status: Target Bus Phase Sequence Failure

dmesg:
(pass3:ahc1:0:0:0): No or incomplete CDB sent to device.
(pass3:ahc1:0:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(pass3:ahc1:0:0:0): Abort Tag Message Sent
(pass3:ahc1:0:0:0): SCB 8 - Abort Tag Completed.

> If that works, there is some other problem.  If it fails, then we're
> fairly close to the problem.

So, if it's a problem with the RAIDs firmware and/or maybe hardware,
do you expect there's a workaround in FreeBSD for it? 

Regards
Raphael Becker