unable to access disk block number > 2^32

[Pierre Beyssac]

Hello,

I've acquired a 5-Terabyte SCSI RAID disk that I seem to be unable
to use satisfactorily under FreeBSD. It's most likely a FreeBSD
problem because a Linux 2.6/Knoppix CD on this machine is able to
access the disk nicely.

To make a long story short, any access to a disk block number over
2^32 returns 0 bytes without any error logged anywhere or returned
to userland.

Apparently the read request is correctly routed through by the
kernel to the CAM layer, I checked this using a printf in
/sys/cam/scsi/scsi_all.c in the READ_16 request code. Any ideas
where I should look then? An obvious candidate would be the Symbios
driver but I don't see how it could possibly process a READ_16
request differently from a READ_12 request.

Note: I'm looking for a fix, not for a workaround such as defining
several LUNs under 2TB each and gconcat'ing them.

sym0: <1010-66> port 0x8000-0x80ff mem 0xfc200000-0xfc201fff,0xfc202000-0xfc2023ff irq 48 at device 1.0 on pci3
sym0: Symbios NVRAM, ID 7, Fast-80, LVD, parity checking
sym0: open drain IRQ line driver, using on-chip SRAM
sym0: using LOAD/STORE-based firmware.
sym0: handling phase mismatch from SCRIPTS.
sym0: [GIANT-LOCKED]

da3 at sym0 bus 0 target 9 lun 0
da3: <transtec T6100S16R1-D 341B> Fixed Direct Access SCSI-3 device 
da3: 160.000MB/s transfers (80.000MHz, offset 62, 16bit), Tagged Queueing Enabled
da3: 5338102MB (10932432896 512 byte sectors: 255H 63S/T 680512C)

# dd if=/dev/da3 skip=9000000000 count=100 of=/dev/null
0+0 records in
0+0 records out
0 bytes transferred in 0.000898 secs (0 bytes/sec)
# dd if=/dev/da3 skip=5000000000 count=100 of=/dev/null
0+0 records in
0+0 records out
0 bytes transferred in 0.000620 secs (0 bytes/sec)
# dd if=/dev/da3 skip=2000000000 count=100 of=/dev/null
100+0 records in
100+0 records out
51200 bytes transferred in 0.044276 secs (1156383 bytes/sec)

The only thing that appears in the logs is the following, which
seems mostly harmless (it appears even on blocks < 2^32 and does
not prevent reading/writing to them).

(da3:sym0:0:9:0): phase change 2-7 16 at 607ad968 resid=10.
(da3:sym0:0:9:0): phase change 2-7 16 at 607ad968 resid=10.
(da3:sym0:0:9:0): phase change 2-7 16 at 0035b368 resid=10.
(da3:sym0:0:9:0): phase change 2-7 16 at 0035b368 resid=10.
(da3:sym0:0:9:0): phase change 2-7 16 at 60c2fb68 resid=10.
(da3:sym0:0:9:0): phase change 2-7 16 at 60c2fb68 resid=10.
(da3:sym0:0:9:0): phase change 2-7 16 at 002eb968 resid=10.
(da3:sym0:0:9:0): phase change 2-7 16 at 002eb968 resid=10.
(da3:sym0:0:9:0): phase change 2-7 16 at 002eb968 resid=10.

Under Linux the disk works like a charm (but only Reiserfs is able
to newfs it):

root at ttyp1[knoppix]# dd if=/dev/sdd skip=9000000000 of=/dev/null count=1000
1000+0 records in
1000+0 records out
512000 bytes transferred in 0.006852 seconds (74721797 bytes/sec)

root at ttyp1[tmp]# mount -t reiserfs /dev/sdd /mnt
root at ttyp1[tmp]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 3471       948      2523  28% /
/dev/scd0               715764    715764         0 100% /cdrom
/dev/cloop             1947338   1947338         0 100% /KNOPPIX
/ramdisk               1644620      2396   1642224   1% /ramdisk
/dev/sdd             5466049628     32840 5466016788   1% /mnt
root at ttyp1[tmp]#

-- 
A: Yes.					  Pierre Beyssac pb at enst.fr
>Q: Are you sure?
>>A: Because it reverses the logical flow of conversation.
>>>Q: Why is top posting annoying in email?