mfi corrupts JBOD disks >2TB due to LBA overflow (was: ZFS corruption due to lack of space?)

[Steven Hartland]

Copying in freebsd-scsi@ for visability.

----- Original Message ----- 
From: "Steven Hartland"
> Ok after revisiting all the facts and spotting that
> the corruption only seemed to happen after my zpool
> was nearly full I came up with a wild idea, could
> the corruption be being caused by writes after 2TB?
> 
> A few command lines latter and this was confirmed
> writes to the 3TB disks under mfi are wrapping at
> 2TB!!!
> 
> Steps to prove:-
> 1. zero out block 1 on the disk
> dd if=/dev/zero bs=512 count=1 of=/dev/mfisyspd0
> 1+0 records in
> 1+0 records out
> 512 bytes transferred in 0.000728 secs (703171 bytes/sec)
> 
> 2. confirm the first block is zeros
> dd if=/dev/mfisyspd0 bs=512 count=1 | hexdump -C
> 1+0 records in
> 1+0 records out
> 512 bytes transferred in 0.000250 secs (2047172 bytes/sec)
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
> *
> 00000200
> 
> 3. write 1 block random after the 2TB boundary
> dd if=/dev/random bs=512 count=1 of=/dev/mfisyspd0 oseek=4294967296
> 1+0 records in
> 1+0 records out
> 512 bytes transferred in 0.000717 secs (714162 bytes/sec)
> 
> 4. first block of the disk now contains random data
> dd if=/dev/mfisyspd0 bs=512 count=8 | hexdump -C
> 00000000  9c d1 d2 1d 9f 2c fc 30  ab 09 7a f7 64 16 2a 58  |.....,.0..z.d.*X|
> 00000010  18 27 9d 1f ae 4d 27 53  1a 50 e7 c1 b1 3a 9b e4  |.'...M'S.P...:..|
> 00000020  c3 7c d0 25 83 e2 bd 85  33 f2 33 8e 71 55 70 7c  |.|.%....3.3.qUp||
> 00000030  8c 15 af 55 f6 88 8d 6e  40 1c f3 1a 5c e7 80 4b  |...U...n at ...\..K|
> ...
> 
> Looking at the driver code the problem is that IO on syspd
> disks aka JBOD is always done using 10 byte CDB commands
> in mfi_build_syspdio. This is clearly a serious problem as
> it results in total corruption on disks > 2^32 sectors
> when sectors above 2^32 are accessed.
> 
> The fix doesn't seem too hard and I think I've already
> got a basic version working, just needs more testing need.
> 
> The bug also effects kernel mfi_dump_blocks but thats
> less likely to trigger due to how its used.
> 
> Will create PR when I've finished testing and am happy
> with the patch, but wanted to let others know in the
> mean time given how serious the bug is.

PR which includes a patch which fixes this issue is:-
http://www.freebsd.org/cgi/query-pr.cgi?pr=173291

Given its critical nature I would strongly advise this gets
MFC'ed to all branches ASAP.

While someone is looking at this would be good to get the
following mfi related PR's I've submitted could also be
committed as well ;-)

Add deviceid to mfi disk startup output
http://www.freebsd.org/cgi/query-pr.cgi?pr=173290

Improvements to mfi support including foreign disks / configs
in mfiutil
http://www.freebsd.org/cgi/query-pr.cgi?pr=172091

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.