Dreamplug stable/10 usb - sd file transfer corruption error - cache related?

[Steve]

Ian,

Thanks for testing this out on your setup.  Given that you can't reproduce the problem, I guess it's either a bug that's been eliminated from the 10 stable branch over the last few months, or some specific problem with my hardware (maybe driver issue specific to my thumb drive).  If I get a chance to revisit this and get any useful data out of it I'll let you know.

- Steven

Sent from my iPad

> On Jan 7, 2014, at 7:55 PM, Ian Lepore <ian at FreeBSD.org> wrote:
> 
> Oddly enough, I never forgot about this, I just never got to it until
> today.  I put a concerted effort into getting data corruption on
> usb->usb copying of files and even entire filesystems, and couldn't get
> a glitch.  I used the external sdcard (da1) and a thumb drive.  I used
> 'mtree -ck md5' on both source and destination, then diff'd the results,
> and the files were all identical.
> 
> I tested with 11-current (@r260393) and 10-stable (@r260394) on my
> DreamPlug, compiled with clang-eabi in both cases.
> 
> -- Ian
> 
>> On Fri, 2013-10-25 at 18:40 -0700, Steven Lee wrote:
>> Ian, thanks for taking a look to see if you can recreate the problem.  In
>> case it makes any difference in behavior, the
>> command I used for the transfer was:
>> 
>> (dump -0Lf - /) | (cd /mnt/flash_root ; restore -rf -)
>> 
>> with /dev/da2s2 mounted at / and /dev/da0s2 mounted at /mnt/flash_root
>> 
>> Regards,
>> 
>> Steve
>>> On Fri, Oct 25, 2013 at 3:01 PM, Ian Lepore <ian at freebsd.org> wrote:
>>> 
>>> That old advice was based on bugs in old code that has been fixed.  I've
>>> just been double-checking that the various fixes are in 10 (some of them
>>> were things I submitted a PR for before I became a committer).  I'm not
>>> sure about the usb driver, but there have been big changes in the dma
>>> cache coherency code between 9 and 10, although much of that is the
>>> fixes I was alluding to.
>>> 
>>> Basically what you're doing is a usb->usb copy, I'll see if I can
>>> recreate the corruption on my dreamplug here the same way.
>>> 
>>> -- Ian
>>> 
>>>> On Fri, 2013-10-25 at 14:38 -0700, Steven Lee wrote:
>>>> ---------- Forwarded message ----------
>>>> From: Steven Lee <ssl.qlt at gmail.com>
>>>> Date: Fri, Oct 25, 2013 at 2:37 PM
>>>> Subject: Re: Dreamplug stable/10 usb - sd file transfer corruption error
>>> -
>>>> cache related?
>>>> To: Ian Lepore <ian at freebsd.org>
>>>> 
>>>> 
>>>> Ian, thanks for your reply.
>>>> 
>>>> I am using the DREAMPLUG-1001 stock kernel config, and you are right the
>>>> internal sd card shows up as a usb device, /dev/da0.
>>>> 
>>>> As I mentioned, I tried modifying pmap.c as per one of your suggestions
>>> to
>>>> an earlier poster / problem, but it didn't seem to help the issue.  I
>>>> changed the pmap_pte_init_arm10() function to look like the
>>>> pmap_pte_init_arm9() function.
>>>> 
>>>> Any other thoughts about things to try?  Since it seems like this is a
>>>> regression between 9 and 10, are you aware of any specific cache related
>>>> changes to the usb driver between the branches?
>>>> 
>>>> - Steve
>>>> 
>>>> 
>>>>> On Fri, Oct 25, 2013 at 2:03 PM, Ian Lepore <ian at freebsd.org> wrote:
>>>>> 
>>>>>> On Fri, 2013-10-25 at 13:37 -0700, Steven Lee wrote:
>>>>>> Hello,
>>>>>> 
>>>>>> In the process of upgrading my Dreamplug from the stable/9 branch to
>>> the
>>>>>> stable/10 branch, I have run into a fairly
>>>>>> significant file corruption problem.  The steps that create this
>>> problem
>>>>>> are as follows:
>>>>>> 
>>>>>> 1. Cross-compile the kernel and buildworld from the stable/10 branch
>>> on a
>>>>>> separate host.
>>>>>> 2. Install the kernel and installworld on a usb stick drive
>>>>>> 3. Boot the Dreamplug from the usb drive
>>>>>> 
>>>>>> There are no problems that show up to this point, the Dreamplug boots
>>>>> with
>>>>>> no errors.
>>>>>> 
>>>>>> 4. Copy the kernel from the usb drive to the internal sd card
>>>>>> 5. Copy the root filesystem from the usb drive to the internal sd
>>> card
>>>>>> using (dump | restore)
>>>>>> 
>>>>>> It is at step 5 that the error manifests itself, as I found that
>>>>>> approximately 20 shell scripts in the /etc/rc.d directory
>>>>>> had been randomly corrupted with strings of null characters (in
>>> groups of
>>>>>> 32).  I assume that the rest of the file system
>>>>>> was compromised in a similar fashion.  The bug is repeatable,
>>> however the
>>>>>> corruption is somewhat random, so
>>>>>> each time different files are corrupted in different places.
>>>>>> 
>>>>>> When I followed the exact same steps from the stable/9 branch, the
>>>>> problem
>>>>>> did not occur, so there is clearly some
>>>>>> type of regression error between the 9 and 10 branches.
>>>>>> 
>>>>>> After searching through the arm mailing list, I attempted to work
>>> around
>>>>>> the problem by:
>>>>>> - mounting the file systems with -o noclusterr -o noclusterrw
>>>>>> - mounting the file systems with -o sync
>>>>>> - as per comments on bug arm/158950, attempted to modify the pmap.c
>>>>>> functions to change the cache from
>>>>>> write-back mode to write-through mode
>>>>>> 
>>>>>> but all of these attempts were ineffective.
>>>>>> 
>>>>>> Given that I am relatively new to FreeBSD, I was hoping to get any
>>>>> insights
>>>>>> as to any next steps that might make
>>>>>> sense in terms of either working around the problem, narrowing down
>>> the
>>>>>> bug, or any obvious rookie mistakes I
>>>>>> am making before I give up and revert back to the 9 branch.
>>>>>> 
>>>>>> Thanks for your help!
>>>>>> 
>>>>>> Regards,
>>>>> 
>>>>> On my dreamplug (model 1001) both the internal and external sd cards
>>> are
>>>>> actually usb devices (they show up as /dev/da0 and da1, not mmcsd0/1).
>>>>> I'm not sure if that's true on all models or not.
>>>>> 
>>>>> Are you using the stock kernel config (DREAMPLUG-1001)?  If not, have
>>>>> you got "option USB_HOST_ALIGN=32" in your kernel config?
>>>>> 
>>>>> Corruption in 32 byte chunks is almost always partial cacheline flush
>>>>> problems, but I haven't seen that happen on dreamplug for a long time
>>>>> (and when I did see it, it was with a sata drive).
>>>>> 
>>>>> -- Ian
>>>> _______________________________________________
>>>> freebsd-arm at freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-arm
>>>> To unsubscribe, send any mail to "freebsd-arm-unsubscribe at freebsd.org"
>> _______________________________________________
>> freebsd-arm at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-arm
>> To unsubscribe, send any mail to "freebsd-arm-unsubscribe at freebsd.org"
> 
>