rsync w/ fake-super -> crashes zfs

[javocado]

A bit more info:

We've been able to reproduce the crash with panic output:

Fatal trap 12: page fault while in kernel mode
cpuid = 11; apic id = 15
fault virtual address   = 0x160
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80abb546
stack pointer           = 0x28:0xffffffa59b580910
frame pointer           = 0x28:0xffffffa59b5809d0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 93937 (rsync)
[thread pid 93937 tid 102776 ]
Stopped at      zfs_freebsd_remove+0x426:       movq    0x160(%rax),%rsi
db>


by simply adding --delete to this command:

rsync -av --rsync-path="rsync --fake-super" /src /dst

the file it was deleting at the time of the crash had valid (readable)
extattrs (rsync.%stat - for the fake-super info) and nothing else special
for acl

            owner@:rw-p--aARWcCos:------:allow
            group@:r-----a-R-c--s:------:allow
         everyone@:r-----a-R-c--s:------:allow


Question: should it be safe to remove that file traditionally, as in:

rm /file/that/zfs+rsync+fake_super/croaked/on

How's would that rm differ from what rsync was trying to do?



On Thu, Mar 20, 2014 at 5:33 PM, javocado <javocado at gmail.com> wrote:

> Gathering more info here...
>
> When re-running the rsync command, stripped down:
>
> rsync -av --rsync-path="rsync --fake-super" /src /dst
>
> we're not seeing any system crashes, yet, but rsync is choking/quitting
> quite a bit saying a file here or there has a corrupt extattr. Does this
> point to a problem with ZFS or more with rsync.
>
> Any thoughts on the fake-super impact and how it leads to crashing?
>
>
> On Fri, Mar 14, 2014 at 4:57 PM, javocado <javocado at gmail.com> wrote:
>
>> System specifics:
>> ZFS version 28
>> FreeBSD 8.3-RELEASE
>>
>> We're seeing a repeatable outcome where a remote rsync command like:
>>
>> rsync -axzHAXS --rsync-path="rsync --fake-super" --exclude '*/rsync.%stat'
>>
>> backing up to our zfs filesystem (with 15M inodes) will lead to a panic
>> with output like:
>>
>> Fatal trap 12: page fault while in kernel modecpuid = 4; apic id = 04fault virtual address   = 0x160fault code              = supervisor read data, page not presentinstruction pointer     = 0x20:0xffffffff80abb546stack pointer           = 0x28:0xffffff976c62b910frame pointer           = 0x28:0xffffff976c62b9d0code segment            = base 0x0, limit 0xfffff, type 0x1b                        = DPL 0, pres 1, long 1, def32 0, gran 1processor eflags        = interrupt enabled, resume, IOPL = 0current process         = 7295 (rsync)[thread pid 7295 tid 101008 ]Stopped at      zfs_freebsd_remove+0x426:       movq    0x160(%rax),%rsi
>>
>>
>> On the sending side (RHEL, ext3), rsync reports errors like:
>>
>> rsync: failed to read xattr rsync.%statrsync: failed to write xattr rsync.%statrsync: get_xattr_names: llistxattr
>>
>> which we've seen occasionally with other systems when running rsync with fake-super, but it usually doesn't lead to a crash.*
>>
>> On the receiving side, other than the crashes, we are seeing a few new files (that don't exist on the source) named:
>>
>> rsync.%stat
>>
>> which correspond to and contain the owner and permission attributes that should have been stored in the extattr's for the file or directory. Not sure if they are a red herring, but they're usually not something we see (perhaps that's related to the --exclude '*/rsync.%stat' and rsync not being able to cleanup properly).
>>
>> We are still testing to see if any options in the rsync command (above) may be contributing to the crash, since fake-super in and of itself runs fine under basic (rsync -av --rsync-path="rsync --fake-super" /src /dst) circumstances. But we suspect that the problem is related to fake-super and its reliance upon extattr's.
>>
>> What we really need is a solution to the crashing - some way to make ZFS stop choking on whatever --fake-super produces and/or how it's interacting with extattr's on ZFS.
>>
>>
>> Thanks!
>>
>> * we sometimes also see on the sending side w/ fake-super:
>> rsync: failed to write xattr rsync.%stat for "xxxxxx/file" : No such file
>> or directory (2)
>>
>> when (1) the file exists, (2) it's a symlink
>> but that isn't happening in this instance. We only mention it here as
>> another oddity of fake-super + ZFS + extattr
>>
>>
>