zfs send/recv: STILL invalid Backup Stream

[Mark Martinec]

> On 2014-07-24 19:56, Allan Jude wrote:
>>> or better yet:
>>>   ssh root at tbh.lerctr.org "zfs send ..." | mbuffer -m 16M | zfs recv 
>>> ...
>>> (The misc/mbuffer compensates for bursty zfs reads and writes.
>>>  A note to myself: I should suggest to Allan to add mbuffer
>>>  in a pipe as used in sysutils/zxfer, instead of patching zxfer
>>>  for our local use :)
>> 
>> zxfer can already do this, with the -D option
>> I actually use misc/clpbar and get a progress bar as well
>> 
>> -D 'bar -s %%size%% -bl 1m -bs 128m'
>> 
>> or in your case: -D 'mbuffer -m 16M'

Thank you Allan! It shows it's been a while since the last time
I inspected the guts of zxfer.



2014-07-25 06:43 Larry Rosenman wrote:
> Ok, I did the mbuffer trick, and the SEND side is where the memory 
> issue is:
> borg.lerctr.org /home/ler/bin $ tail zfs-send.log
> 23:28:12   15.7G   zroot/home at 2014-07-24_22:56
> 23:28:13   15.7G   zroot/home at 2014-07-24_22:56
> 23:28:14   15.7G   zroot/home at 2014-07-24_22:56
> 23:28:15   15.7G   zroot/home at 2014-07-24_22:56
> 23:28:16   15.7G   zroot/home at 2014-07-24_22:56
> 23:28:17   15.7G   zroot/home at 2014-07-24_22:56
> 23:28:18   15.7G   zroot/home at 2014-07-24_22:56
> 23:28:19   15.7G   zroot/home at 2014-07-24_22:56
> 23:28:20   15.8G   zroot/home at 2014-07-24_22:56
> Write failed: Cannot allocate memory
> borg.lerctr.org /home/ler/bin $

Good information. So the "Write failed: Cannot allocate memory"
on the send side is what is causing the "invalid backup stream"
on the receiving side.


> borg.lerctr.org /home/ler/bin $ tail zfs-recv.log
> cannot receive new filesystem stream: invalid backup stream
> borg.lerctr.org /home/ler/bin $
> 
> borg.lerctr.org /home/ler/bin $  cat backup-TBH-ZFS-initial.sh
> #!/bin/sh
> DATE=`date "+%Y-%m-%d_%H:%M"`
> #DATE2=2013-03-24
> #DATE2=`date -v "-1d" "+%Y-%m-%d"`
> # snap the source
> ssh root at tbh.lerctr.org zfs snapshot -r zroot@${DATE}
> # zfs copy the source to here.
> ssh root at tbh.lerctr.org 2>zfs-send.log "zfs send  -v -R zroot@${DATE} " 
> | \
>      mbuffer -m 16M 2>mbuffer.log | \
>      zfs recv -F -u -v -d zroot/backups/TBH3 2>zfs-recv.log
> # make sure we NEVER allow the backup stuff to automount.
> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH3| \
>     awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
> borg.lerctr.org /home/ler/bin $
> 
> borg.lerctr.org /home/ler/bin $ ssh tbh vmstat -z
> ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL 
> SLEEP
[...]
> Where to now?

Don't know, I'd guess some network-related memory limit is being hit
on the sending site.

Why not try to decouple the 'zfs send' from a network copy and ssh:
Login to a remote side, do a 'zfs send' to some local temporary file
there, then feed that file to ssh and send it over to a home host,
where it can be piped into some simple program like md5 or 'wc -c'
instead of a zfs recv.

   Mark