Slow disk access while rsync - what should I tune?

[krad]

On 1 November 2010 14:11, Daan Vreeken <Daan at vehosting.nl> wrote:

> Hi Cronfy,
>
> On Saturday 30 October 2010 23:48:45 cronfy wrote:
> > Hello.
> >
> > > Every time backup starts server slows down significantly, disk
> > > operations become very slow. It may take up to 10 seconds to stat() a
> > > file that is not in filesystem cache. At the same time, rsync on
> > > remote server does not affect disk load much, server works without
> > > slowdown.
> >
> > Thank you all for the answers.
> ...
> > Also, is it possible to limit disk operations for rm -rf somehow? The
> > only idea I have at the moment is to replace rm -rf with 'find |
> > slow_down_script | xargs rm' (or use similar patch as for rsync)...
>
> Yes there is. You could use the same 'trick' I've added to rsync and limit
> the
> amount of I/O-creating system calls an application creates.
> You could even create a small wrapper library that does this for a specific
> application, without having to recompile or change the application.
>
> You can find a working proof of concept in "slowdown.c" here :
>        http://vehosting.nl/pub_diffs/
>
> The library can be compiled with :
>        gcc -Wall -fPIC -shared -o slowdown.so slowdown.c
>
> Then start the application you want to I/O-limit with something like :
>        (
>                export LD_PRELOAD=slowdown.so
>                export LD_LIBRARY_PATH=.:${LD_LIBRARY_PATH}
>                ls -R /a/random/huge/directory/
>        )
>
> (Assuming you start the application from withing the directory
> where "slowdown.so" resides.)
> This should work with rsync, ls and rm "out of the box", without changing
> the
> source of the applications.
>
>
> Regards,
> --
> Daan Vreeken
> VEHosting
> http://VEHosting.nl
> tel: +31-(0)40-7113050 / +31-(0)6-46210825
> KvK nr: 17174380
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"
>


Just a wild punt here, but are you using zfs on both systems? If you are
look at doing incremental zfs sends as an alternative.