Slow disk access while rsync - what should I tune?

[Daan Vreeken]

Hi Cronfy,

On Saturday 30 October 2010 23:48:45 cronfy wrote:
> Hello.
>
> > Every time backup starts server slows down significantly, disk
> > operations become very slow. It may take up to 10 seconds to stat() a
> > file that is not in filesystem cache. At the same time, rsync on
> > remote server does not affect disk load much, server works without
> > slowdown.
>
> Thank you all for the answers.
...
> Also, is it possible to limit disk operations for rm -rf somehow? The
> only idea I have at the moment is to replace rm -rf with 'find |
> slow_down_script | xargs rm' (or use similar patch as for rsync)...

Yes there is. You could use the same 'trick' I've added to rsync and limit the 
amount of I/O-creating system calls an application creates.
You could even create a small wrapper library that does this for a specific 
application, without having to recompile or change the application.

You can find a working proof of concept in "slowdown.c" here :
	http://vehosting.nl/pub_diffs/

The library can be compiled with :
	gcc -Wall -fPIC -shared -o slowdown.so slowdown.c

Then start the application you want to I/O-limit with something like :
	(
		export LD_PRELOAD=slowdown.so
		export LD_LIBRARY_PATH=.:${LD_LIBRARY_PATH}
		ls -R /a/random/huge/directory/
	)

(Assuming you start the application from withing the directory 
where "slowdown.so" resides.)
This should work with rsync, ls and rm "out of the box", without changing the 
source of the applications.


Regards,
-- 
Daan Vreeken
VEHosting
http://VEHosting.nl
tel: +31-(0)40-7113050 / +31-(0)6-46210825
KvK nr: 17174380