Slow disk access while rsync - what should I tune?

[cronfy]

Hello.

> Every time backup starts server slows down significantly, disk
> operations become very slow. It may take up to 10 seconds to stat() a
> file that is not in filesystem cache. At the same time, rsync on
> remote server does not affect disk load much, server works without
> slowdown.

Thank you all for the answers.

Matthew, yes, I know about --bwlimit in rsync, but this will mostly
slow down the transfer of data and not  stat() operations that are
used for comparing files. I afraid bwlimiting will not make any better
(I tried it some time ago without success).

Daan, thanks for the patch! I will try it.

A lot of impact also produced by rm -rf of old backups. I assume that
low performance is also related to a large numbers of hardlinks. There
was a moment when I had ~15 backups hardlinked by rsync, and rm -rf of
single backup was VERY slow and slowed down the server dramatically. I
had no choice except installing a new clean disk for backups, limiting
number of future backups and issuing a newfs over old backup disk.
When there is less number of hardlinked copies, backups cleanup works
much better. Can large number of hardlinks produce such an impact on
filesystem operations?

May be it is possible to increase disk performance somehow? Server has
a lot of memory. At this time vfs.ufs.dirhash_maxmem = 67108864 (max
monitored value for vfs.ufs.dirhash_mem was 52290119) and
kern.maxvnodes = 500000 (max monitored value for vfs.numvnodes was
450567). Can increasing of these (or other) sysctls help? I ask
because (as you can see) these tunables are already incremented, and I
am not sure further increment really makes sense.

Also, is it possible to limit disk operations for rm -rf somehow? The
only idea I have at the moment is to replace rm -rf with 'find |
slow_down_script | xargs rm' (or use similar patch as for rsync)...

Or, probably, it is possible to limit the IO bandwith for a particular
device somehow? I would then limit disk operations for SATA backup
disk to prevent it affecting the rest of services that work on SAS
mirror.

And also, maybe there are other ways to create incremental backups
instead of using rsync/hardlinks? I was  thinking  about generating
list of changed files with own script and packing it with tar, but I
did not find a way to remove old backups with such an easy way as it
is with hardlnks..

Thanks in advance!


-- 
// cronfy