Slow disk access while rsync - what should I tune?

[cronfy]

Hello,

I have a web-server (nginx + apache + mysql, FreeBSD 7.3) with many
sites. Every night it creates a backup of /home on a separate disk.
/home is a RAID1 mirror on Adaptec 3405 (128M write cache) with SAS
drives; /backup is a single SATA drive on the same controller.

Rsync creates backups using hardlinks, it stores 7 daily and 4 weekly
copies. Total amount of data is ~300G and 11M of files. The server is
under heavy web load every time (appox 100 queries/sec).

Every time backup starts server slows down significantly, disk
operations become very slow. It may take up to 10 seconds to stat() a
file that is not in filesystem cache. At the same time, rsync on
remote server does not affect disk load much, server works without
slowdown.

I think that problem can be caused by two reasons:

 * either bulk of reads on SATA /backup drive, that fills OS
filesystem cache and many file access operations require real disk
read.
 * or bulk of writes on /backup fills controller write cache and geom
disk operations queue grown, causing  all disk operations to wait.

This is only my assumption of course, I may be wrong.

How can I find a real reason of these slowdowns, to either conclude
that it is not possible to solve this because of hardware/software
limits, or tune my software/hardware system to make this all work at
an acceptable speed?

Here is my current sysctl  setup (what should I tune?):

kern.maxvnodes=500000
vfs.ufs.dirhash_maxmem=67108864
vfs.lookup_shared=1

kern.dirdelay=6
kern.metadelay=5
kern.filedelay=7

sysctl's counters (which others should I monitor?):

vfs.numvnodes: 407690
vfs.ufs.dirhash_mem: 27158118

I tried to enable async (in hope it will make rsync faster) or even
disable softupdates on /backup partition (in hope it will make rsync
slower and OS filesytem cache will not be flushed by backups), it did
not help. I also want to try to upgrate to Adaptec 5405 (it has 256M
of write cache) or move mysql databases on a separate SAS disk, but I
just not quite sure what will help better.

What would I start from to diagnose the issue?

Thanks in advance!

-- 
// cronfy