process size

[Jason Evans]

Alexandre Biancalana wrote:
> I'm trying to copy ~800GB to my new backup server via rsync, but the rsync
> process die when it's vsz is close to 1G with the following message:
> 
> # rsync -av 192.168.0.13::root/backup/* .
> receiving file list ... ERROR: out of memory in make_file
> rsync error: error allocating core memory buffers (code 22) at util.c(120)
> [sender=2.6.8]
> rsync: connection unexpectedly closed (155637057 bytes received so far)
> [receiver]
> rsync error: error in rsync protocol data stream (code 12) at io.c(462)
> [receiver=2.6.9]
> 
> 
> Here is the last ps -aux line of the process before die:
> 
> root 856 0.4 68.1 915248 707048 p0 S+ 6:32PM 0:12.51 rsync -av
> 192.168.0.13::root/backup/* .

You're running your system as i386 rather than amd64, right?  It looks 
like rsync is exhausting its address space while trying to reallocate 
the growing (and apparently very large) file list.  There's nothing 
surprising here to me.  If you use amd64 rather than i386 you won't have 
this problem, though you will still see poor performance due to 
swapping.  The most prudent solution is probably to use multiple rsync 
calls to copy portions of your data at a time.

Jason