What would be the best way to copy lots of files from oneserver to another.

[Simon]

I recently used rsync to copy ~35gb worth of data, 1,170,168 inodes,
without any problem over 100mbps interface. Took 3 or so hours.
You do need lots of RAM, if you swap, you're screwed. RAM
usage would depend on number of files you need to copy at once.

-Simon

On Sat, 21 Jun 2003 11:26:28 -0400 (EDT), Andy Dills wrote:

>On Fri, 20 Jun 2003, Chuck Swiger wrote:
>
>> Etienne Ledoux wrote:
>> > I need to copy lots of directories/files from one server to another. Approx
>> > 45Gb . These files are mailboxes stored in maildir format. What would be the
>> > best/safest/quickest way to do this ?
>>
>> Putting the disks onto one machine and copying locally is going to be a win
>> considering the amount of data you have, but if you have to copy over the
>> network, consider using something like rsync.
>
>I love rsync and use it for lots of things.
>
>A one time move of 45 GB is not something I would use rsync for.
>
>Rsync is good for just what is says it's for: keeping thing in sync.
>
>Rsync will build metadata information about the ENTIRE data set before
>moving a single bit. This is very costly, making rsync's utility plummet
>as the data set size increases. You'll find it preferrable to write a
>script to traverse and descend directories, spawning numerous rsync
>processes for smaller sized directories.
>
>
>Of course, considering that it's a one-time move, I would advise
>physically moving the drive, to remove any sort of network involvement.
>
>I've been in your position before, and I can promise you that even using
>NFS and spawning multiple moves (mv /var/mail/a* /mnt &; mv /var/mail/b*
>/mnt &; etc.) will quickly outpace rsync.
>
>The number of concurrent processes involved in the move can really only be
>determined by watching the aggregate traffic to your server. You'll see it
>level off at a certain point (for me it was X=7), after which additional
>processes don't yield any more optimization.
>
>Andy
>
>---
>Andy Dills
>Xecunet, Inc.
>www.xecu.net
>301-682-9972
>---
>
>_______________________________________________
>freebsd-isp at freebsd.org mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-isp
>To unsubscribe, send any mail to "freebsd-isp-unsubscribe at freebsd.org"
>