copy 150G over 100Mbit

[David Gilbert]

>>>>> "Damir" == Damir Horvat <damir at voljatel.si> writes:

Damir> Hi!  I need to copy ~150Gbytes over switched 100Mbit network to
Damir> new machine (in max 7-8 hours).

Damir> Source machine (IDE RAID 5) is in production, iostat shows
Damir> average transfer of 8-9Mbytes/sec. Destination box is FBC RAID
Damir> 10.

Damir> I've tryed rsync and cp over NFS. Both came down roughly to
Damir> 2Gig's per hour, which is unacceptable. The only option left
Damir> (as I see it) is to try with cross-over cable.

Damir> I don't care for security as all transfers are in private
Damir> network.

Damir> Any ideas, how to approach this problem?

Here are a few notes.

As others have said, read and write speed are big factors.  If you
already have an 8 to 9 Mb/sec read rate on the old system, you may not
be able to achieve another 12Mb/sec to fill the link.  Also, many
motherboards are going to have an overall limit to performance which
is non linear (depends on number of cards and how much bandwidth each
one does).

NFS and rsync are not your best tools.  They have no streaming.  You
want buffers at every point to eliminate bottlenecks.

On one machine you might do:

tar -cvf - . | team 1m 16 | nc two 6464

On the other:

nc -l 6464 | team 1m 16 | tar -xvf -

Someone also mentioned cpio being faster... I don't know if that will
be true with enough buffering.  Team (a port) is a good friend to have
around.

Softupdates will likely save your but.  Make sure it's on for the
target.  You might even try the following sequence:

touch FOOBAR
do the backup
wait for final day
do the backup with files newer than FOOBAR

most utilities will have a -newer switch.

Dave.

-- 
============================================================================
|David Gilbert, Velocet Communications.       | Two things can only be     |
|Mail:       dgilbert at velocet.net             |  equal if and only if they |
|http://daveg.ca                              |   are precisely opposite.  |
=========================================================GLO================