sending large amount of data securely

[Alex Zbyslaw]

Nathan Vidican wrote:

> Dave wrote:
>
>> Hello,
>>    I've got two freebsd6 boxes on one i have a large collection of 
>> files, like about 3.5 gb worth, that i want to send securely to the 
>> other box. I've tried:
>>
>> scp -24Cpr
>>
>> to do it, it transfered 1gb of data then quit. I'm suspecting it 
>> didn't get the sym links if any in this data. Any suggestions 
>> appreciated.
>> Thanks.
>> Dave.
>>
>>
> We mirror (nightly) about 300GB worth of data accross one master and 
> three slave servers (three copies of 300GB of data nightly) using 
> rsync, we do not use ssh as it's a local-only gigabit ethernet link 
> between servers, but it's not overly difficult to accomplish in your 
> case.
>
> One thing to note; rsync will allow you to keep data synchronized by 
> only copying the difference from one directory tree to another, so in 
> our case the actual nightly transfer is only in the neighborhood of 
> about 30GB, and it takes about 1/2hr to complete to all slaves in 
> sequence.
>
> Hope it helps, but my two cents - use rsync for that.
>
If you do need to use ssh with rsync, then you could try out the latest 
openssh-portable port which now seems to support the HPN (high 
performance networking) patches.  (There's also a ssh-hpn port, but I'm 
guessing that will become redundant).  HPN improved my data transfer 
over gigabit by a factor of up to 4 - without them gigabit was actually 
slower than 100Mbit.  The patches also support a "no encryption" option 
which is useful for internal networks as you get all the authentication 
without unnecessary overhead in the copying.  I've seen reports that HPN 
patches also improve ssh performance between different OSes, but haven't 
tried that myself.  Why those patches aren't yet part of ssh by default 
is beyond me.

--Alex

PS the only drawback of rsync is that it won't mirror "flags" (nodump, 
schg etc).  But then neither would scp :-)

PPS Using compression with SSH may actually make things slower depending 
on your CPU, load etc.  You should benchmark with and without if you are 
considering using it.  Over a modem it might be a no-brainer, but 
anything faster is much less obvious, especially if your data set 
doesn't compress well, which we have no way of knowing.