copy 150G over 100Mbit

[Michael Conlen]

12.5MBytes/sec = 750 MBytes/min = 4500 MBytes/hour.

Two problems, protocol overhead and disk speed.

How fast can you read off that IDE system? Try tarring the filesystem 
and dumping it to /dev/null. If you can't get 12.5MBytes/sec then don't 
look to the network first.You should consider the size of your files. 
Sure your disks can move data fast, however if you have lots and lots of 
really small files your disks are going to be seeking all the time and 
reading very little of it. If your moving one large file there's less 
seek and more read. Also with lots of small files whatever protocol you 
use will have more overhead to eat more of your network bandwith. The 
difference can be an order of magnitude in disk throughput on reads.

Over the network you will end up with all kinds of overhead for various 
protocols. NFS isn't the fastest thing in the world. Rsync has it's 
uses, but I don't think this is one of them.

Consider using cpio over the network. Quick tests show that cpio gets 
better performance than tar by a long shot.

Another consideration might be to move the entire filesystem. If you 
create identical sized partitions on each system, instead of mounting 
them just copy the whole filesystem across using ssh or rsh. I haven't 
tried this on FreeBSD but it shouldn't be to hard to get working. The 
idea being that you read from /dev/ad0s1f on one box and write to 
/dev/ad0s1f on the other. Something like

dd if=/dev/ad0s1f | ssh hostname 'dd of=/dev/ad0s1f'

You can try playing with the bs parameter to get optimal read/write 
speeds. A quick test on my system shows that the disk throughput 
increases as the block size increases up to about 8k from 4.5MBytes/sec 
at 512 to 12MBytes/sec at 8k. On a RAID setup it's likely to be larger.

You may be able to dump to a larger partition on the destination system 
though the extra space would go unused. I don't have a good way to test 
this here.

--
Michael Conlen


Achim Patzner wrote:

>Am 02.06.2003 16:23 Uhr schrieb "Damir Horvat" unter <damir at voljatel.si>:
>  
>
>>I need to copy ~150Gbytes over switched 100Mbit network to new machine (in max
>>7-8 hours).
>>
>>Source machine (IDE RAID 5) is in production, iostat shows average transfer of
>>8-9Mbytes/sec. Destination box is FBC RAID 10.
>>    
>>
>
>Excuse me, but what would you expect using a 100 M*bit*/s network
>connection?
>
>
>Achim
>
>_______________________________________________
>freebsd-performance at freebsd.org mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"
>  
>