copy 150G over 100Mbit
ahk at spb.edu
Mon Jun 2 08:08:40 PDT 2003
On 2 Jun 2003 at 11:05, Michael Conlen wrote:
> 12.5MBytes/sec = 750 MBytes/min = 4500 MBytes/hour.
12,5MBytes/sec = 750 MBytes/min = 45000 Mbytes/hour
> Two problems, protocol overhead and disk speed.
> How fast can you read off that IDE system? Try tarring the filesystem
> and dumping it to /dev/null. If you can't get 12.5MBytes/sec then
> don't look to the network first.You should consider the size of your
> files. Sure your disks can move data fast, however if you have lots
> and lots of really small files your disks are going to be seeking all
> the time and reading very little of it. If your moving one large file
> there's less seek and more read. Also with lots of small files
> whatever protocol you use will have more overhead to eat more of your
> network bandwith. The difference can be an order of magnitude in disk
> throughput on reads.
> Over the network you will end up with all kinds of overhead for
> various protocols. NFS isn't the fastest thing in the world. Rsync has
> it's uses, but I don't think this is one of them.
> Consider using cpio over the network. Quick tests show that cpio gets
> better performance than tar by a long shot.
> Another consideration might be to move the entire filesystem. If you
> create identical sized partitions on each system, instead of mounting
> them just copy the whole filesystem across using ssh or rsh. I haven't
> tried this on FreeBSD but it shouldn't be to hard to get working. The
> idea being that you read from /dev/ad0s1f on one box and write to
> /dev/ad0s1f on the other. Something like
> dd if=/dev/ad0s1f | ssh hostname 'dd of=/dev/ad0s1f'
> You can try playing with the bs parameter to get optimal read/write
> speeds. A quick test on my system shows that the disk throughput
> increases as the block size increases up to about 8k from
> 4.5MBytes/sec at 512 to 12MBytes/sec at 8k. On a RAID setup it's
> likely to be larger.
> You may be able to dump to a larger partition on the destination
> system though the extra space would go unused. I don't have a good way
> to test this here.
> Michael Conlen
> Achim Patzner wrote:
> >Am 02.06.2003 16:23 Uhr schrieb "Damir Horvat" unter
> ><damir at voljatel.si>:
> >>I need to copy ~150Gbytes over switched 100Mbit network to new
> >>machine (in max 7-8 hours).
> >>Source machine (IDE RAID 5) is in production, iostat shows average
> >>transfer of 8-9Mbytes/sec. Destination box is FBC RAID 10.
> >Excuse me, but what would you expect using a 100 M*bit*/s network
> >freebsd-performance at freebsd.org mailing list
> >To unsubscribe, send any mail to
> >"freebsd-performance-unsubscribe at freebsd.org"
> freebsd-performance at freebsd.org mailing list
> To unsubscribe, send any mail to
> "freebsd-performance-unsubscribe at freebsd.org"
More information about the freebsd-performance