Anybody seeing problems with large TCP streams over rsync?

[Kris Kennaway]

João Carlos Mendes Luís wrote:
> Hi folks,
> 
>    I am migrating a backup server from a server running FreeBSD 6.2/i386 
> to another server running FreeBSD 7/amd64.  It is currently running 
> 7.0-BETA4, as distributed from the CD image.
> 
>    Both machines have SATA discs and some mix of gmirror and gstripe of 
> its partitions.
> 
>    Since it is a Backup server, I am in the process of transferring 
> large files from one to another.  But I always have trouble when 
> transferring large files (75GB).
> 
>    The first trouble I had was in the back-to-back Gigabit Ethernet 
> connection, to speed-up transfers.  Somehow, data got quickly screwed, 
> even above TCP layer!
> 
>    One side is an onboard nVidia nfe(4), and the other a generic Realtek 
> re(4).  The troubles went on, even after I disable TSO (on fbsd 7), 
> RXCSUM and TXCSUM on both sides.  They went on even after I reduced the 
> speed to 100Mbps.
> 
>    The copy process is basically rsync+rsh, but I have also tried 
> rsync+ssh, scp and rcp.  If ssh does not detect an error in its stream 
> first, rsync does.
> 
>    Assuming it was a problem with the re(4) interface, being it a low 
> end adapter, I got back to the main network.  The new machine is on 
> (another) onboard nfe(4), and the old one is on a old xl(4) (3Com 905).  
> Just to be sure, I've also disabled TSO, RXCSUM and TXCSUM on these 
> interfaces.  Also, the copy process is now a raw rsyncd.
> 
>    I can now copy the files, but the big ones still get screwed.  
> (Verification by rsync -c and/or md5)
> 
>    After some days experimenting, I finally could make the copy by 
> split(1)ing them first and cat(1)ing ater network copy.
> 
>   This makes me wonder: Where is (are?) the problem(s)?
> 
>   It's could be rsync, but not only, since scp got screwed too, when 
> copying over gigabit.  But even when working over FastEthernet, there 
> were errors.  These could be at TCP layer and/or at rsync layer.
> 
>    I have found some reports with rsync failing for large files, but 
> they were dated from 2004, and fixed since then.

Looks like you didn't rule out nfe.

Kris