Anybody seeing problems with large TCP streams over rsync?

[João Carlos Mendes Luís]

Hi folks,

    I am migrating a backup server from a server running FreeBSD 
6.2/i386 to another server running FreeBSD 7/amd64.  It is currently 
running 7.0-BETA4, as distributed from the CD image.

    Both machines have SATA discs and some mix of gmirror and gstripe of 
its partitions.

    Since it is a Backup server, I am in the process of transferring 
large files from one to another.  But I always have trouble when 
transferring large files (75GB).

    The first trouble I had was in the back-to-back Gigabit Ethernet 
connection, to speed-up transfers.  Somehow, data got quickly screwed, 
even above TCP layer!

    One side is an onboard nVidia nfe(4), and the other a generic 
Realtek re(4).  The troubles went on, even after I disable TSO (on fbsd 
7), RXCSUM and TXCSUM on both sides.  They went on even after I reduced 
the speed to 100Mbps.

    The copy process is basically rsync+rsh, but I have also tried 
rsync+ssh, scp and rcp.  If ssh does not detect an error in its stream 
first, rsync does.

    Assuming it was a problem with the re(4) interface, being it a low 
end adapter, I got back to the main network.  The new machine is on 
(another) onboard nfe(4), and the old one is on a old xl(4) (3Com 905).  
Just to be sure, I've also disabled TSO, RXCSUM and TXCSUM on these 
interfaces.  Also, the copy process is now a raw rsyncd.

    I can now copy the files, but the big ones still get screwed.  
(Verification by rsync -c and/or md5)

    After some days experimenting, I finally could make the copy by 
split(1)ing them first and cat(1)ing ater network copy.

   This makes me wonder: Where is (are?) the problem(s)?

   It's could be rsync, but not only, since scp got screwed too, when 
copying over gigabit.  But even when working over FastEthernet, there 
were errors.  These could be at TCP layer and/or at rsync layer.

    I have found some reports with rsync failing for large files, but 
they were dated from 2004, and fixed since then.

   Can anyone help me?

     Jonny

PS: Even without old files, the new server is up, running and making new 
daily backups, without problems.

-- 
João Carlos Mendes Luís - Networking Engineer - jonny at jonny.eng.br