Odd lag/hanging issue with production ftp server - Please help AS AP!

[Elliott Freis]

>   I have been struggling with this one for over a month now.  Here is a
> quick layout of my setup:
> 
> Primary FTP server:
> Compaq DL380 1gb ram
> FreeBSD 4.5
> 3x36gb RAID 5 drives as local boot/storage
> ProFTPd
> 
> Array server for FTP:
> AMD Athlon 2200 512mb ram
> FreeBSD 4.8
> 7x36gb Fiber channel drives, RAID 5 via Vinum.
> 
> Both machines are connected via a cross-over cable, that has been tested
> good and swapped just in case.  The primary storage for FTP is done on the
> Fiber drives via nfs from FTP to Array server.  NFS options are -U -3.
> 
> My problem is this.  As more users connect and store files, the primary
> FTP machine becomes increasingly unresponsive.  Currently, I max at about
> 350 concurrent FTP connections.  The most basic test I have been doing is
> just holding down enter on an SSH session.  As you hold enter down, you
> see it visually just hang for a second or more (up to about 5 seconds
> depending on the load).  It is even worse if I spam "df -k" for example.
> For part of the time, it responds fine, though its randomly a second to
> multiple seconds.  In other words, it is randomly responsive and not
> responsive every 5 seconds or so.  During the "hanging" time, ftp sessions
> are also hung.  So you see very bursty data transfers.  Now thankfully, no
> ftp sessions drop, so we do get the data we need.  But this is a terrible
> thing to be happening to a production server.
> 
>  One other thing of note, this happened to me about 4-5 months ago, but a
> reboot eventually fixed it for some reason.  So I concluded it was just a
> hiccup.  But it has returned after a different reboot, and won't go away.
> Also, if I move the ftp logins to the local drives (which I can't do
> permanently due to lack of space) there is no problem at all, no lag.
> 
>   Any help in troubleshooting this is very appreciated!  Happy new year,
> 
>  -Elliott
> 
> Example of "enter" latency (this is from a LAN connection):
> At shell prompt ">"
> >
> >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> >
> >
> >
> >
> >
> 
> >
> 
> >
> >
> >
> 
> >
> >
> >
> 
> >
> >
> 
> 
> 
> >
> >
> >