Odd lag/hanging issue with production ftp server - Please help AS AP!

[Elliott Freis]

I am forwarding this to this list per recommendation.


Note on recent test:  Trying to do a transfer on the public LAN off the ftp
server, I am only able to get a bursty 20k/s! down from the ftp server.
This was with nfs completely unmounted, ftping from a local drive so its not
an NFS problem.  It should get a full 2-3mb a sec at least.  It used to.  Up
is still full speed.  The machine has a Intel pro NIC, direct to a Cisco
ArrowPoint.  Both are set to 100 FD.  ifconfig_fxp2="inet 66.151.XXX.XXX
netmask 255.255.255.224 media 100baseTX"

Any help GREATLY appreciated, I have exhausted all avenues I can think of,
including hardware swaps.  Here is some diag:

last pid: 49903;  load averages:  0.12,  0.20,  0.25
up 0+15:19:19  14:19:00
73 processes:  1 running, 72 sleeping
CPU states:  1.4% user,  0.0% nice, 17.1% system,  2.3% interrupt, 79.2%
idle
Mem: 24M Active, 1233M Inact, 178M Wired, 68M Cache, 163M Buf, 3600K Free
Swap: 1024M Total, 12K Used, 1024M Free

353/17120/262144 mbufs in use (current/peak/max):
        271 mbufs allocated to data
        82 mbufs allocated to packet headers
213/16806/65536 mbuf clusters in use (current/peak/max)
37892 Kbytes allocated to network (19% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

Name  Mtu   Network       Address            Ipkts Ierrs    Opkts Oerrs
Coll
fxp2  1500  <Link#3>    00:02:a5:13:fc:b5 30214691     0 22672636     0
589096
fxp2  1500  66.151.XXX.XXX	ftp             30150589     - 22683851
-     -
fxp2  1500  fe80:3::202 fe80:3::202:a5ff:        0     -        0     -
-

>  -----Original Message-----
> From: 	Elliott Freis  
> Sent:	Friday, January 09, 2004 3:28 PM
> To:	'freebsd-questions at FreeBSD.ORG'
> Subject:	Odd lag/hanging issue with production ftp server - Please
> help ASAP!
> 
>   I have been struggling with this one for over a month now.  Here is a
> quick layout of my setup:
> 
> Primary FTP server:
> Compaq DL380 1.5gb ram
> FreeBSD 4.5  
> 3x36gb RAID 5 drives as local boot/storage
> ProFTPd
> 
> Array server for FTP:
> AMD Athlon 2200 512mb ram
> FreeBSD 4.8
> 7x36gb Fiber channel drives, RAID 5 via Vinum.
> 
> Both machines are connected via a cross-over cable, that has been tested
> good and swapped just in case.  The primary storage for FTP is done on the
> Fiber drives via nfs from FTP to Array server.  NFS options are -U -3.
> 
> My problem is this.  As more users connect and store files, the primary
> FTP machine becomes increasingly unresponsive.  Currently, I max at about
> 350 concurrent FTP connections.  The most basic test I have been doing is
> just holding down enter on an SSH session.  As you hold enter down, you
> see it visually just hang for a second or more (up to about 5 seconds
> depending on the load).  It is even worse if I spam "df -k" for example.
> For part of the time, it responds fine, though its randomly a second to
> multiple seconds.  In other words, it is randomly responsive and not
> responsive every 5 seconds or so.  During the "hanging" time, ftp sessions
> are also hung.  So you see very bursty data transfers.  Now thankfully, no
> ftp sessions drop, so we do get the data we need.  But this is a terrible
> thing to be happening to a production server.
> 
>  One other thing of note, this happened to me about 4-5 months ago, but a
> reboot fixed it for some reason.  So I concluded it was just a hiccup.
> But it has returned after a different reboot, and won't go away.  
> 
>   Any help in troubleshooting this is very appreciated!  Happy new year,
> 
>  -Elliott
> 
> Example of "enter" latency (this is from a LAN connection):
> At shell prompt ">"
> >
> >
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> >
> >
> >
> >
> >
> 
> >
> 
> >
> >
> >
> 
> >
> >
> >
> 
> >
> >
> 
> 
> 
> >
> >
> >