Odd lag/hanging issue with production ftp server - Please help AS AP!

[Brad Davis]

Elliott,

You might try downloading a file from the internet to locate where the
problem is. Or a different FTP server. Etc. Keep trying to narrow down
where the problem is.


Regards,
Brad

On Sat, Jan 17, 2004 at 02:24:51PM -0800, Elliott Freis wrote:
> I am forwarding this to this list per recommendation.
> 
> 
> Note on recent test:  Trying to do a transfer on the public LAN off the ftp
> server, I am only able to get a bursty 20k/s! down from the ftp server.
> This was with nfs completely unmounted, ftping from a local drive so its not
> an NFS problem.  It should get a full 2-3mb a sec at least.  It used to.  Up
> is still full speed.  The machine has a Intel pro NIC, direct to a Cisco
> ArrowPoint.  Both are set to 100 FD.  ifconfig_fxp2="inet 66.151.XXX.XXX
> netmask 255.255.255.224 media 100baseTX"
> 
> Any help GREATLY appreciated, I have exhausted all avenues I can think of,
> including hardware swaps.  Here is some diag:
> 
> last pid: 49903;  load averages:  0.12,  0.20,  0.25
> up 0+15:19:19  14:19:00
> 73 processes:  1 running, 72 sleeping
> CPU states:  1.4% user,  0.0% nice, 17.1% system,  2.3% interrupt, 79.2%
> idle
> Mem: 24M Active, 1233M Inact, 178M Wired, 68M Cache, 163M Buf, 3600K Free
> Swap: 1024M Total, 12K Used, 1024M Free
> 
> 353/17120/262144 mbufs in use (current/peak/max):
>         271 mbufs allocated to data
>         82 mbufs allocated to packet headers
> 213/16806/65536 mbuf clusters in use (current/peak/max)
> 37892 Kbytes allocated to network (19% of mb_map in use)
> 0 requests for memory denied
> 0 requests for memory delayed
> 0 calls to protocol drain routines
> 
> Name  Mtu   Network       Address            Ipkts Ierrs    Opkts Oerrs
> Coll
> fxp2  1500  <Link#3>    00:02:a5:13:fc:b5 30214691     0 22672636     0
> 589096
> fxp2  1500  66.151.XXX.XXX	ftp             30150589     - 22683851
> -     -
> fxp2  1500  fe80:3::202 fe80:3::202:a5ff:        0     -        0     -
> -
> 
> >  -----Original Message-----
> > From: 	Elliott Freis  
> > Sent:	Friday, January 09, 2004 3:28 PM
> > To:	'freebsd-questions at FreeBSD.ORG'
> > Subject:	Odd lag/hanging issue with production ftp server - Please
> > help ASAP!
> > 
> >   I have been struggling with this one for over a month now.  Here is a
> > quick layout of my setup:
> > 
> > Primary FTP server:
> > Compaq DL380 1.5gb ram
> > FreeBSD 4.5  
> > 3x36gb RAID 5 drives as local boot/storage
> > ProFTPd
> > 
> > Array server for FTP:
> > AMD Athlon 2200 512mb ram
> > FreeBSD 4.8
> > 7x36gb Fiber channel drives, RAID 5 via Vinum.
> > 
> > Both machines are connected via a cross-over cable, that has been tested
> > good and swapped just in case.  The primary storage for FTP is done on the
> > Fiber drives via nfs from FTP to Array server.  NFS options are -U -3.
> > 
> > My problem is this.  As more users connect and store files, the primary
> > FTP machine becomes increasingly unresponsive.  Currently, I max at about
> > 350 concurrent FTP connections.  The most basic test I have been doing is
> > just holding down enter on an SSH session.  As you hold enter down, you
> > see it visually just hang for a second or more (up to about 5 seconds
> > depending on the load).  It is even worse if I spam "df -k" for example.
> > For part of the time, it responds fine, though its randomly a second to
> > multiple seconds.  In other words, it is randomly responsive and not
> > responsive every 5 seconds or so.  During the "hanging" time, ftp sessions
> > are also hung.  So you see very bursty data transfers.  Now thankfully, no
> > ftp sessions drop, so we do get the data we need.  But this is a terrible
> > thing to be happening to a production server.
> > 
> >  One other thing of note, this happened to me about 4-5 months ago, but a
> > reboot fixed it for some reason.  So I concluded it was just a hiccup.
> > But it has returned after a different reboot, and won't go away.  
> > 
> >   Any help in troubleshooting this is very appreciated!  Happy new year,
> > 
> >  -Elliott
> > 
> > Example of "enter" latency (this is from a LAN connection):
> > At shell prompt ">"
> > >
> > >
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > >
> > >
> > >
> > >
> > >
> > 
> > >
> > 
> > >
> > >
> > >
> > 
> > >
> > >
> > >
> > 
> > >
> > >
> > 
> > 
> > 
> > >
> > >
> > >
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"