frustratingly slow box at 4GB, but not 1GB of memory

[Adrian Filipi]

Hi folks,

	I've got a box I'm helping tune and it's not really responding as
I'd expect.

	It's a dual 2GHz Xeon box running on a Supermicro P4DLR motherboard
with 4GB of physical RAM.  It has 4GB of swap configured on a hardware
RAID5 configuration.

	It's running 4.9-RELEASE.  It's also running Apache with PHP and
Perl CGI's.  The CGI's read/write to a MySQL DB.

	Briefly, it responds pretty quickly when I only let the kernel use
1GB of the RAM, but gets bogged down when I let it use the full 4GB.  I'm
limiting the RAM by setting hw.physmem.

	I've tried some of the things that I could dig up in the archives,
but the knowledge is pretty widely scattered.  :-(

	I have MAXUSERS set to 96 as that gets me plenty of file
descriptors and pid's.  While tight, I believe I could get away with only
32, as there should not be more than 300-400 processes running at any one
time.

	I set vm.pmap.shpgperproc="300" because of warnings from the
kernel.  I'm not sure if this is the best value, but it made the warnings
go away.

	First, upping the KVA_PAGES to 768 to achieve a 3:1 ration of
kernel to user memory, didn't change things.

	Disabling swap seems to make a huge difference.  It changes the
time results for "time top -bu" by two orders of magnitude from 0.006s to
0.6s!  In fact, I can run the box reasonably well with 2GB of RAM with swap
disabled.

	If I let the box boot using all of the RAM, it is very slow.  e.g.
"time top -bu >/dev/null" takes about 6 seconds.

	I am seeing large runs of time (15-30 seconds) where the system
time usage is in the high range (75-99).  Does anyone have a suggestion on
how to determine which kernel data structures are the problems?

thanks,

	Adrian
--
[ adrian at ubergeeks.com ]