Seemingly odd disc i/o behaviour, need help to diagnose

[Oren Baum]

We're having a hard to track down a recurring problem that we think has some 
problem to do with disc i/o access.
This is slightly beyond our level of expertise and were hoping someone could 
shed some light on the following problem.
If this has been dealt with previously, please let me know under what topic so 
I can do relevant searches.

Description of Server configuration and problems.

Configuration:

FreeBSD 4.4-RELEASE
apache+mod_ssl-1.3.22+2.8.5_1
mysql-server-3.23.42
Hard Drive (dual ATA disks, no RAID) on a Dell P4 PowerEDGE server

We had many speed and timeout issues so we recompiled the kernel with 
maxusers=128 instead of the previous 32 and moved the hard drives into a new 
P4 2.8Ghz PowerEdge Case.

atapci0: <Generic PCI ATA controller> port 0xffa0-0xffaf,0x374-0x377,0x170-
0x177,0x3f4-0x3f7,0x1f0-0x1f7 irq 11 at device 31.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
atapci1: <Generic PCI ATA controller> port 0xfea0-0xfeaf,0xfe30-0xfe33,0xfe20-
0xfe27,0xfe10-0xfe13,0xfe00-0xfe07 mem 0xdff3fc00-0xdff3ffff irq 5 at device 
31.2 on pci0
ata2: at 0xfe00 on atapci1
ata3: at 0xfe20 on atapci1

This cleared up all errors found in fstat and the "file table is full" errors.

Server is configured to hold 2 medium sized MySQL DB's accessed through 
various perl and php scripts via websites on the server.

Uptime: 78463  Threads: 18  Questions: 351685  Slow queries: 328  Opens: 2793  
Flush tables: 1  Open tables: 64 Queries per second avg: 4.482

Each httpd process is of size 15MB to 20MB * about 15 processes
Mysqld process is of size 32MB, resident 16MB

Server load is now usually between .24 and .44

With the new kernel, we observed the server for about a week and are trying to 
pinpoint this issue:

If we run pine on a large mailbox or any other disk i/o intensive task, all 
other processes in motion seem to stall until the disk i/o is complete.
This manifests itself in timeouts on webpages that require DB data, IMAP 
timeouts for other mail accounts, and even odd console/shell behaviour.

For instance, at a console/ssh prompt we would run "uptime" or "pwd" and there 
would be up to a 10 second delay before results are returned. The results also 
show no heavy load (<.40) on the CPU.

Rebooting the server eliminates speed issues for up to a couple of hours.

Any thoughts on how to diagnose if this really is a disc I/O issue and how to 
resolve it would be most helpful!

Thank you,
__________________________________
Oren Baum
Creative Image Communications Inc.
oren at creativeimage.ca