Performance Problems.. Server hardware smoked by $500 box?

[John Straiton]

Greets!
	
I'm pretty confused right now with trying to determine the nature of a
performance problem I'm having on one of my servers. The server is a
webserver with a separate db/file server sitting behind it. The issue is
that in pulling up websites from the machine, my silly POS development
box has nearly double performance although one would think it shouldn't.


Serving pages from the production machine over NFS or even the db/file
server via local filesystem is slower than pages from the development
machine over NFS. By that I mean that if you view the page on the
development machine over the LAN, it appears instantly...Do that on the
production machines and 1-2 seconds pass as you watch images flow in...
With zero load. Now granted the development machine has the most Mhz,
but we're talking *visible* differences in speed between a 1.5GhzAMD &
~1Ghz Intel Server Class machines. And to top it all off, with the
miniscule amount of ram in the AMD box, it's paging around 50MB (with
only a few K free of physical ram) by the time apache finshes spawning
during the boot. By all rights, to me, this thing should be no better
than equivalent to the production machines.

They are all plugged in to the same Cisco 3524 switch, and I've tried
mulitple cables (even swapping between the fast & slow machines to try
to hurt the fast one) and the IOS reports no errors on the interfaces,
all at 100/FullDuplex. I can reproduce the problem on any # of viewing
machines at will.

Please tell me that I shouldn't toss these multi-thousand dollar servers
and just buy a bunch of these silly do-it-yourself'ers..(*kidding*, but
you get the idea)


The hardware involved:

Production Machine: (Dell PowerEdge 1U server 1G/UWSCSI)
========================================
FreeBSD 5.1-Current  (Was 4.8 Release, upgraded in attempt to find
problem)
CPU: Intel Pentium III (927.11-MHz 686-class CPU) Origin =
"GenuineIntel"  
real memory  = 1342111744 (1279 MB)
fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> 
da0: <SEAGATE ST318305LC 2203> Fixed Direct Access SCSI-3 device 
da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit)
da0: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C)

DB/File Server: (Dell PowerEdge 2U Server 768MB/RAID5)
=======================================
FreeBSD 4.8 Stable
CPU: Intel(R) Pentium(R) III CPU family      1266MHz (1263.45-MHz
686-class CPU)
real memory  = 805240832 (786368K bytes)
fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet>
aac0: <Dell PERC 3/Di> mem 0xf0000000-0xf7ffffff irq 2 at device 2.1 on
pci2
aac0: i960RX 100MHz, 118MB cache memory, optional battery present
aacd0: <RAID 5> on aac0
aacd0: 69425MB (142182912 sectors)

Development Machine: (Whitebox machine)
========================================
FreeBSD 5.0-Release
CPU: AMD Athlon(tm) XP 1800+ (1544.67-MHz 686-class CPU)
real memory  = 234815488 (223 MB)
rl0: <RealTek 8139 10/100BaseTX>
ad0: 39205MB <Maxtor 6E040L0> [79656/16/63] at ata0-master UDMA133

Ideas? Could the CPU alone be doing that much towards the difference?
RAM (DDRvsECC)? I don't have any other AMD machines to use in a control
group so I can't persue that route. 

I have benchmarks attached below...

Thanks for any ideas,
John Straiton
jks@ clickcom.com
Clickcom, Inc
704-365-9970x101 




The benchmarks that show the most difference are these...my notes in
()'s:
========================================================================
====
#bonnie -s 1    (run on NFS share)
              -------Sequential Output-------- ---Sequential Input--
--Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
PROD    1  6364 25.0  9121  4.9   166  0.3  2906 13.5 94108 28.6 398.8
3.3
              -------Sequential Output-------- ---Sequential Input--
--Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
DEV     1 11246 65.3 11259  6.4  6733  4.3 11009 41.9 292571 95.3 5612.2
28.9


#ubench
PROD: (ECC SDRAM)
Ubench CPU:    25713
Ubench MEM:    22707
--------------------
Ubench AVG:    24210

DEV: (DDR)
Ubench CPU:    66048
Ubench MEM:    57434
--------------------
Ubench AVG:    61741


#netperf (TO DB MACHINE FROM NOTED MACHINE)
TCP STREAM TEST to DB FROM DEV
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 57344  32768  32768    10.01      92.74   

TCP STREAM TEST to DB FROM PROD
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 57344  32768  32768    10.00      87.61