Performance Problems.. Server hardware smoked by $500 box?

[John Straiton]

> Hrmmm....   I wonder if 5.0-R is faster than 4-STABLE?

The question is really, is it faster than 4.8-S AND 5.1-C. That was the
reason we updated the production machine to 5.1 from 4.8... To determine
if that was a factor at all. I was hoping that 5.X was just *that much
faster* but it would appear that it's not the case unless 5.0-R is *that
much faster* than 5.1-C. 

> 
> Well.  If anything, production should be kicking 
> development's heiney in this one.
You'd think so, at least on the reads. I understand that RAID5 isn't the
fastest thing on the block writing, but 99.5% of the traffic to the
machine should be requesting pages, not doing disk/database writes other
than standard apache logging.

> > > - memory speed (PC100 vs. PC133, DDR200 vs. DDR400...)
> > Production  512MB PC133 ECC Registered  vs Development  
> 256MB 266Mhz 
> > DDR
> Is the DDR registered at 266MHz?  It could be that it's 
> 532MHz...  I don't remember which way the DDR speed rating 
> goes, I've only just set up my first DDR server yesterday 
> (didn't trust it just yet).  I /think/ it's 266MHz in both 
> directions, but you'd probably know better than I.
I really have no clue. I'm not a hardware guy when it comes to stuff I
haven't seen in a server, and I haven't seen anything but various speeds
of ECC in the machines I tinker with.


> Any network errors on either cards?  Have you tried doing 
> some netperf stuff, to see how the machines handle data transfers?
Yeah, netperf scores were in the original email. The development machine
topped around 96Mbit/sec and the production one topped like at
87Mbit/sec I think it was. The more interesting numbers were how the
development machine nearly *doubled* the production one in NFS scores.


> > The servers use Intel integrated cards, vs we have a $7 no-name 
> > realtek card in the development machine.
I made a mistake. The development machine indeed has a $7 realtek in it,
but the configured interface is actually a 3c905TX 3Com card.

Zero Ierrs, Zero Oerrs, Zero Collisions confirmed 100/full on
development
Zero Ierrs, Zero Oerrs, Zero Collisions confirmed 100/full on production
Of course, the DB server serves files from local but it also has
Zero Ierrs, Zero Oerrs, Zero Collisions confirmed 100/full 

I too have had a periodic problem with auto negotiation on Cisco gear. I
wish it was something simple like that here but it'd appear that we're
all synced up just fine.


> Well, the differences you've pointed out are the RAM and CPU 
> speed differences.  What about other things -- CPU cache 
> size?  What about looking up benchmark differences for the 
> AMD vs. Intel, and see what they show (you'll probably only 
> find similar ratings, but if the AMD beats the Intel at 
> comparitive speeds (1.5GHz), you can bet it'll beat the Intel 
> at a lower speed.  Just make sure you're looking at IO 
> benchmarks, and not gaming benchmarks.)
Well I can't say I know offhand what cache size the Dell PowerEdges use
but I'd imagine it's at least equivalent to the AMD chip. I'd say I'd
look down that route if it weren't for the next question..

> What about FSB speed -- what does the AMD motherboard run at? 
>  The Intel motherboard?
Now I think I just might buy (and agree) that all the difference in the
world would be there if I have faster ram, faster cpu AND a faster bus
speed. I don't know how to determine that from the DMESG tho' (it didn't
seem blantantly obvious to me) so I guess I'd have to try to dig through
dell's new site (the site formerly known as the best hardware vendor
site when you could put in a ID # and it'd tell you everything exactly
for *your* machine instead of the family of machines) to find out what
it's running. I could find the MB manual for the development one to look
that up if necessary.

Would anyone like to concurr that the FSB (et al) could be enough of a
difference to explain all this? If so, I'm upgrading my 100Mhz FSB box
at home like tomorrow... This development machine just screams in
comparison. 

I've attached the DMESGs from all 3 machines below:

John Straiton
jks@ clickcom.com
Clickcom, Inc
704-365-9970x101 

>> Production Web
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights
reserved.
FreeBSD 5.1-CURRENT #0: Mon Sep  8 12:35:03 EDT 2003
    root@:/usr/obj/usr/src/sys/GENERIC
Preloaded elf kernel "/boot/kernel/kernel" at 0xc072b000.
Preloaded elf module "/boot/kernel/acpi.ko" at 0xc072b1f4.
Timecounter "i8254"  frequency 1193182 Hz
Timecounter "TSC"  frequency 927108262 Hz
CPU: Intel Pentium III (927.11-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x68a  Stepping = 10
 
Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,
MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE
>
real memory  = 1342111744 (1279 MB)
avail memory = 1297186816 (1237 MB)
Pentium Pro MTRR support enabled
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <DELL   PA1550  > on motherboard
acpi0: power button is handled as a fixed feature programming model.
Timecounter "ACPI-safe"  frequency 3579545 Hz
pcibios: BIOS version 2.10
Using $PIR table, 6 entries at 0xc00fc730
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_cpu1: <CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib0: slot 1 INTA is routed to irq 11
pcib0: slot 2 INTA is routed to irq 10
fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
0xecc0-0xecff mem 0xfe100000-0xfe1fffff,0xfe2f
f000-0xfe2fffff irq 11 at device 1.0 on pci0
fxp0: Ethernet address 00:06:5b:3a:48:d5
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
0xec80-0xecbf mem 0xfe000000-0xfe0fffff,0xfe2f
e000-0xfe2fefff irq 10 at device 2.0 on pci0
fxp1: Ethernet address 00:06:5b:3a:48:d6
miibus1: <MII bus> on fxp1
inphy1: <i82555 10/100 media interface> on miibus1
inphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pci0: <display, VGA> at device 3.0 (no driver attached)
isab0: <PCI-ISA bridge> port 0x580-0x58f at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks ROSB4 UDMA33 controller> port 0x8b0-0x8bf at
device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pcib1: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI Host-PCI bridge> on acpi0
pci2: <ACPI PCI bus> on pcib2
pcib2: slot 5 INTA is routed to irq 5
pcib2: slot 5 INTB is routed to irq 3
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0xdc00-0xdcff mem
0xfeaff000-0xfeafffff irq 5 at device 5
.0 on pci2
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0xd800-0xd8ff mem
0xfeafe000-0xfeafefff irq 3 at device 5
.1 on pci2
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port
0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
orm0: <Option ROMs> at iomem 0xc8000-0xcdfff,0xc0000-0xc7fff on isa0
pmtimer0 on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
Timecounters tick every 10.000 msec
acd0: CDROM <SAMSUNG CD-ROM SN-124> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
ses0 at ahc0 bus 0 target 6 lun 0
ses0: <DELL 1x3 U2W SCSI BP 1.21> Fixed Processor SCSI-2 device 
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
da0 at ahc0 bus 0 target 0 lun 0
da0: <SEAGATE ST318305LC 2203> Fixed Direct Access SCSI-3 device 
da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged
Queueing Enabled
da0: 17366MB (35566478 512 byte sectors: 255H 63S/T 2213C)
Mounting root from ufs:/dev/da0s1a

>> Development web
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights
reserved.
FreeBSD 5.0-RELEASE #0: Thu Jan 16 22:16:53 GMT 2003
    root at hollin.btc.adaptec.com:/usr/obj/usr/src/sys/GENERIC
Preloaded elf kernel "/boot/kernel/kernel" at 0xc0673000.
Preloaded elf module "/boot/kernel/acpi.ko" at 0xc06730a8.
Timecounter "i8254"  frequency 1193182 Hz
Timecounter "TSC"  frequency 1544669680 Hz
CPU: AMD Athlon(tm) XP 1800+ (1544.67-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x681  Stepping = 1
 
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,
MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
  AMD Features=0xffffffffc0400000<AMIE,DSP,3DNow!>
real memory  = 1040121856 (991 MB)
avail memory = 1003769856 (957 MB)
Initializing GEOMetry subsystem
Pentium Pro MTRR support enabled
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <VT8375 MSI ACPI> on motherboard
    ACPI-0625: *** Info: GPE Block0 defined as GPE0 to GPE15
Using $PIR table, 6 entries at 0xc00fda00
acpi0: power button is handled as a fixed feature programming model.
Timecounter "ACPI-fast"  frequency 3579545 Hz
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_tz0: <thermal zone> on acpi0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port
0x5000-0x500f,0x4080-0x40ff,0x4000-0x407f,0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA Generic host to PCI bridge> mem 0xd0000000-0xd7ffffff at
device 0.0 on pci0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
xl0: <3Com 3c905-TX Fast Etherlink XL> port 0xa000-0xa03f irq 11 at
device 5.0 on pci0
xl0: Ethernet address: 00:10:5a:02:ee:3f
miibus0: <MII bus> on xl0
nsphy0: <DP83840 10/100 media interface> on miibus0
nsphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: <RealTek 8139 10/100BaseTX> port 0xa400-0xa4ff mem
0xe3000000-0xe30000ff irq 12 at device 7.0 on pci0
rl0: Realtek 8139B detected. Warning, this may be unstable in autoselect
mode
rl0: Ethernet address: 00:50:fc:3d:f4:c0
miibus1: <MII bus> on rl0
rlphy0: <RealTek internal media interface> on miibus1
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 8233 ATA133 controller> port 0xa800-0xa80f at device 17.1
on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
uhci0: <VIA 83C572 USB controller> port 0xac00-0xac1f irq 5 at device
17.2 on pci0
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xb000-0xb01f irq 5 at device
17.3 on pci0
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
pci0: <multimedia, audio> at device 17.5 (no driver attached)
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port
0x3f7,0x3f2-0x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
sio1 port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0 port 0x778-0x77b,0x378-0x37f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/16 bytes threshold
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
orm0: <Option ROM> at iomem 0xc0000-0xcbfff on isa0
pmtimer0 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x64,0x60 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
Timecounters tick every 10.000 msec
acpi_cpu: CPU throttling enabled, 2 steps from 100% to 50.0%
ad0: 39205MB <Maxtor 6E040L0> [79656/16/63] at ata0-master UDMA133
Mounting root from ufs:/dev/ad0s1a

>> Production DB
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights
reserved.
FreeBSD 4.8-STABLE #0: Tue Sep  2 16:14:44 EDT 2003
 
dlostboy at depot.clickcomworks.net:/usr/obj/usr/src/sys/SiteWizardServers
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel(R) Pentium(R) III CPU family      1266MHz (1263.45-MHz
686-class CPU)
  Origin = "GenuineIntel"  Id = 0x6b1  Stepping = 1
 
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,
MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 805240832 (786368K bytes)
avail memory = 778670080 (760420K bytes)
Changing APIC ID for IO APIC #0 from 0 to 1 on chip
Changing APIC ID for IO APIC #1 from 0 to 2 on chip
Programming 16 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
Programming 16 pins in IOAPIC #1
FreeBSD/SMP: Multiprocessor motherboard
 cpu0 (BSP): apic id:  0, version: 0x00040011, at 0xfee00000
 io0 (APIC): apic id:  1, version: 0x000f0011, at 0xfec00000
 io1 (APIC): apic id:  2, version: 0x000f0011, at 0xfec01000
Preloaded elf kernel "kernel" at 0xc0437000.
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 8 entries at 0xc00fc250
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <ServerWorks host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pci0: <ATI Mach64-GR graphics accelerator> at 14.0
isab0: <ServerWorks IB6566 PCI to ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks ROSB4 ATA33 controller> port 0x8b0-0x8bf at device
15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pcib2: <ServerWorks host to PCI bridge> on motherboard
IOAPIC #1 intpin 15 -> irq 2
IOAPIC #1 intpin 0 -> irq 5
pci2: <PCI bus> on pcib2
pcib4: <PCI to PCI bridge (vendor=8086 device=0962)> at device 2.0 on
pci2
IOAPIC #1 intpin 14 -> irq 10
pci3: <PCI bus> on pcib4
pci3: <unknown card> (vendor=0x9005, dev=0x00c5) at 4.0 irq 2
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0xc800-0xc8ff mem
0xfe7fe000-0xfe7fefff irq 10 at device 4.1 on pci3
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
aac0: <Dell PERC 3/Di> mem 0xf0000000-0xf7ffffff irq 2 at device 2.1 on
pci2
aac0: i960RX 100MHz, 118MB cache memory, optional battery present
aac0: Kernel 2.5-0, Build 2991, S/N 3c21d2
aac0: Supported Options=0
fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port
0xbcc0-0xbcff mem 0xfe500000-0xfe5fffff,0xfe900000-0xfe900fff irq 5 at
device 4.0 on pci2
fxp0: Ethernet address 00:06:5b:3e:54:88
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pcib1: <ServerWorks host to PCI bridge> on motherboard
IOAPIC #1 intpin 1 -> irq 11
pci1: <PCI bus> on pcib1
bge0: <Broadcom BCM5700 Gigabit Ethernet, ASIC rev. 0x7102> mem
0xfeb00000-0xfeb0ffff irq 11 at device 8.0 on pci1
bge0: Ethernet address: 00:06:5b:3e:54:89
miibus1: <MII bus> on bge0
brgphy0: <BCM5401 10/100/1000baseTX PHY> on miibus1
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX,
1000baseTX-FDX, auto
pcib3: <ServerWorks host to PCI bridge> on motherboard
pci4: <PCI bus> on pcib3
orm0: <Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xcbfff,0xec000-0xeffff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0
intpin 2
APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0
IP Filter: v3.4.31 initialized.  Default = pass all, Logging = enabled
ata0-slave: ATAPI identify retries exceeded
acd0: CDROM <TEAC CD-ROM CD-224E> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
aacd0: <RAID 5> on aac0
aacd0: 69425MB (142182912 sectors)
Mounting root from ufs:/dev/aacd0s1a