Terrible (3Ware) disk performance in 6.2-BETA3

[Philip Murray]

Hi,

I just installed FreeBSD 6.2-BETA3 (amd64) on a Dual Xeon DC 5140 w/  
4GB of RAM and a 3ware 9550SX connected to 8x Seagate 250GB SATA  
drives (RAID5) and instead of the normal great performance I get from  
FreeBSD, it's taking over an hour to do a 'portsnap extract' and  
while it's doing it, the system gets very unresponsive (trouble  
logging in via SSH, commands sit for several seconds before executing).

On two other machines with the same 3Ware 9550SX (and disks) but dual  
Opteron 248s, the portsnap extract operation only takes a short  
(relatively) amount of time to complete.

The RAID array is fine, it's not rebuilding or degraded of anything.  
I've tried turning the write cache on the controller on/off.

Where do I start looking to find the culprit?

gstat reports the following during the portsnap extract:

  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
   416     70      0      0    0.0     70    160 12149.0  101.7| da0


Cheers

Phil



dmesg:

Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights  
reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-BETA3 #0: Mon Oct 30 22:03:42 UTC 2006
     root at meyers.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP
ACPI APIC Table: <PTLTD          APIC  >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU            5140  @ 2.33GHz (2333.43-MHz K8- 
class CPU)
   Origin = "GenuineIntel"  Id = 0x6f6  Stepping = 6
    
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE 
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
    
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15 
 >,<b18>>
   AMD Features=0x20000800<SYSCALL,LM>
   AMD Features2=0x1<LAHF>
   Cores per package: 2
real memory  = 5368709120 (5120 MB)
avail memory = 4114276352 (3923 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
cpu2 (AP): APIC ID:  6
cpu3 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413,  
RF5413)
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci4: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port  
0x2000-0x201f mem 0xc8200000-0xc821ffff irq 18 at device 0.0 on pci4
em0: Ethernet address: 00:30:48:88:a5:06
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port  
0x2020-0x203f mem 0xc8220000-0xc823ffff irq 19 at device 0.1 on pci4
em1: Ethernet address: 00:30:48:88:a5:07
pcib5: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci5: <ACPI PCI bus> on pcib5
3ware device driver for 9000 series storage controllers, version:  
3.60.02.012
twa0: <3ware 9000 series Storage Controller> port 0x3000-0x303f mem  
0xca000000-0xcbffffff,0xc8300000-0xc8300fff irq 24 at device 1.0 on pci5
twa0: [FAST]
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-8LP, 8  
ports, Firmware FE9X 3.01.01.028, BIOS BE9X 3.01.00.024
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci6: <ACPI PCI bus> on pcib6
uhci0: <UHCI (generic) USB controller> port 0x1800-0x181f irq 17 at  
device 29.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0x1820-0x183f irq 19 at  
device 29.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0x1840-0x185f irq 18 at  
device 29.2 on pci0
uhci2: [GIANT-LOCKED]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhci3: <UHCI (generic) USB controller> port 0x1860-0x187f irq 16 at  
device 29.3 on pci0
uhci3: [GIANT-LOCKED]
usb3: <UHCI (generic) USB controller> on uhci3
usb3: USB revision 1.0
uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xc8000000-0xc80003ff  
irq 17 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <EHCI (generic) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pci7: <display, VGA> at device 1.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port  
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1880-0x188f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10  
on acpi0
sio0: type 16550A
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on  
acpi0
fdc0: [FAST]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcc7ff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on  
isa0
Timecounters tick every 1.000 msec
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9550SX-8LP DISK 3.01> Fixed Direct Access SCSI-3 device
da0: 100.000MB/s transfers
da0: 1668856MB (3417817088 512 byte sectors: 255H 63S/T 212749C)
Trying to mount root from ufs:/dev/da0s1a