kern/86411: Very low performance of amr(4) under FreeBSD-6.0-BETA5

[Benjamin Lutz]

>Number:         86411
>Category:       kern
>Synopsis:       Very low performance of amr(4) under FreeBSD-6.0-BETA5
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Sep 21 14:00:23 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator:     Benjamin Lutz
>Release:        FreeBSD 6.0-BETA5 i386
>Organization:
>Environment:
System: FreeBSD gaia.intranet 6.0-BETA5 FreeBSD 6.0-BETA5 #0: Tue Sep 20 03:26:52 CEST 2005 maxlor at gaia.intranet:/usr/obj/usr/src/sys/GAIA i386

This is a GENERIC kernel with us.dvorak added as standard keyboard
layout.

----- dmesg start -----
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 6.0-BETA5 #0: Tue Sep 20 03:26:52 CEST 2005
    maxlor at gaia.intranet:/usr/obj/usr/src/sys/GAIA
ACPI APIC Table: <DELL   PE600SC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2399.34-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x400<CNTX-ID>
real memory  = 268369920 (255 MB)
avail memory = 253079552 (241 MB)
ioapic0: Changing APIC ID to 1
ioapic1: Changing APIC ID to 2
ioapic2: Changing APIC ID to 3
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic2 <Version 1.1> irqs 32-47 on motherboard
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <DELL PE600SC> on motherboard
acpi0: Power Button (fixed)
pci_link0: <ACPI PCI Link LNK0> on acpi0
pci_link1: <ACPI PCI Link LNK1> irq 10 on acpi0
pci_link2: <ACPI PCI Link LNK2> on acpi0
pci_link3: <ACPI PCI Link LNK3> on acpi0
pci_link4: <ACPI PCI Link LNK4> irq 5 on acpi0
pci_link5: <ACPI PCI Link LNK5> on acpi0
pci_link6: <ACPI PCI Link LNK6> on acpi0
pci_link7: <ACPI PCI Link LNK7> on acpi0
pci_link8: <ACPI PCI Link LNK8> on acpi0
pci_link9: <ACPI PCI Link LNK9> on acpi0
pci_link10: <ACPI PCI Link LNKA> on acpi0
pci_link11: <ACPI PCI Link LNKB> on acpi0
pci_link12: <ACPI PCI Link LNKC> on acpi0
pci_link13: <ACPI PCI Link LNKD> on acpi0
pci_link14: <ACPI PCI Link LNKE> on acpi0
pci_link15: <ACPI PCI Link LNKF> on acpi0
pci_link16: <ACPI PCI Link LN10> on acpi0
pci_link17: <ACPI PCI Link LN11> on acpi0
pci_link18: <ACPI PCI Link LN12> on acpi0
pci_link19: <ACPI PCI Link LN13> on acpi0
pci_link20: <ACPI PCI Link LN14> on acpi0
pci_link21: <ACPI PCI Link LN15> on acpi0
pci_link22: <ACPI PCI Link LN16> on acpi0
pci_link23: <ACPI PCI Link LN17> on acpi0
pci_link24: <ACPI PCI Link LN18> on acpi0
pci_link25: <ACPI PCI Link LN19> on acpi0
pci_link26: <ACPI PCI Link LN1A> on acpi0
pci_link27: <ACPI PCI Link LN1B> on acpi0
pci_link28: <ACPI PCI Link LN1C> on acpi0
pci_link29: <ACPI PCI Link LN1D> on acpi0
pci_link30: <ACPI PCI Link LN1E> on acpi0
pci_link31: <ACPI PCI Link LN1F> on acpi0
pci_link32: <ACPI PCI Link LUSB> irq 3 on acpi0
pci_link33: <ACPI PCI Link LTID> irq 11 on acpi0
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
pcib0: <ACPI Host-PCI bridge> on acpi0
pci0: <ACPI PCI bus> on pcib0
em0: <Intel(R) PRO/1000 Network Connection, Version - 2.1.7> port 0xecc0-0xecff mem 0xfe100000-0xfe11ffff irq 17 at device 2.0 on pci0
em0: Ethernet address: 00:c0:9f:21:ac:ff
em0:  Speed:N/A  Duplex:N/A
amr0: <LSILogic MegaRAID 1.51> mem 0xfeb00000-0xfeb0ffff irq 20 at device 4.0 on pci0
amr0: <LSILogic PERC 3/SC> Firmware 1.78, BIOS 3.29, 32MB RAM
pci0: <display, VGA> at device 8.0 (no driver attached)
atapci0: <ServerWorks CSB6 UDMA66 controller> port 0x1e8-0x1ef,0x3ec-0x3ef,0x168-0x16f,0x36c-0x36f,0x900-0x90f irq 11 at device 14.0 on pci0
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
atapci1: <ServerWorks CSB6 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x8b0-0x8bf at device 15.1 on pci0
ata0: <ATA channel 0> on atapci1
ata1: <ATA channel 1> on atapci1
ohci0: <OHCI (generic) USB controller> mem 0xfe120000-0xfe120fff irq 3 at device 15.2 on pci0
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
isab0: <PCI-ISA bridge> at device 15.3 on pci0
isa0: <ISA bus> on isab0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
ppc0: <ECP parallel printer port> port 0x378-0x37f,0x778-0x77f irq 7 drq 1 on acpi0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xec000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
uhid0: American Power Conversion Back-UPS 350 FW: 5.4.I USB FW: c1, rev 1.10/1.00, addr 2, iclass 3/0
umass0: Western Digital Corporation Western Digital USB Hard Drive, rev 2.00/ff.ff, addr 3
uhub0: 2 ports with 2 removable, self powered
isab0: <PCI-ISA bridge> at device 15.3 on pci0
isa0: <ISA bus> on isab0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
ppc0: <ECP parallel printer port> port 0x378-0x37f,0x778-0x77f irq 7 drq 1 on acpi0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xec000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
uhid0: American Power Conversion Back-UPS 350 FW: 5.4.I USB FW: c1, rev 1.10/1.00, addr 2, iclass 3/0
umass0: Western Digital Corporation Western Digital USB Hard Drive, rev 2.00/ff.ff, addr 3
Timecounter "TSC" frequency 2399337620 Hz quality 800
Timecounters tick every 1.000 msec
acd0: CDROM <GCR-8481B/2.01> at ata2-master UDMA33
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 34678MB (71020544 sectors) RAID 1 (optimal)
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <WDC WD800BB-00FRA0 77.0> Fixed Direct Access SCSI-2 device 
da0: 1.000MB/s transfers
da0: 76319MB (156301488 512 byte sectors: 255H 63S/T 9729C)
Trying to mount root from ufs:/dev/amrd0s2a
----- dmesg end -----

>Description:
A RAID-1 array connected to a Dell PERC 3/SC RAID Controller (looks
like a rebranded LSILogic MegaRAID SCSI Controller of some sort) gives
very low write performance:

  root at gaia/tmp> dd if=/dev/zero of=zeroes bs=1m count=100
  ^Tload: 0.00  cmd: dd 6920 [wdrain] 0.00u 0.44s 1% 1656k
  47+0 records in
  46+0 records out
  48234496 bytes transferred in 6.768442 secs (7126381 bytes/sec)
  100+0 records in
  100+0 records out
  104857600 bytes transferred in 14.106778 secs (7433136 bytes/sec)
  root at gaia/tmp> swapoff -a
  swapoff: removing /dev/amrd0s2b as swap device
  root at gaia/tmp> dd if=/dev/zero of=/dev/amrd0s2b bs=1m count=100
  ^Tload: 0.00  cmd: dd 6923 [physwr] 0.00u 0.08s 0% 1656k
  31+0 records in
  30+0 records out
  31457280 bytes transferred in 3.652500 secs (8612534 bytes/sec)
  100+0 records in
  100+0 records out
  104857600 bytes transferred in 11.750886 secs (8923378 bytes/sec)
  root at gaia/tmp> dd if=/dev/amrd0s2b of=/dev/null bs=1m count=100
  100+0 records in
  100+0 records out
  104857600 bytes transferred in 5.468511 secs (19174799 bytes/sec)
  root at gaia/tmp> swapon -a
  swapon: adding /dev/amrd0s2b as swap device
  root at gaia/tmp>

I pressed ^T during the dd so it shows which state the process is in. In
both cases, it spends nearly all of its time in that state.

With 7.1MB/sec for a filesystem and 8.5MB/sec for a raw sequential
write, the performance of this RAID array remains *far* below its
capabilities. I'm also thinking that 18.3MB/sec for a sequential read is
a bit low.

This machine was running under FreeBSD 4.11 before, and performance was
as can be expected, a few dozen MB/sec (around 40MB/sec iirc).


>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted: