7.1-PRERELEASE: arcmsr write performance problem
Jan Mikkelsen
janm at transactionware.com
Mon Dec 1 21:45:31 PST 2008
Replying to my own post ...
I have done a test on the same machine comparing 6.3-p1 to 7.1-PRE. The
performance is the expected ~6MB/s (because of the lack of cache) on
6.3-p1, so the BIOS change doesn't seem to be at fault.
This seems to be a regression somewhere between 6.3 to 7.1. The Areca
driver is the same in 6.3 and 7.1, so the problem seems to be elsewhere.
I think this is more than just a "performance" problem. The
observations with gstat showing extremely high ms/w values (I have seen
them as high as 22000) makes it look like IO completion interrupts are
being lost.
Any suggestions on where to look next? Are there obvious candidates?
Jan Mikkelsen wrote:
> Hi,
>
> I am seeing extremely poor performance (~100kB/s) when untaring large
> tar files into fresh ufs filesystems. I see the problem with
> softupdates and without softupdates but with an async mount. This is a
> Supermicro X7DB8 board, 4GB, 2 x Xeon 5140.
>
> Sample gstat output:
>
> dT: 1.033s w: 1.000s
> L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
> 585 61 0 0 0.0 61 170 13812.0 100.1| da2
>
> I see ms/w start at about 200ms with a ~3MB/s throughput, and then I see
> ms/w rise and kBps drop. ms/w goes as high as 16-20s, and then suddenly
> drops back down to about 200ms. Using iostat, while the performance is
> high(er), kb/t is 64kB, as the problem starts it drops towards 2kB.
>
> Copying a single large file doesn't exhibit this problem, although
> throughput isn't great (~3-5MB/s). However, that's better that 100kB/s.
>
> arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
>> mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0
>> on pci10
> ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07
> ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06
> arcmsr0: [ITHREAD]
>
> There are eight disks connected in a RAID-6 configuration. The
> controller's cache is write-through and the disks' write caches are
> disabled. NCQ is enabled on the drives.
>
> The same hardware when it ran 6.3-p1 didn't have this problem. However,
> the system BIOS was updated at the same time as the operating system (in
> an attempt to solve a recent em problem), so it is possible that it is a
> BIOS related problem. The same build on an entirely different machine
> with an aac controller and SAS disks also doesn't show this problem.
>
> Running 'devinfo -r' doesn't list arcmsr as having an interrupt at all.
> (see below). That strikes me as odd; checking another machine that is
> still running 6.2 with an arcmsr controller, I can see the interrupt
> just fine.
>
> So:
>
> - Does anyone have any suggestions?
>
> - Is it normal for arcmsr to not show an interrupt in the output from
> devinfo in 7.1?
>
> Full dmesg, devinfo below.
>
> Thanks,
>
> Jan Mikkelsen
>
>
> Copyright (c) 1992-2008 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 7.1-PRERELEASE #0: Mon Dec 1 14:53:12 EST 2008
>
> root at valhalla.transactionware.com:/home/janm/p4/freebsd-image-std-2008.2/work/base-freebsd/home/janm/p4/freebsd-image-std-2008.2/FreeBSD/src/sys/TW-SMP
>
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2333.35-MHz
> K8-class CPU)
> Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>
> Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
>
> AMD Features=0x20100800<SYSCALL,NX,LM>
> AMD Features2=0x1<LAHF>
> Cores per package: 2
> usable memory = 4280651776 (4082 MB)
> avail memory = 4117843968 (3927 MB)
> ACPI APIC Table: <PTLTD APIC >
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> cpu0 (BSP): APIC ID: 0
> cpu1 (AP): APIC ID: 1
> cpu2 (AP): APIC ID: 6
> cpu3 (AP): APIC ID: 7
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> ioapic1 <Version 2.0> irqs 24-47 on motherboard
> kbd1 at kbdmux0
> ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
> acpi0: <SMCI SMCISLP2> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
> pci2: <ACPI PCI bus> on pcib2
> pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
> pci3: <ACPI PCI bus> on pcib3
> pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
> pci4: <ACPI PCI bus> on pcib4
> ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port
> 0x2400-0x24ff,0x2000-0x20ff mem 0xd8500000-0xd8501fff irq 16 at device
> 2.0 on pci4
> ahd0: [ITHREAD]
> aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
> ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port
> 0x2c00-0x2cff,0x2800-0x28ff mem 0xd8502000-0xd8503fff irq 17 at device
> 2.1 on pci4
> ahd1: [ITHREAD]
> aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
> pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
> pci5: <ACPI PCI bus> on pcib5
> bge0: <Altima Gigabit Ethernet Controller, ASIC rev. 0x105> mem
> 0xd8600000-0xd860ffff irq 16 at device 1.0 on pci5
> miibus0: <MII bus> on bge0
> brgphy0: <BCM5701 10/100/1000baseTX PHY> PHY 1 on miibus0
> brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
> 1000baseT-FDX, auto
> bge0: Ethernet address: 00:40:f4:66:b1:56
> bge0: [ITHREAD]
> pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
> pci6: <ACPI PCI bus> on pcib6
> em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3000-0x301f mem
> 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6
> em0: Using MSI interrupt
> em0: [FILTER]
> em0: Ethernet address: 00:30:48:31:67:86
> em1: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x3020-0x303f mem
> 0xd8420000-0xd843ffff irq 19 at device 0.1 on pci6
> em1: Using MSI interrupt
> em1: [FILTER]
> em1: Ethernet address: 00:30:48:31:67:87
> pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
> pci7: <ACPI PCI bus> on pcib7
> pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
> pci8: <ACPI PCI bus> on pcib8
> pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
> pci9: <ACPI PCI bus> on pcib9
> pcib10: <PCI-PCI bridge> at device 0.0 on pci9
> pci10: <PCI bus> on pcib10
> arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
>> mem 0xd8900000-0xd8900fff,0xd8000000-0xd83fffff irq 16 at device 14.0
>> on pci10
> ARECA RAID ADAPTER0: Driver Version 1.20.00.15 2007-10-07
> ARECA RAID ADAPTER0: FIRMWARE VERSION V1.46 2008-08-06
> arcmsr0: [ITHREAD]
> pcib11: <PCI-PCI bridge> at device 0.2 on pci9
> pci11: <PCI bus> on pcib11
> pci0: <base peripheral> at device 8.0 (no driver attached)
> pcib12: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
> pci12: <ACPI PCI bus> on pcib12
> uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port
> 0x1800-0x181f irq 17 at device 29.0 on pci0
> uhci0: [GIANT-LOCKED]
> uhci0: [ITHREAD]
> usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
> usb0: USB revision 1.0
> uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
> uhub0: 2 ports with 2 removable, self powered
> uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port
> 0x1820-0x183f irq 19 at device 29.1 on pci0
> uhci1: [GIANT-LOCKED]
> uhci1: [ITHREAD]
> usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
> usb1: USB revision 1.0
> uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
> uhub1: 2 ports with 2 removable, self powered
> uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
> 0x1840-0x185f irq 18 at device 29.2 on pci0
> uhci2: [GIANT-LOCKED]
> uhci2: [ITHREAD]
> usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
> usb2: USB revision 1.0
> uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
> uhub2: 2 ports with 2 removable, self powered
> ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd8c00400-0xd8c007ff irq
> 17 at device 29.7 on pci0
> ehci0: [GIANT-LOCKED]
> ehci0: [ITHREAD]
> usb3: EHCI version 1.0
> usb3: companion controllers, 2 ports each: usb0 usb1 usb2
> usb3: <Intel 63XXESB USB 2.0 controller> on ehci0
> usb3: USB revision 2.0
> uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
> uhub3: 6 ports with 6 removable, self powered
> pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci13: <ACPI PCI bus> on pcib13
> vgapci0: <VGA-compatible display> port 0x4000-0x40ff mem
> 0xd0000000-0xd7ffffff,0xd8800000-0xd880ffff irq 18 at device 1.0 on pci13
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel 63XXESB2 UDMA100 controller> port
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
> ata0: <ATA channel 0> on atapci0
> ata0: [ITHREAD]
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> acpi_button0: <Power Button> on acpi0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> atkbd0: [ITHREAD]
> psm0: <PS/2 Mouse> irq 12 on atkbdc0
> psm0: [GIANT-LOCKED]
> psm0: [ITHREAD]
> psm0: model Generic PS/2 mouse, device ID 0
> sio0: configured irq 4 not in bitmap of probed irqs 0
> sio0: port may not be enabled
> sio0: configured irq 4 not in bitmap of probed irqs 0
> sio0: port may not be enabled
> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
> acpi0
> sio0: type 16550A
> sio0: [FILTER]
> sio1: configured irq 3 not in bitmap of probed irqs 0
> sio1: port may not be enabled
> sio1: configured irq 3 not in bitmap of probed irqs 0
> sio1: port may not be enabled
> sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
> sio1: type 16550A
> sio1: [FILTER]
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
> fdc0: [FILTER]
> ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> ppc0: FIFO with 16/16/9 bytes threshold
> ppbus0: <Parallel port bus> on ppc0
> ppbus0: [ITHREAD]
> plip0: <PLIP network interface> on ppbus0
> plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> ppc0: [GIANT-LOCKED]
> ppc0: [ITHREAD]
> cpu0: <ACPI CPU> on acpi0
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> cpu1: <ACPI CPU> on acpi0
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> cpu2: <ACPI CPU> on acpi0
> est2: <Enhanced SpeedStep Frequency Control> on cpu2
> p4tcc2: <CPU Frequency Thermal Control> on cpu2
> cpu3: <ACPI CPU> on acpi0
> est3: <Enhanced SpeedStep Frequency Control> on cpu3
> p4tcc3: <CPU Frequency Thermal Control> on cpu3
> orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> Timecounters tick every 1.000 msec
> acd0: DVDR <PIONEER DVD-RW DVR-111D/1.23> at ata0-master UDMA66
> Waiting 5 seconds for SCSI devices to settle
> (probe46:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step
> da0 at arcmsr0 bus 0 target 0 lun 0
> da0: <Areca ARC-1220-VOL#00 R001> Fixed Direct Access SCSI-5 device
> da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
> da0: 77247MB (158201856 512 byte sectors: 255H 63S/T 9847C)
> da1 at arcmsr0 bus 0 target 1 lun 0
> da1: <Areca ARC-1220-VOL#01 R001> Fixed Direct Access SCSI-5 device
> da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
> da1: 953673MB (1953122304 512 byte sectors: 255H 63S/T 121576C)
> da2 at arcmsr0 bus 0 target 2 lun 0
> da2: <Areca ARC-1220-VOL#02 R001> Fixed Direct Access SCSI-5 device
> da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
> da2: 800131MB (1638669312 512 byte sectors: 255H 63S/T 102002C)
> sa0 at ahd1 bus 0 target 6 lun 0
> sa0: <SEAGATE DAT 04106-XXX 7600> Removable Sequential Access SCSI-2
> device
> sa0: 10.000MB/s transfers (10.000MHz, offset 15)
> SMP: AP CPU #1 Launched!
> SMP: AP CPU #3 Launched!
> SMP: AP CPU #2 Launched!
> Trying to mount root from ufs:/dev/da0s2a
> This module (opensolaris) contains code covered by the
> Common Development and Distribution License (CDDL)
> see http://opensolaris.org/os/licensing/opensolaris_license/
> WARNING: ZFS is considered to be an experimental feature in FreeBSD.
> ZFS filesystem version 6
> ZFS storage pool version 6
> bge0: link state changed to UP
> em0: link state changed to UP
> em1: link state changed to UP
>
>
> nexus0
> acpi0
> Interrupt request lines:
> 9
> I/O ports:
> 0x10-0x1f
> 0x24-0x25
> 0x28-0x29
> 0x2c-0x2d
> 0x2e-0x2f
> 0x30-0x31
> 0x34-0x35
> 0x38-0x39
> 0x3c-0x3d
> 0x4e-0x4f
> 0x50-0x53
> 0x63
> 0x65
> 0x67
> 0x72-0x77
> 0x80
> 0x90-0x9f
> 0xa4-0xa5
> 0xa8-0xa9
> 0xac-0xad
> 0xb0-0xb5
> 0xb8-0xb9
> 0xbc-0xbd
> 0x295-0x296
> 0x4d0-0x4d1
> 0x800-0x80f
> 0xca2-0xca3
> 0xca8-0xcaf
> 0x1000-0x107f
> 0x1180-0x11bf
> 0xfe00
> I/O memory addresses:
> 0xe0000000-0xefffffff
> 0xfe000000-0xfe01ffff
> 0xfe600000-0xfe6fffff
> 0xfec80000-0xfec80fff
> 0xfed1c000-0xfed1ffff
> 0xfee00000-0xfee0ffff
> cpu0
> acpi_perf0
> est0
> p4tcc0
> cpufreq0
> cpu1
> acpi_perf1
> est1
> p4tcc1
> cpufreq1
> cpu2
> acpi_perf2
> est2
> p4tcc2
> cpufreq2
> cpu3
> acpi_perf3
> est3
> p4tcc3
> cpufreq3
> pcib0
> pci0
> hostb0
> pcib1
> pci1
> pcib2
> pci2
> pcib3
> pci3
> pcib4
> pci4
> ahd0
> Interrupt request lines:
> 16
> I/O ports:
> 0x2000-0x20ff
> 0x2400-0x24ff
> I/O memory addresses:
> 0xd8500000-0xd8501fff
> ahd1
> Interrupt request lines:
> 17
> I/O ports:
> 0x2800-0x28ff
> 0x2c00-0x2cff
> I/O memory addresses:
> 0xd8502000-0xd8503fff
> pcib5
> pci5
> bge0
> I/O memory addresses:
> 0xd8600000-0xd860ffff
> miibus0
> brgphy0
> pcib6
> pci6
> em0
> Interrupt request lines:
> 256
> I/O ports:
> 0x3000-0x301f
> I/O memory addresses:
> 0xd8400000-0xd841ffff
> em1
> Interrupt request lines:
> 257
> I/O ports:
> 0x3020-0x303f
> I/O memory addresses:
> 0xd8420000-0xd843ffff
> pcib7
> pci7
> pcib8
> pci8
> pcib9
> pci9
> pcib10
> pci10
> arcmsr0
> I/O memory addresses:
> 0xd8000000-0xd83fffff
> 0xd8900000-0xd8900fff
> pcib11
> pci11
> hostb1
> hostb2
> hostb3
> hostb4
> hostb5
> hostb6
> hostb7
> pcib12
> pci12
> uhci0
> I/O ports:
> 0x1800-0x181f
> usb0
> uhub0
> uhci1
> Interrupt request lines:
> 19
> I/O ports:
> 0x1820-0x183f
> usb1
> uhub1
> uhci2
> Interrupt request lines:
> 18
> I/O ports:
> 0x1840-0x185f
> usb2
> uhub2
> ehci0
> I/O memory addresses:
> 0xd8c00400-0xd8c007ff
> usb3
> uhub3
> pcib13
> pci13
> vgapci0
> I/O ports:
> 0x4000-0x40ff
> I/O memory addresses:
> 0xd0000000-0xd7ffffff
> 0xd8800000-0xd880ffff
> isab0
> isa0
> sc0
> vga0
> I/O ports:
> 0x3c0-0x3df
> I/O memory addresses:
> 0xa0000-0xbffff
> orm0
> I/O memory addresses:
> 0xc0000-0xcafff
> atapci0
> I/O ports:
> 0x170-0x177
> 0x1f0-0x1f7
> 0x376
> 0x3f6
> 0x1860-0x186f
> ata0
> Interrupt request lines:
> 14
> acd0
> acpi_sysresource0
> atdma0
> fpupnp0
> attimer0
> attimer1
> pci_link0
> pci_link1
> pci_link2
> pci_link3
> pci_link4
> pci_link5
> pci_link6
> pci_link7
> atkbdc0
> I/O ports:
> 0x60
> 0x64
> atkbd0
> Interrupt request lines:
> 1
> psm0
> Interrupt request lines:
> 12
> psmcpnp0
> sio0
> Interrupt request lines:
> 4
> I/O ports:
> 0x3f8-0x3ff
> sio1
> Interrupt request lines:
> 3
> I/O ports:
> 0x2f8-0x2ff
> fdc0
> Interrupt request lines:
> 6
> DMA request lines:
> 2
> I/O ports:
> 0x3f0-0x3f5
> 0x3f7
> ppc0
> Interrupt request lines:
> 7
> DMA request lines:
> 3
> I/O ports:
> 0x378-0x37f
> ppbus0
> plip0
> lpt0
> ppi0
> acpi_button0
> acpi_timer0
> ACPI I/O ports:
> 0x1008-0x100b
> apic0
> I/O memory addresses:
> 0xfec00000-0xfec0001f
> ram0
> I/O memory addresses:
> 0x0-0x9dfff
> 0x100000-0xcff4ffff
> 0x100000000-0x12fffffff
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
More information about the freebsd-stable
mailing list