Interrupt/speed problems with 6.2 NFS server
Eric Anderson
anderson at freebsd.org
Wed Oct 17 04:27:44 PDT 2007
Doug Clements wrote:
> Hi,
> I have an new NFS server that is processing roughly 15mbit of NFS traffic
> that we recently upgraded from an older 4.10 box. It has a 3-ware raid card,
> and is serving NFS out a single em nic to LAN clients. The machine works
> great just serving NFS, but when I try to copy data from one raid volume to
> another for backups, the machine's NFS performance goes way down, and
> NFSops start taking multiple seconds to perform. The file copy goes
> quite
> quickly, as would be expected. The console of the machine also starts to lag
> pretty badly, and I get the 'typing through mud' effect. I use rdist6 to do
> the backup.
>
> My first impression was that I was having interrupt issues, since during the
> backup, the em interfaces were pushing over 200k interrupts/sec (roughly 60%
> CPU processing interrupts). So I recompiled the kernel with polling enabled
> and enabled it on the NICs. The strange thing is that polling shows enabled
> in ifconfig, but systat -vm still shows the same amount of interrupts. I get
> the same performance with polling enabled.
>
> I'm looking for some guidance on why the machine bogs so much during what
> seems to me to be something that should barely impact machine performance at
> all, and also why polling didn't seem to lower the number of interrupts
> processed. The old machine was 6 years old running an old intel raid5, and
> it handled NFS and the concurrent file copies without a sweat.
>
> My 3ware is setup as follows:
> a 2 disk mirror, for the system
> a 4 disk raid10, for /mnt/data1
> a 4 disk raid10, for /mnt/data2
>
> Copyright (c) 1992-2007 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 6.2-RELEASE-p8 #0: Thu Oct 11 10:43:22 PDT 2007
> admin at madonna.linkline.com :/usr/obj/usr/src/sys/MADONNA
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Genuine Intel(R) CPU @ 2.66GHz (2670.65-MHz K8-class
> CPU)
> Origin = "GenuineIntel" Id = 0x6f4 Stepping = 4
> Features=0xbfebfbff<FPU,VME,DE
> ,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>
> Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>,<b18>>
>
> AMD Features=0x20100800<SYSCALL,NX,LM>
> AMD Features2=0x1<LAHF>
> Cores per package: 2
> real memory = 4831838208 (4608 MB)
> avail memory = 4125257728 (3934 MB)
> ACPI APIC Table: <INTEL S5000PSL>
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> ioapic1 <Version 2.0> irqs 24-47 on motherboard
> lapic0: Forcing LINT1 to edge trigger
> kbd1 at kbdmux0
> ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
> acpi0: <INTEL S5000PSL> on motherboard
> acpi0: Power Button (fixed)
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
> cpu0: <ACPI CPU> on acpi0
> acpi_throttle0: <ACPI CPU Throttling> on cpu0
> acpi_button0: <Sleep Button> on acpi0
> acpi_button1: <Power Button> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xca2,0xca3,0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
> pci2: <ACPI PCI bus> on pcib2
> pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
> pci3: <ACPI PCI bus> on pcib3
> pcib4: <ACPI PCI-PCI bridge> irq 17 at device 1.0 on pci2
> pci4: <ACPI PCI bus> on pcib4
> pcib5: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
> pci5: <ACPI PCI bus> on pcib5
> em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
> 0x3020-0x303f mem 0xf8820000-0xf883ffff,0xf8400000-0xf87fffff irq 18 at
> device 0.0 on pci5
> em0: Ethernet address: 00:15:17:21:bf:30
> em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
> 0x3000-0x301f mem 0xf8800000-0xf881ffff,0xf8000000-0xf83fffff irq 19 at
> device 0.1 on pci5
> em1: Ethernet address: 00:15:17:21:bf:31
> pcib6: <ACPI PCI-PCI bridge> at device 0.3 on pci1
> pci6: <ACPI PCI bus> on pcib6
> 3ware device driver for 9000 series storage controllers, version:
> 3.60.02.012
> twa0: <3ware 9000 series Storage Controller> port 0x2000-0x203f mem
> 0xfa000000-0xfbffffff,0xf8900000-0xf8900fff irq 26 at device 2.0 on pci6
> twa0: [GIANT-LOCKED]
> twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-12, 12 ports,
> Firmware FE9X 3.08.00.004, BIOS BE9X 3.08.00.002
> pcib7: <PCI-PCI bridge> at device 3.0 on pci0
> pci7: <PCI bus> on pcib7
> pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
> pci8: <ACPI PCI bus> on pcib8
> pcib9: <ACPI PCI-PCI bridge> at device 5.0 on pci0
> pci9: <ACPI PCI bus> on pcib9
> pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0
> pci10: <ACPI PCI bus> on pcib10
> pcib11: <PCI-PCI bridge> at device 7.0 on pci0
> pci11: <PCI bus> on pcib11
> pci0: <base peripheral> at device 8.0 (no driver attached)
> pcib12: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
> pci12: <ACPI PCI bus> on pcib12
> uhci0: <UHCI (generic) USB controller> port 0x4080-0x409f irq 23 at device
> 29.0 on pci0
> uhci0: [GIANT-LOCKED]
> usb0: <UHCI (generic) USB controller> on uhci0
> usb0: USB revision 1.0
> uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub0: 2 ports with 2 removable, self powered
> uhci1: <UHCI (generic) USB controller> port 0x4060-0x407f irq 22 at device
> 29.1 on pci0
> uhci1: [GIANT-LOCKED]
> usb1: <UHCI (generic) USB controller> on uhci1
> usb1: USB revision 1.0
> uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub1: 2 ports with 2 removable, self powered
> uhci2: <UHCI (generic) USB controller> port 0x4040-0x405f irq 23 at device
> 29.2 on pci0
> uhci2: [GIANT-LOCKED]
> usb2: <UHCI (generic) USB controller> on uhci2
> usb2: USB revision 1.0
> uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub2: 2 ports with 2 removable, self powered
> uhci3: <UHCI (generic) USB controller> port 0x4020-0x403f irq 22 at device
> 29.3 on pci0
> uhci3: [GIANT-LOCKED]
> usb3: <UHCI (generic) USB controller> on uhci3
> usb3: USB revision 1.0
> uhub3: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub3: 2 ports with 2 removable, self powered
> ehci0: <EHCI (generic) USB 2.0 controller> mem 0xf8c00400-0xf8c007ff irq 23
> at device 29.7 on pci0
> ehci0: [GIANT-LOCKED]
> usb4: EHCI version 1.0
> usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
> usb4: <EHCI (generic) USB 2.0 controller> on ehci0
> usb4: USB revision 2.0
> uhub4: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
> uhub4: 8 ports with 8 removable, self powered
> pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci13: <ACPI PCI bus> on pcib13
> pci13: <display, VGA> at device 12.0 (no driver attached)
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel 63XXESB2 UDMA100 controller> port
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 20 at device 31.1 on
> pci0
> ata0: <ATA channel 0> on atapci0
> ata1: <ATA channel 1> on atapci0
> atapci1: <Intel 63XXESB2 SATA300 controller> port
> 0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem
> 0xf8c00000-0xf8c003ff irq 20 at device 31.2 on pci0
> ata2: <ATA channel 0> on atapci1
> ata3: <ATA channel 1> on atapci1
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
> acpi0
> sio0: type 16550A
> sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
> sio1: type 16550A
> orm0: <ISA Option ROMs> at iomem
> 0xc0000-0xc8fff,0xc9000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0
> ppc0: cannot reserve I/O port range
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> Timecounter "TSC" frequency 2670647184 Hz quality 800
> Timecounters tick every 1.000 msec
> acd0: CDRW <HL-DT-ST RW/DVD GCC-H21N/1.01> at ata0-master UDMA33
> da0 at twa0 bus 0 target 0 lun 0
> da0: <AMCC 9550SX-12 DISK 3.08> Fixed Direct Access SCSI-5 device
> da0: 100.000MB/s transfers
> da0: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C)
> da1 at twa0 bus 0 target 1 lun 0
> da1: <AMCC 9550SX-12 DISK 3.08> Fixed Direct Access SCSI-5 device
> da1: 100.000MB/s transfers
> da1: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C)
> da2 at twa0 bus 0 target 2 lun 0
> da2: <AMCC 9550SX-12 DISK 3.08> Fixed Direct Access SCSI-5 device
> da2: 100.000MB/s transfers
> da2: 476816MB (976519168 512 byte sectors: 255H 63S/T 60785C)
> Trying to mount root from ufs:/dev/da0s1a
> em1: link state changed to UP
>
>
> [root at madonna ~]# mount
> /dev/da0s1a on / (ufs, local)
> devfs on /dev (devfs, local)
> /dev/da0s1e on /usr (ufs, local, soft-updates)
> /dev/da0s1d on /var (ufs, local, soft-updates)
> /dev/da1s1d on /mnt/data1 (ufs, NFS exported, local, noatime, soft-updates)
> /dev/da2s1d on /mnt/data2 (ufs, NFS exported, local, noatime, soft-updates)
>
> [root at madonna ~]# df -hi
> Filesystem Size Used Avail Capacity iused ifree %iused Mounted
> on
> /dev/da0s1a 4.8G 337M 4.1G 7% 3648 655806 1% /
> devfs 1.0K 1.0K 0B 100% 0 0 100% /dev
> /dev/da0s1e 203G 2.5G 184G 1% 235268 27320570 1% /usr
> /dev/da0s1d 9.7G 3.3M 8.9G 0% 220 1318690 0% /var
> /dev/da1s1d 451G 36G 379G 9% 1220061 59920929 2%
> /mnt/data1
> /dev/da2s1d 451G 124G 290G 30% 4143375 56997615 7%
> /mnt/data2
>
> [root at madonna ~]# more /etc/exports
> /mnt/data1 -maproot=0 -alldirs courtney bjork cher joan beth
> kelli miho deborah
> /mnt/data2 -maproot=0 -alldirs courtney bjork cher joan beth
> kelli miho deborah
>
> [root at madonna ~]# ifconfig -a
> em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
> options=4b<RXCSUM,TXCSUM,VLAN_MTU,POLLING>
> ether 00:15:17:21:bf:30
> media: Ethernet autoselect
> status: no carrier
> em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> options=4b<RXCSUM,TXCSUM,VLAN_MTU,POLLING>
> inet x.x.x.x netmask 0xffffffe0 broadcast x.x.x.x
> ether 00:15:17:21:bf:31
> media: Ethernet 100baseTX <full-duplex>
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
> inet6 ::1 prefixlen 128
> inet 127.0.0.1 netmask 0xff000000
>
> Anyone have any clue about what might be going on?
Can you also send the output of ps -auxl?
Also - do you notice this performance drop when running something like
one of the network performance tools? I'd like to isolate the disk
activity from the network activity for a clean test..
Eric
More information about the freebsd-hackers
mailing list