em very slow, shared irq... on 6.3p8
Holger Kipp
hk at alogis.com
Wed Feb 27 11:01:26 UTC 2008
Hello,
I updated a system with 12 dc-interfaces to a new hardware
with 14 em-interfaces. Yes, it is a firewall.
New System is 6.2-RELEASE-p8.
What I now experience between two internal networks (100MBit/s each)
is the following:
1318 packets transmitted, 1317 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.387/246.153/2441.392/324.142 ms
tcpdump on the firewall shows similar delays (on the outgoing
interface).
tcpdump on the system I ping however shows very quick responses
for incoming packages (ie usually less than a millisecond).
I therefore assume that the problem is between receiving the
irq from em<x> and getting the data from the interface on the firewall
itself.
My first option would be to activate polling on em-interfaces - but as
I did not experience this sort of notieceable slowdown with the old
dc-based firewall (without polling), maybe someone can shed some light
on this strange behaviour or has other suggestions as well?
More details and dmesg below.
Help and suggestions welcome :-)
Best regards,
Holger Kipp
PS: Regarding the MPSAFE-issue - I need to use ipsec, so don't have a choice
here.
On the firewall I see many irqs (sometimes more than 5000) very often
on one of the em-interfaces using sysctl/vmstat:
--------------------------------------------------------------------------------
2 users Load 0.00 0.00 0.04 Feb 27 12:20
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 14804 3724 32800 3800 1327728 count
All 722324 5248 76723364 5696 pages
Interrupts
Proc:r p d s w Csw Trp Sys Int Sof Flt cow 7606 total
27 7862 1 462 4105 158840 wire 1: atkb
14988 act 6: fdc0
0.2%Sys 3.0%Intr 0.0%User 0.0%Nice 96.8%Idl 547868 inact 12: psm
| | | | | | | | | | 1244 cache 14: ata
++ 1326484 free 359 16: em4
daefr 2761 17: em5
Namei Name-cache Dir-cache prcfr 28: twe
Calls hits % hits % react 306 52: em0
pdwak 1 53: em1
zfod pdpgs 177 55: em3
Disks twed0 ozfod intrn 2001 cpu0: time
KB/t 21.33 %slo-z 114464 buf 2001 cpu1: time
tps 1 1 tfree 10 dirtybuf
MB/s 0.01 100000 desiredvnodes
% busy 0 66594 numvnodes
19758 freevnodes
--------------------------------------------------------------------------------
dmesg
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p8 #1: Thu Oct 4 16:07:31 CEST 2007
WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant
WARNING: MPSAFE network stack disabled, expect reduced performance.
MPTable: <INTEL MUKILTEO >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 CPU 6420 @ 2.13GHz (2128.01-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
real memory = 2146349056 (2046 MB)
avail memory = 2095296512 (1998 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0: Assuming intbase of 0
ioapic1: Assuming intbase of 24
ioapic2: Assuming intbase of 48
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
kbd1 at kbdmux0
cpu0 on motherboard
cpu1 on motherboard
pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
pcib0: unable to route slot 28 INTB
pcib1: <PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pcib2: <MPTable PCI-PCI bridge> at device 0.0 on pci1
pci2: <PCI bus> on pcib2
twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x4000-0x400f mem 0xe0100000-0xe010000f,0xe0800000-0xe0ffffff irq 28 at device 2.0 on pci2
twe0: [GIANT-LOCKED]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
pci1: <base peripheral, interrupt controller> at device 0.1 (no driver attached)
pcib3: <PCI-PCI bridge> at device 0.2 on pci1
pci3: <PCI bus> on pcib3
pcib4: <MPTable PCI-PCI bridge> at device 3.0 on pci3
pci4: <PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x5000-0x503f mem 0xe1000000-0xe101ffff irq 52 at device 4.0 on pci4
em0: Ethernet address: 00:0e:0c:c6:cc:54
em0: [GIANT-LOCKED]
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x5040-0x507f mem 0xe1020000-0xe103ffff irq 53 at device 4.1 on pci4
em1: Ethernet address: 00:0e:0c:c6:cc:55
em1: [GIANT-LOCKED]
em2: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x5080-0x50bf mem 0xe1040000-0xe105ffff irq 54 at device 6.0 on pci4
em2: Ethernet address: 00:0e:0c:c6:cc:56
em2: [GIANT-LOCKED]
em3: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x50c0-0x50ff mem 0xe1060000-0xe107ffff irq 55 at device 6.1 on pci4
em3: Ethernet address: 00:0e:0c:c6:cc:57
em3: [GIANT-LOCKED]
pci1: <base peripheral, interrupt controller> at device 0.3 (no driver attached)
pcib5: <PCI-PCI bridge> irq 16 at device 3.0 on pci0
pci10: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 0.0 on pci10
pci11: <PCI bus> on pcib6
pcib7: <MPTable PCI-PCI bridge> at device 0.0 on pci11
pci12: <PCI bus> on pcib7
em4: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x6000-0x601f mem 0xe1120000-0xe113ffff,0xe1100000-0xe111ffff irq 16 at device 0.0 on pci12
em4: Memory Access and/or Bus Master bits were not set!
em4: Ethernet address: 00:15:17:25:48:88
em4: [GIANT-LOCKED]
em5: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x6020-0x603f mem 0xe1160000-0xe117ffff,0xe1140000-0xe115ffff irq 17 at device 0.1 on pci12
em5: Memory Access and/or Bus Master bits were not set!
em5: Ethernet address: 00:15:17:25:48:89
em5: [GIANT-LOCKED]
pcib8: <MPTable PCI-PCI bridge> at device 1.0 on pci11
pci13: <PCI bus> on pcib8
em6: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x7000-0x701f mem 0xe1220000-0xe123ffff,0xe1200000-0xe121ffff irq 16 at device 0.0 on pci13
em6: Memory Access and/or Bus Master bits were not set!
em6: Ethernet address: 00:15:17:25:48:8a
em6: [GIANT-LOCKED]
em7: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x7020-0x703f mem 0xe1260000-0xe127ffff,0xe1240000-0xe125ffff irq 17 at device 0.1 on pci13
em7: Memory Access and/or Bus Master bits were not set!
em7: Ethernet address: 00:15:17:25:48:8b
em7: [GIANT-LOCKED]
pcib9: <PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci14: <PCI bus> on pcib9
pcib10: <PCI-PCI bridge> at device 0.0 on pci14
pci15: <PCI bus> on pcib10
pcib11: <MPTable PCI-PCI bridge> at device 0.0 on pci15
pci16: <PCI bus> on pcib11
em8: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x8000-0x801f mem 0xe1320000-0xe133ffff,0xe1300000-0xe131ffff irq 16 at device 0.0 on pci16
em8: Memory Access and/or Bus Master bits were not set!
em8: Ethernet address: 00:15:17:25:48:64
em8: [GIANT-LOCKED]
em9: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x8020-0x803f mem 0xe1360000-0xe137ffff,0xe1340000-0xe135ffff irq 17 at device 0.1 on pci16
em9: Memory Access and/or Bus Master bits were not set!
em9: Ethernet address: 00:15:17:25:48:65
em9: [GIANT-LOCKED]
pcib12: <MPTable PCI-PCI bridge> at device 1.0 on pci15
pci17: <PCI bus> on pcib12
em10: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x9000-0x901f mem 0xe1420000-0xe143ffff,0xe1400000-0xe141ffff irq 16 at device 0.0 on pci17
em10: Memory Access and/or Bus Master bits were not set!
em10: Ethernet address: 00:15:17:25:48:66
em10: [GIANT-LOCKED]
em11: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x9020-0x903f mem 0xe1460000-0xe147ffff,0xe1440000-0xe145ffff irq 17 at device 0.1 on pci17
em11: Memory Access and/or Bus Master bits were not set!
em11: Ethernet address: 00:15:17:25:48:67
em11: [GIANT-LOCKED]
pcib13: <MPTable PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci18: <PCI bus> on pcib13
em12: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xa000-0xa01f mem 0xe1500000-0xe151ffff irq 16 at device 0.0 on pci18
em12: Ethernet address: 00:30:48:90:86:7a
em12: [GIANT-LOCKED]
pcib14: <MPTable PCI-PCI bridge> irq 7 at device 28.5 on pci0
pci19: <PCI bus> on pcib14
em13: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xb000-0xb01f mem 0xe1600000-0xe161ffff irq 17 at device 0.0 on pci19
em13: Ethernet address: 00:30:48:90:86:7b
em13: [GIANT-LOCKED]
pcib15: <MPTable PCI-PCI bridge> at device 30.0 on pci0
pci20: <PCI bus> on pcib15
pci20: <display, VGA> at device 4.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3020-0x302f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model MouseMan+, device ID 0
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0303> can't assign resources (port)
unknown: <INT0800> can't assign resources (memory)
unknown: <PNP0c02> can't assign resources (memory)
unknown: <PNP0f13> can't assign resources (irq)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0700> can't assign resources (port)
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
IP Filter: v4.1.13 initialized. Default = pass all, Logging = enabled
acd0: DVDROM <HL-DT-STDVD-ROM GDR8164B/0L06> at ata0-slave UDMA33
twed0: <Unit 0, TwinStor, Normal> on twe0
twed0: 76318MB (156299440 sectors)
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/twed0s1a
em9: link state changed to UP
em8: link state changed to UP
em10: link state changed to UP
em0: link state changed to UP
em3: link state changed to UP
[...]
More information about the freebsd-stable
mailing list