Weird "ignoring syn" problem

Bill Moran wmoran at
Tue Jun 12 14:29:53 UTC 2007

This one has got me pretty befuddled.

We're seeing some really odd behaviour with FreeBSD ignoring SYN packets.
I've been trying to diagnose this for a couple of weeks now, and my current
guess is that there's something wrong with the em driver.  Here's a narrowed
down list of what I've ruled out:
*) I've done my best to eliminate other network components as the problem.
   My theory at this point is that it can't possibly be any other network
   hardware, based on the tcpdump show below.
*) The problem occurred on both FreeBSD 6.1 and FreeBSD 6.2-p3.
*) The problem does not appear to be tied to CPU usage -- the CPU is nearly
   idle when the problem occurs.
*) I can now reproduce it pretty easily, so I'll know when it's fixed.
*) The system exhibiting the problem is running 15 jails, but they are
   idle 95% of the time.  The problem initially occurred inside one of
   the jails, but I just recreated it outside the jail (on the host) and
   it's _easier_ to reproduce outside the jail.
*) The problem occurred with both GENERIC, and the SMP kernel (this is a
   dual-CPU, hyperthreaded system)
*) I've tested and the behavior occurs both with a dynamically generated
   file (from PHP) or from a static file.

The nature of the beast is that we've got a SOAP application running under
Apache and PHP.  This application is subject to many requests in rapid
succession, such that load can be simulated by the following loop:

while true; do fetch; done

The problem is that occasionally, the Apache server machine just ignores
SYN packets.  Take the following tcpdump output for example:

13:34:17.312296 IP > S 2645061726:2645061726(0) win 65535 <mss 1380,nop,wscale 1,nop,nop,timestamp 2690201156 0,sackOK,eol>
13:34:20.312398 IP > S 2645061726:2645061726(0) win 65535 <mss 1380,nop,wscale 1,nop,nop,timestamp 2690204156 0,sackOK,eol>
13:34:23.512626 IP > S 2645061726:2645061726(0) win 65535 <mss 1380,nop,wscale 1,nop,nop,timestamp 2690207356 0,sackOK,eol>

This is the _only_ traffic on port 80 during the test.  It looks like the
kernel has ignored the initial syn packet and two duplicates.  I've seen it
take as long as 45 seconds to establish a connection, and this causes
ugly performance problems, as well as frequent timeouts on the client end.
The only clue I've found so far is this output from netstat -s.

        153099 syncache entries added
                6184 retransmitted
                6491 dupsyn
                0 dropped
                150923 completed
                0 bucket overflow
                0 cache overflow
                235 reset
                1941 stale
                0 aborted
                0 badack
                0 unreach
                0 zone failures

Unfortunately, I've been unable to determine how to fix the problem.  Any
advice is welcome.

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p3 #2: Thu Jun  7 21:37:54 UTC 2007
    root at is00:/usr/obj/usr/src/sys/SMP
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.00GHz (2992.71-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf43  Stepping = 3
  AMD Features=0x20100000<NX,LM>
  Logical CPUs per core: 2
real memory  = 2147221504 (2047 MB)
avail memory = 2096107520 (1999 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  6
 cpu3 (AP): APIC ID:  7
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic1: WARNING: intbase 32 != expected base 24
ioapic2: Changing APIC ID to 10
ioapic2: WARNING: intbase 64 != expected base 56
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 32-55 on motherboard
ioapic2 <Version 2.0> irqs 64-87 on motherboard
kbd1 at kbdmux0
ath_hal: (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <DELL PE BKC> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
amr0: <LSILogic MegaRAID 1.53> mem 0xd80f0000-0xd80fffff,0xdfde0000-0xdfdfffff irq 46 at device 14.0 on pci2
amr0: delete logical drives supported by controller
amr0: <LSILogic PERC 4e/Si> Firmware 521X, BIOS H430, 256MB RAM
pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1
pci3: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xecc0-0xecff mem 0xdfbe0000-0xdfbfffff irq 37 at device 11.0 on pci3
em0: Ethernet address: 00:04:23:c8:ff:f4
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xec80-0xecbf mem 0xdfbc0000-0xdfbdffff irq 38 at device 11.1 on pci3
em1: Ethernet address: 00:04:23:c8:ff:f5
pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
em2: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xdcc0-0xdcff mem 0xdf8e0000-0xdf8fffff irq 64 at device 7.0 on pci6
em2: Ethernet address: 00:13:72:4f:71:23
pcib7: <ACPI PCI-PCI bridge> at device 0.2 on pci5
pci7: <ACPI PCI bus> on pcib7
em3: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xccc0-0xccff mem 0xdf6e0000-0xdf6fffff irq 65 at device 8.0 on pci7
em3: Ethernet address: 00:13:72:4f:71:24
pcib8: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci8: <ACPI PCI bus> on pcib8
uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0xace0-0xacff irq 16 at device 29.0 on pci0
usb0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0xacc0-0xacdf irq 19 at device 29.1 on pci0
usb1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801EB (ICH5) USB controller USB-C> port 0xaca0-0xacbf irq 18 at device 29.2 on pci0
usb2: <Intel 82801EB (ICH5) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xdff00000-0xdff003ff irq 23 at device 29.7 on pci0
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 6 ports with 6 removable, self powered
uhub4: vendor 0x413c product 0xa001, class 9/0, rev 2.00/0.00, addr 2
uhub4: multiple transaction translators
uhub4: 2 ports with 2 removable, self powered
pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pci9: <unknown> at device 5.0 (no driver attached)
pci9: <unknown> at device 5.1 (no driver attached)
pci9: <unknown> at device 5.2 (no driver attached)
atapci0: <SiI 0680 UDMA133 controller> port 0xbcf0-0xbcf7,0xbce4-0xbce7,0xbcd8-0xbcdf,0xbcd0-0xbcd3,0xbc70-0xbc7f mem 0xdf3fec00-0xdf3fecff irq 23 at device 6.0 on pci9
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
pci9: <display, VGA> at device 13.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci1
ata1: <ATA channel 1> on atapci1
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xec000-0xeffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ukbd0: Dell DRAC4, rev 1.10/0.00, addr 2, iclass 3/1
kbd2 at ukbd0
ums0: Dell DRAC4, rev 1.10/0.00, addr 2, iclass 3/1
ums0: 3 buttons and Z dir.
Timecounters tick every 1.000 msec
acd0: CDROM <TEAC CD-ROM CD-224E-N/3.AB> at ata0-master UDMA33
device_attach: afd0 attach returned 6
acd1: CDROM <VIRTUALCDROM DRIVE/> at ata2-slave PIO3
amr0: delete logical drives supported by controller
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 34680MB (71024640 sectors) RAID 1 (optimal)
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/amrd0s1a

Bill Moran
Collaborative Fusion Inc.

wmoran at
Phone: 412-422-3463x4023

More information about the freebsd-net mailing list