kern/123463: repeatable crash related to ipsec-tools

matthew.seaman at thebunker.net matthew.seaman at thebunker.net
Tue May 6 14:00:07 UTC 2008 

>Number:         123463
>Category:       kern
>Synopsis:       repeatable crash related to ipsec-tools
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue May 06 14:00:06 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Matthew Seaman
>Release:        FreeBSD 7.0-RELEASE-p1 amd64
>Organization:
The Bunker
>Environment:
System: FreeBSD obol.hosted-at.thebunker.net 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #3: Sun May 4 10:46:11 BST 2008 root at obol.hosted-at.thebunker.net:/usr/obj/usr/src/sys/OBOL amd64


	
>Description:

I have a new HP DL140G3 server runing RELENG_7_0 which has been stable up to now.
However the combination of configuring it as an IPSec tunnel end-point and then
turning on some Nagios monitoring via the tunnel causes the machine to crash within
a few minutes.  kgdb backtrace attached from the latest crash attached.

I'm using racoon from security/ipsec-tools for IKE -- I had tried previously using
security/isakmpd but in that case I found the process would run fine for maybe 20
minutes, then get into a loop where it chewed up lots of RAM very fast, until the
kernel killed it.

>How-To-Repeat:
	
>Fix:

	

--- kgdb.out begins here ---
Script started on Tue May  6 14:01:27 2008
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".

Unread portion of the kernel message buffer:


Fatal trap 9: general protection fault while in kernel mode
cpuid = 3; apic id = 03
instruction pointer	= 0x8:0xffffffff80706048
stack pointer	        = 0x10:0xffffffffae5cbf30
frame pointer	        = 0x10:0xffffff0001e1c300
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 779 (snmpd)
trap number		= 9
panic: general protection fault
cpuid = 3
GEOM_MIRROR: Device gm0: rebuilding provider da0 stopped.
Uptime: 7m23s
Physical memory: 2034 MB
Dumping 314 MB: 299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11

#0  doadump () at pcpu.h:194
194		__asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:194
#1  0x0000000000000004 in ?? ()
#2  0xffffffff8045b9cf in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#3  0xffffffff8045bdf8 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:563
#4  0xffffffff8071f8ca in trap_fatal (frame=0xffffff00014f8350, eva=18446742974219888848) at /usr/src/sys/amd64/amd64/trap.c:724
#5  0xffffffff80720388 in trap (frame=0xffffffffae5cbe80) at /usr/src/sys/amd64/amd64/trap.c:526
#6  0xffffffff8070738e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169
#7  0xffffffff80706048 in bus_dmamap_load_mbuf_sg (dmat=0xffffff00012a7a00, map=0x0, m0=Variable "m0" is not available.
) at /usr/src/sys/amd64/amd64/busdma_machdep.c:816
#8  0xffffffff80270a85 in bge_start_locked (ifp=0xffffff0001277000) at /usr/src/sys/dev/bge/if_bge.c:3390
#9  0xffffffff802714a7 in bge_start (ifp=0xffffff0001277000) at /usr/src/sys/dev/bge/if_bge.c:3572
#10 0xffffffff804e977e in ether_output_frame (ifp=0xffffff0001277000, m=0xffffff0001977100) at /usr/src/sys/net/if_ethersubr.c:405
#11 0xffffffff804e9cdf in ether_output (ifp=0xffffff0001277000, m=0xffffff0001977100, dst=Variable "dst" is not available.
) at /usr/src/sys/net/if_ethersubr.c:374
#12 0xffffffff805333e9 in ip_output (m=0xffffff0001977100, opt=Variable "opt" is not available.
) at /usr/src/sys/netinet/ip_output.c:583
#13 0xffffffff805bf747 in ipsec_process_done (m=0xffffff000177bc00, isr=0xffffff000174f800) at /usr/src/sys/netipsec/ipsec_output.c:177
#14 0xffffffff805cd8f8 in esp_output_cb (crp=0xffffff0001e24cb8) at /usr/src/sys/netipsec/xform_esp.c:965
#15 0xffffffff80606109 in crypto_done (crp=0xffffff0001e24cb8) at /usr/src/sys/opencrypto/crypto.c:1148
#16 0xffffffff8060934c in swcr_process (dev=Variable "dev" is not available.
) at /usr/src/sys/opencrypto/cryptosoft.c:975
#17 0xffffffff80606e89 in crypto_invoke (cap=Variable "cap" is not available.
) at cryptodev_if.h:53
#18 0xffffffff80607974 in crypto_dispatch (crp=0xffffff0001e24cb8) at /usr/src/sys/opencrypto/crypto.c:798
#19 0xffffffff805cdf91 in esp_output (m=0xffffff000161c360, isr=0xffffff000174f800, mp=Variable "mp" is not available.
) at /usr/src/sys/netipsec/xform_esp.c:875
#20 0xffffffff805bf95b in ipsec4_process_packet (m=0xffffff000177bc00, isr=0xffffff000174f800, flags=Variable "flags" is not available.
) at /usr/src/sys/netipsec/ipsec_output.c:486
#21 0xffffffff805312e7 in ip_ipsec_output (m=0xffffffffae5cc8b8, inp=0xffffff000168c360, flags=0xffffffffae5cc8ac, error=0xffffffffae5cc8f8, ro=Variable "ro" is not available.
) at /usr/src/sys/netinet/ip_ipsec.c:331
#22 0xffffffff80532814 in ip_output (m=0xffffff000177bc00, opt=Variable "opt" is not available.
) at /usr/src/sys/netinet/ip_output.c:418
#23 0xffffffff80594ab3 in udp_send (so=Variable "so" is not available.
) at /usr/src/sys/netinet/udp_usrreq.c:972
#24 0xffffffff804abb60 in sosend_dgram (so=0xffffff0001aadae0, addr=0xffffff000161c090, uio=Variable "uio" is not available.
) at /usr/src/sys/kern/uipc_socket.c:1053
#25 0xffffffff804af176 in kern_sendit (td=0xffffff00014f8350, s=11, mp=0xffffffffae5ccb10, flags=0, control=0x0, segflg=Variable "segflg" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:789
#26 0xffffffff804b1c6a in sendit (td=0xffffff00014f8350, s=11, mp=0xffffffffae5ccb10, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:730
#27 0xffffffff804b1d4a in sendto (td=Variable "td" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:841
#28 0xffffffff8071fedc in syscall (frame=0xffffffffae5ccc70) at /usr/src/sys/amd64/amd64/trap.c:852
#29 0xffffffff8070759b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:290
#30 0x00000008018d607c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) up
#1  0x0000000000000004 in ?? ()
(kgdb) up
#2  0xffffffff8045b9cf in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
409			doadump();
(kgdb) up
#3  0xffffffff8045bdf8 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:563
563		boot(bootopt);
(kgdb) up
#4  0xffffffff8071f8ca in trap_fatal (frame=0xffffff00014f8350, eva=18446742974219888848) at /usr/src/sys/amd64/amd64/trap.c:724
724			panic("%s", trap_msg[type]);
(kgdb) up
#5  0xffffffff80720388 in trap (frame=0xffffffffae5cbe80) at /usr/src/sys/amd64/amd64/trap.c:526
526			trap_fatal(frame, 0);
(kgdb) up
#6  0xffffffff8070738e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169
169		call	trap
Current language:  auto; currently asm
(kgdb) up
#7  0xffffffff80706048 in bus_dmamap_load_mbuf_sg (dmat=0xffffff00012a7a00, map=0x0, m0=Variable "m0" is not available.
) at /usr/src/sys/amd64/amd64/busdma_machdep.c:816
816				if (m->m_len > 0) {
Current language:  auto; currently c
(kgdb) list
811			int first = 1;
812			bus_addr_t lastaddr = 0;
813			struct mbuf *m;
814	
815			for (m = m0; m != NULL && error == 0; m = m->m_next) {
816				if (m->m_len > 0) {
817					error = _bus_dmamap_load_buffer(dmat, map,
818							m->m_data, m->m_len,
819							NULL, flags, &lastaddr,
820							segs, nsegs, first);
(kgdb) quit

Script done on Tue May  6 14:05:50 2008
--- kgdb.out ends here ---

--- dmesg.boot begins here ---
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-RELEASE-p1 #3: Sun May  4 10:46:11 BST 2008
    root at obol.hosted-at.thebunker.net:/usr/obj/usr/src/sys/OBOL
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz (1995.01-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6fb  Stepping = 11
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4
usable memory = 2133483520 (2034 MB)
avail memory  = 2058792960 (1963 MB)
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
hptrr: HPT RocketRAID controller driver v1.1 (May  4 2008 10:46:04)
acpi0: <PTLTD   RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib4
mpt0: <LSILogic SAS/SATA Adapter> port 0x2000-0x20ff mem 0xdc210000-0xdc213fff,0xdc200000-0xdc20ffff irq 24 at device 1.0 on pci7
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.14.0
mpt0: mpt_cam_event: 0x16
mpt0: mpt_cam_event: 0x16
mpt0: mpt_cam_event: 0x16
mpt0: mpt_cam_event: 0x12
mpt0: mpt_cam_event: 0x12
mpt0: mpt_cam_event: 0x16
mpt0: mpt_cam_event: 0x16
mpt0: mpt_cam_event: 0x16
pcib5: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci8: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci12: <ACPI PCI bus> on pcib6
pcib7: <PCI-PCI bridge> at device 5.0 on pci0
pci13: <PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci14: <ACPI PCI bus> on pcib8
pcib9: <PCI-PCI bridge> at device 7.0 on pci0
pci15: <PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci22: <ACPI PCI bus> on pcib10
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xdc300000-0xdc30ffff irq 16 at device 0.0 on pci22
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:1e:0b:5a:b2:e4
bge0: [ITHREAD]
pcib11: <ACPI PCI-PCI bridge> at device 28.1 on pci0
pci23: <ACPI PCI bus> on pcib11
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xdc400000-0xdc40ffff irq 17 at device 0.0 on pci23
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1: Ethernet address: 00:1e:0b:5a:b2:e5
bge1: [ITHREAD]
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 23 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 23 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 23 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xdc000000-0xdc0003ff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 63XXESB USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 6 ports with 6 removable, self powered
pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci24: <ACPI PCI bus> on pcib12
vgapci0: <VGA-compatible display> mem 0xde000000-0xdeffffff,0xdc500000-0xdc503fff,0xdc800000-0xdcffffff irq 17 at device 2.0 on pci24
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel 63XXESB2 SATA300 controller> port 0x1890-0x1897,0x1884-0x1887,0x1888-0x188f,0x1880-0x1883,0x1870-0x187f mem 0xdc000400-0xdc0007ff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
cryptosoft0: <software crypto> on motherboard
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xdc000-0xdffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ukbd0: <ServerEngines SE USB Device, class 0/0, rev 1.10/0.01, addr 2> on uhub2
kbd2 at ukbd0
ums0: <ServerEngines SE USB Device, class 0/0, rev 1.10/0.01, addr 2> on uhub2
ums0: 8 buttons and Z dir.
Timecounters tick every 1.000 msec
Fast IPsec: Initialized Security Association Processing.
hptrr: no controller detected.
acd0: CDRW <DW-224E-V/C.CA> at ata0-master UDMA33
da0 at mpt0 bus 0 target 1 lun 0
da0: <ATA FB160C4081 HPF0> Fixed Direct Access SCSI-5 device 
da0: 300.000MB/s transfers
da0: Command Queueing Enabled
da0: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C)
da1 at mpt0 bus 0 target 2 lun 0
da1: <ATA FB160C4081 HPF0> Fixed Direct Access SCSI-5 device 
da1: 300.000MB/s transfers
da1: Command Queueing Enabled
da1: 152627MB (312581808 512 byte sectors: 255H 63S/T 19457C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
GEOM_MIRROR: Device mirror/gm0 launched (1/2).
GEOM_MIRROR: Device gm0: rebuilding provider da0.
Trying to mount root from ufs:/dev/mirror/gm0s1a
WARNING: / was not properly dismounted
--- dmesg.boot ends here ---

--- OBOL begins here ---
#
# Kernel config for FreeBSD 7.0+ server
#

include GENERIC

ident OBOL

nooptions       SCHED_4BSD
options         SCHED_ULE

device crypto
device cryptodev

options IPSEC
options IPSEC_DEBUG

options ALTQ

#
# That's All Folks!
#
--- OBOL ends here ---


>Release-Note:
>Audit-Trail:
>Unformatted:

More information about the freebsd-bugs mailing list