Racoon(ipsec-tools) enters sbwait state or 100% CPU utilization quite often on RELENG_1_2

[Scott Ullrich]

On 8/17/07, Scott Ullrich <sullrich at gmail.com> wrote:
> Hello!
>
> We are trying to track down a problem that involves a large number of
> ipsec tunnels (in this case 80).  Frequently racoon (ipsec-tools
> 0.7rc1 and also 0.6) will deadlock into the sbwait state or will enter
> a 100% cpu usage state and will not recover without killing the
> process and restarting.
>
> # uname -a
> FreeBSD pfsense.geekgod.com 6.2-RELEASE-p7 FreeBSD 6.2-RELEASE-p7 #0:
> Sat Aug  4 18:35:24 EDT 2007
> sullrich at builder6.pfsense.com:/usr/obj.pfSense/usr/src/sys/pfSense.6
> i386
>
> Kernel configuration file:
> http://cvs.pfsense.com/cgi-bin/cvsweb.cgi/tools/builder_scripts/conf/pfSense.6?rev=1.65;content-type=text%2Fplain
>
> We where able to obtain a backtrace of what happens when the process
> enters a 100% tail-spin:
>
> $ more /root/racoon.20070817.2112.txt
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-marcel-freebsd"...
> Attaching to program: /usr/local/sbin/racoon, process 9144
> Reading symbols from /lib/libutil.so.5...done.
> Loaded symbols for /lib/libutil.so.5
> Reading symbols from /lib/libcrypto.so.4...done.
> Loaded symbols for /lib/libcrypto.so.4
> Reading symbols from /lib/libreadline.so.6...done.
> Loaded symbols for /lib/libreadline.so.6
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /lib/libncurses.so.6...done.
> Loaded symbols for /lib/libncurses.so.6
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> 0x2827a187 in recvfrom () from /lib/libc.so.6
> #0  0x2827a187 in recvfrom () from /lib/libc.so.6
> #1  0x28225904 in recv () from /lib/libc.so.6
> #2  0x0805f4f5 in pk_recv (so=11, lenp=0xbfbfe558) at pfkey.c:2826
> #3  0x0805f622 in pfkey_dump_sadb (satype=3) at pfkey.c:314
> #4  0x0805ac3d in purge_ipsec_spi (dst0=0x81b1080, proto=3, spi=0x8188140, n=1)
>     at isakmp_inf.c:1173
> #5  0x0805ba5c in isakmp_info_recv (iph1=0x81c1e00, msg0=0x1)
>     at isakmp_inf.c:565
> #6  0x0804ec49 in isakmp_main (msg=0x8218240, remote=0xbfbfe7f0,
>     local=0xbfbfe770) at isakmp.c:671
> #7  0x0805003e in isakmp_handler (so_isakmp=24) at isakmp.c:395
> #8  0x0804bf88 in session () at session.c:223
> #9  0x0804b901 in main (ac=0, av=0xbfbfee4c) at main.c:264
> #0  0x2827a187 in recvfrom () from /lib/libc.so.6
> #1  0x28225904 in recv () from /lib/libc.so.6
> #2  0x0805f4f5 in pk_recv (so=11, lenp=0xbfbfe558) at pfkey.c:2826
> #3  0x0805f622 in pfkey_dump_sadb (satype=3) at pfkey.c:314
> #4  0x0805ac3d in purge_ipsec_spi (dst0=0x81b1080, proto=3, spi=0x8188140, n=1)
>     at isakmp_inf.c:1173
> #5  0x0805ba5c in isakmp_info_recv (iph1=0x81c1e00, msg0=0x1)
>     at isakmp_inf.c:565
> #6  0x0804ec49 in isakmp_main (msg=0x8218240, remote=0xbfbfe7f0,
>     local=0xbfbfe770) at isakmp.c:671
> #7  0x0805003e in isakmp_handler (so_isakmp=24) at isakmp.c:395
> #8  0x0804bf88 in session () at session.c:223
> #9  0x0804b901 in main (ac=0, av=0xbfbfee4c) at main.c:264
>
> Does anyone know what we can look at further to try and eliminate the
> problem or does anyone have suggestions on how we can debug further?
>

Sorry, that title should have read RELENG_6_2.   Freudian slip.

Any help is appreciated.

Scott