misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail)

Wed Jan 1 13:30:01 UTC 2014

The following reply was made to PR misc/185092; it has been noted by GNATS.

From: Nikolay Denev <ndenev at gmail.com>
To: FreeBSD-gnats-submit at freebsd.org, freebsd-bugs at freebsd.org, 
	"freebsd-net at freebsd.org" <freebsd-net at freebsd.org>
Cc:  
Subject: Re: misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail)
Date: Wed, 1 Jan 2014 13:21:13 +0000

 --001a11c1ba200f4f8504eee88df5
 Content-Type: text/plain; charset=windows-1252
 Content-Transfer-Encoding: quoted-printable

 On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <ndenev at gmail.com> wrote:

 > On Sun, Dec 22, 2013 at 1:10 PM, <FreeBSD-gnats-submit at freebsd.org> wrote=
 :
 >
 >> Thank you very much for your problem report.
 >> It has the internal identification `misc/185092'.
 >> The individual assigned to look at your
 >> report is: freebsd-bugs.
 >>
 >> You can access the state of your problem report at any time
 >> via this link:
 >>
 >> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092
 >>
 >> >Category:       misc
 >> >Responsible:    freebsd-bugs
 >> >Synopsis:       panic: rtfree 2 (using RADIX_MPATH in a VNET jail)
 >> >Arrival-Date:   Sun Dec 22 13:10:00 UTC 2013
 >>
 >
 > I'm trying to understand exactly what is happening here, and examining a
 > core dump with kgdb I'm getting some output that confuses me :
 >
 > (kgdb) bt
 > #0  doadump (textdump=3D-1011569920) at pcpu.h:233
 > #1  0xc06069b2 in kern_reboot (howto=3D260) at
 > /usr/src/sys/kern/kern_shutdown.c:447
 > #2  0xc0606d0e in panic (fmt=3D<value optimized out>) at
 > /usr/src/sys/kern/kern_shutdown.c:754
 > #3  0xc06de639 in rtfree (rt=3D<value optimized out>) at
 > /usr/src/sys/net/route.c:464
 > #4  0xc06e188d in route_output (m=3D<value optimized out>) at
 > /usr/src/sys/net/rtsock.c:951
 > #5  0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<=
 value
 > optimized out>, nam=3D0x0, control=3D<value optimized out>,
 >     td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
 > #6  0xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D<value optimized
 > out>, top=3D<value optimized out>, control=3D0x0,
 >     flags=3D<value optimized out>, td=3D<value optimized out>) at
 > /usr/src/sys/kern/uipc_socket.c:1271
 > #7  0xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10,
 > top=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000)
 >     at /usr/src/sys/kern/uipc_socket.c:1315
 > #8  0xc0654af4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10,
 > active_cred=3D0xc3f1dd00, flags=3D0, td=3D0xc3bd2000)
 >     at /usr/src/sys/kern/sys_socket.c:103
 > #9  0xc064c866 in dofilewrite (td=3D0xc3bd2000, fd=3D3, fp=3D0xc3c0c818,
 > auio=3D0xd9b9cc10, offset=3D-1, flags=3D0) at file.h:303
 > #10 0xc064c566 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D<value opt=
 imized
 > out>) at /usr/src/sys/kern/sys_generic.c:467
 > #11 0xc064c4bc in sys_write (td=3D<value optimized out>, uap=3D<value
 > optimized out>) at /usr/src/sys/kern/sys_generic.c:382
 > #12 0xc08614d3 in syscall (frame=3D<value optimized out>) at
 > subr_syscall.c:134
 > #13 0xc084cca1 in Xint0x80_syscall () at
 > /usr/src/sys/i386/i386/exception.s:270
 > #14 0x281975b7 in ?? ()
 > Previous frame inner to this frame (corrupt stack?)
 > Current language:  auto; currently minimal
 > (kgdb) fr 3
 > #3  0xc06de639 in rtfree (rt=3D<value optimized out>) at
 > /usr/src/sys/net/route.c:464
 > 464 panic("rtfree 2");
 > (kgdb) print *rt
 > $1 =3D {rt_nodes =3D {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_bi=
 t =3D 0,
 > rn_bmask =3D 0 '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D {
 >           rn_Key =3D 0xc0882687 "shutdown_post_sync", rn_Mask =3D 0x10300=
 00
 > <Address 0x1030000 out of bounds>, rn_Dupedkey =3D 0x0}, rn_node =3D {
 >           rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x0}}},
 > {rn_mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =3D -9=
 4 '?',
 >       rn_flags =3D 195 '?', rn_u =3D {rn_leaf =3D {rn_Key =3D 0xc3a545e0 =
 "",
 > rn_Mask =3D 0xc3a4e440 " ??(???\020'", rn_Dupedkey =3D 0xc3a4e880},
 >         rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R =
 =3D
 > 0xc3a4e880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060,
 >   rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735f74, rt_ifa =3D 0x636e, rt_=
 rmx =3D
 > {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pksent =3D 0, rmx_weight =3D 0},
 >   rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flags=
  =3D 0,
 > lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}}
 >
 >
 >
 > rn_Key with value of =93shutdown_post_sync=94 ?
 >
 > It=92s visible also in the raw_usend() frame:
 >
 > (kgdb) fr 5
 > #5  0xc06de18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<=
 value
 > optimized out>, nam=3D0x0, control=3D<value optimized out>,
 >     td=3D0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
 > 238 return ((*so->so_proto->pr_output)(m, so));
 > (kgdb) print *m
 > $2 =3D {m_hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data =
 =3D 0x0,
 > mh_len =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0},
 >   M_dat =3D {MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first =
 =3D 0x4}, len =3D
 > -1012745856, flowid =3D 3282388448,
 >         csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosqos =3D=
  117
 > 'u', rsstype =3D 116 't', l2hlen =3D 100 'd', l3hlen =3D 111 'o',
 >         l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_per =3D {eigth =3D "_p=
 ost_sy",
 > sixteen =3D {28767, 29551, 24436, 31091}, thirtytwo =3D {1936683103,
 >             2037604212}, sixtyfour =3D {8751443454668533855}, unintptr =
 =3D
 > {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D {
 >           eigth =3D "nc\000\000\000\000\000", sixteen =3D {25454, 0, 0, 0=
 },
 > thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unintptr =3D {25454},
 >           ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext_=
 buf =3D
 > 0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0,
 >           ext_arg1 =3D 0x0, ext_arg2 =3D 0x0},
 >         MH_databuf =3D '\0' <repeats 56 times>, "file", '\0' <repeats 20
 > times>,
 > "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000=
 \000\000\000\004\000\000\000\000\000\000\00000Y?",
 > '\0' <repeats 12 times>,
 > "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??=
 \000\000\000\000\000???",
 > '\0' <repeats 23 times>}},
 >     M_databuf =3D
 > "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync",
 > '\0' <repeats 62 times>, "file", '\0' <repeats 20 times>,
 > "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000=
 \000\000\000\004\000\000\000\000\000\000\00000Y?",
 > '\0' <repeats 12 times>,
 > "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??=
 \000\000\000\000\000???",
 > '\0' <repeats 23 times>}}
 >
 >
 > This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfre=
 e
 > patch, which I  thought earlier might be the cause of the panics I'm
 > seeing).
 >
 > The machine is Soekris Net5501-70 with this kernel config :
 >
 > cpu I586_CPU
 > cpu I686_CPU
 > ident MARS
 > options CPU_GEODE
 > options     CPU_SOEKRIS
 >
 > options HZ=3D2000
 > options DEVICE_POLLING
 > options BPF_JITTER
 >
 > makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug symbols
 >
 > options SCHED_ULE # ULE scheduler
 > options PREEMPTION # Enable kernel thread preemption
 > options INET # InterNETworking
 > options INET6 # IPv6 communications protocols
 > options TCP_OFFLOAD # TCP offload
 > options FFS # Berkeley Fast Filesystem
 > options SOFTUPDATES # Enable FFS soft updates support
 > options UFS_DIRHASH # Improve performance on big directories
 > options PROCFS # Process filesystem (requires PSEUDOFS)
 > options PSEUDOFS # Pseudo-filesystem framework
 > options GEOM_PART_GPT # GUID Partition Tables.
 > options GEOM_LABEL # Provides labelization
 > options COMPAT_FREEBSD4 # Compatible with FreeBSD4
 > options COMPAT_FREEBSD5 # Compatible with FreeBSD5
 > options COMPAT_FREEBSD6 # Compatible with FreeBSD6
 > options COMPAT_FREEBSD7 # Compatible with FreeBSD7
 > options SCSI_DELAY=3D500 # Delay (in ms) before probing SCSI
 > options KTRACE # ktrace(1) support
 > options STACK # stack(9) support
 > options SYSVSHM # SYSV-style shared memory
 > options SYSVMSG # SYSV-style message queues
 > options SYSVSEM # SYSV-style semaphores
 > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
 > options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being interspersed=
 .
 > options KBD_INSTALL_CDEV # install a CDEV entry in /dev
 > options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4)
 > options CAPABILITY_MODE # Capsicum capability mode
 > options CAPABILITIES # Capsicum capabilities
 > options PROCDESC # Support for process descriptors
 > options INCLUDE_CONFIG_FILE     # Include this file in kernel
 >
 > # Debugging support.  Always need this:
 > options KDB # Enable kernel debugger support.
 > options KDB_TRACE # Print a stack trace for a panic.
 > options KDB_UNATTENDED
 >
 > options TEXTDUMP_PREFERRED
 > options TEXTDUMP_VERBOSE
 >
 > device pci
 > device ata # Legacy ATA/SATA controllers
 > options ATA_STATIC_ID # Static device numbering
 >
 > # ATA/SCSI peripherals
 > device scbus # SCSI bus (required for ATA/SCSI)
 > device da # Direct Access (disks)
 > device pass # Passthrough device (direct ATA/SCSI access)
 >
 > # Add suspend/resume support for the i8254.
 > device pmtimer
 >
 > # Serial (COM) ports
 > device uart # Generic UART driver
 >
 > device miibus # MII bus support
 > device vr # VIA Rhine, Rhine II
 >
 > # Wireless NIC cards
 > device wlan # 802.11 support
 > options IEEE80211_DEBUG # enable debug msgs
 > options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's
 > options IEEE80211_SUPPORT_MESH # enable 802.11s draft support
 > device wlan_wep # 802.11 WEP support
 > device wlan_ccmp # 802.11 CCMP support
 > device wlan_tkip # 802.11 TKIP support
 > device wlan_amrr # AMRR transmit rate control algorithm
 > device ath # Atheros NICs
 > device ath_pci # Atheros pci/cardbus glue
 > device ath_hal # pci/cardbus chip support
 > options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors
 > options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation
 > options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later
 > device ath_rate_sample # SampleRate tx rate control for ath
 >
 > # Pseudo devices.
 > device loop # Network loopback
 > device random # Entropy device
 > device ether # Ethernet support
 > device vlan # 802.1Q VLAN support
 > device tun # Packet tunnel.
 > device md # Memory "disks"
 > device gif # IPv6 and IPv4 tunneling
 > device      gre
 > device faith # IPv6-to-IPv4 relaying (translation)
 > device firmware # firmware assist module
 > device      if_bridge
 >
 > options     VIMAGE
 > options     ROUTETABLES=3D8
 > options     RADIX_MPATH
 >
 > options     SW_WATCHDOG
 >
 > device      crypto
 > device      cryptodev
 > device      glxsb
 >
 > options     BOOTVERBOSE=3D1
 >
 > #device      pf
 > #device      pflog
 > #device      pfsync
 > device       carp
 > device       enc
 > device       lagg
 > device       epair
 >
 > #options     ALTQ
 > #options     ALTQ_CBQ
 > #options     ALTQ_RED
 > #options     ALTQ_RIO
 > #options     ALTQ_HFSC
 > #options     ALTQ_PRIQ
 >
 > options     IPFIREWALL
 > options     IPFIREWALL_DEFAULT_TO_ACCEPT
 > options     IPFIREWALL_NAT
 > options     LIBALIAS
 > options     IPDIVERT
 > options     DUMMYNET
 >
 > device bpf # Berkeley packet filter
 >
 > # USB support
 > options USB_DEBUG # enable debug msgs
 > device uhci # UHCI PCI->USB interface
 > device ohci # OHCI PCI->USB interface
 > device ehci # EHCI PCI->USB interface (USB 2.0)
 > device usb     # USB Bus (required)
 > device umass # Disks/Mass storage - Requires scbus and da
 >
 >
 > Also src.conf and make.conf :
 >
 > root at vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf
 > WITHOUT_ACCT=3Dyes
 > WITHOUT_ACPI=3Dyes
 > WITHOUT_AMD=3Dyes
 > WITHOUT_APM=3Dyes
 > WITHOUT_ASSERT_DEBUG=3Dyes
 > WITHOUT_AT=3Dyes
 > WITHOUT_ATF=3Dyes
 > WITHOUT_ATM=3Dyes
 > WITHOUT_AUDIT=3Dyes
 > WITHOUT_BLUETOOTH=3Dyes
 > WITHOUT_CALENDAR=3Dyes
 > WITHOUT_CDDL=3Dyes
 > WITHOUT_CTM=3Dyes
 > WITHOUT_DICT=3Dyes
 > WITHOUT_FLOPPY=3Dyes
 > WITHOUT_GAMES=3Dyes
 > WITHOUT_HTML=3Dyes
 > WITHOUT_INFO=3Dyes
 > WITHOUT_IPFILTER=3Dyes
 > WITHOUT_IPX=3Dyes
 > #WITHOUT_KERNEL_SYMBOLS=3Dyes
 > WITHOUT_LEGACY_CONSOLE=3Dyes
 > WITHOUT_LOCALES=3Dyes
 > WITHOUT_LPR=3Dyes
 > WITHOUT_MAIL=3Dyes
 > WITHOUT_NDIS=3Dyes
 > WITHOUT_QUOTAS=3Dyes
 > WITHOUT_ROUTED=3Dyes
 > WITHOUT_SENDMAIL=3Dyes
 > WITH_SVN=3Dyes
 > WITHOUT_ZFS=3Dyes
 >
 > root at vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf
 > CFLAGS=3D-O2
 > COPTFLAGS=3D -O -pipe
 > CPUTYPE=3Dgeode
 > KERNCONF=3DMARS
 > NO_MODULES=3Dyes
 > BOOTWAIT=3D0
 > DOC_LANG=3Den_US.ISO8859-1
 >
 >
 >
 > --Nikolay
 >
 >
 Also, originally I thought that the panic is when a multi path route is
 being deleted, however again from the coredump it seems that the panic
 happens when openvpn deletes the host route it installs for the remote
 openvpn server pointed to the default gw (before openvpn installs the new
 default gw pointing to the vpn tunnel) :

 (kgdb) fr 12
 (kgdb) x/12sb td->td_proc->p_args
 0xc4269780:  "\001"
 0xc4269782:  ""
 0xc4269783:  ""
 0xc4269784:  "B"
 0xc4269786:  ""
 0xc4269787:  ""
 0xc4269788:  "/sbin/route"
 0xc4269794:  "delete"
 0xc426979b:  "-net"
 0xc42697a0:  "78.90.222.xxx"
 0xc42697ad:  "10.255.255.0"
 0xc42697ba:  "255.255.255.255"
 (kgdb)

 I'm trying to reproduce this on a VirtualBox instance now, however so far
 no luck (no OpenVPN running, just adding and removing routes).

 --Nikolay

 --001a11c1ba200f4f8504eee88df5
 Content-Type: text/html; charset=windows-1252
 Content-Transfer-Encoding: quoted-printable

 <div dir=3D"ltr">On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <span dir=3D=
 "ltr"><<a href=3D"mailto:ndenev at gmail.com" target=3D"_blank">ndenev at gmai=
 l.com</a>></span> wrote:<br><div class=3D"gmail_extra"><div class=3D"gma=
 il_quote">
 <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
 left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
 adding-left:1ex"><div dir=3D"ltr"><div><div class=3D"h5">On Sun, Dec 22, 20=
 13 at 1:10 PM,  <span dir=3D"ltr"><<a href=3D"mailto:FreeBSD-gnats-submi=
 t at freebsd.org" target=3D"_blank">FreeBSD-gnats-submit at freebsd.org</a>></=
 span> wrote:<br>
 <div class=3D"gmail_extra"><div class=3D"gmail_quote">
 <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
 left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
 adding-left:1ex">Thank you very much for your problem report.<br>
 It has the internal identification `misc/185092'.<br>
 The individual assigned to look at your<br>
 report is: freebsd-bugs.<br>
 <br>
 You can access the state of your problem report at any time<br>
 via this link:<br>
 <br>
 <a href=3D"http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092" target=3D"_=
 blank">http://www.freebsd.org/cgi/query-pr.cgi?pr=3D185092</a><br>
 <br>
 >Category: =A0 =A0 =A0 misc<br>
 >Responsible: =A0 =A0freebsd-bugs<br>
 >Synopsis: =A0 =A0 =A0 panic: rtfree 2 (using RADIX_MPATH in a VNET jail=
 )<br>
 >Arrival-Date: =A0 Sun Dec 22 13:10:00 UTC 2013<br>
 </blockquote></div><br></div></div></div><div class=3D"gmail_extra">I'm=
  trying to understand exactly what is happening here, and examining a core =
 dump with kgdb I'm getting some output that confuses me :</div><div cla=
 ss=3D"gmail_extra">

 <br></div><blockquote style=3D"margin:0px 0px 0px 40px;border:none;padding:=
 0px"><div class=3D"gmail_extra"><div class=3D"gmail_extra">(kgdb) bt</div><=
 /div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#0 =A0doadump (t=
 extdump=3D-1011569920) at pcpu.h:233</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#1 =A00xc06069b=
 2 in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:447</di=
 v></div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#2 =A00xc0606=
 d0e in panic (fmt=3D<value optimized out>) at /usr/src/sys/kern/kern_=
 shutdown.c:754</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#3 =A00xc06de63=
 9 in rtfree (rt=3D<value optimized out>) at /usr/src/sys/net/route.c:=
 464</div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#4 =A0=
 0xc06e188d in route_output (m=3D<value optimized out>) at /usr/src/sy=
 s/net/rtsock.c:951</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#5 =A00xc06de18=
 f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<value =
 optimized out>, nam=3D0x0, control=3D<value optimized out>,=A0</di=
 v></div><div class=3D"gmail_extra">

 <div class=3D"gmail_extra">=A0 =A0 td=3D0xc3bd2000) at /usr/src/sys/net/raw=
 _usrreq.c:238</div></div><div class=3D"gmail_extra"><div class=3D"gmail_ext=
 ra">#6 =A00xc066eca9 in sosend_generic (so=3D0xc3e9c1a8, uio=3D<value op=
 timized out>, top=3D<value optimized out>, control=3D0x0,=A0</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">=A0 =A0 flags=
 =3D<value optimized out>, td=3D<value optimized out>) at /usr/s=
 rc/sys/kern/uipc_socket.c:1271</div></div><div class=3D"gmail_extra"><div c=
 lass=3D"gmail_extra">

 #7 =A00xc066efc7 in sosend (so=3D0xc3e9c1a8, addr=3D0x0, uio=3D0xd9b9cc10, =
 top=3D0x0, control=3D0x0, flags=3D0, td=3D0xc3bd2000)</div></div><div class=
 =3D"gmail_extra"><div class=3D"gmail_extra">=A0 =A0 at /usr/src/sys/kern/ui=
 pc_socket.c:1315</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#8 =A00xc0654af=
 4 in soo_write (fp=3D0xc3c0c818, uio=3D0xd9b9cc10, active_cred=3D0xc3f1dd00=
 , flags=3D0, td=3D0xc3bd2000)</div></div><div class=3D"gmail_extra"><div cl=
 ass=3D"gmail_extra">

 =A0 =A0 at /usr/src/sys/kern/sys_socket.c:103</div></div><div class=3D"gmai=
 l_extra"><div class=3D"gmail_extra">#9 =A00xc064c866 in dofilewrite (td=3D0=
 xc3bd2000, fd=3D3, fp=3D0xc3c0c818, auio=3D0xd9b9cc10, offset=3D-1, flags=
 =3D0) at file.h:303</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#10 0xc064c566 =
 in kern_writev (td=3D0xc3bd2000, fd=3D3, auio=3D<value optimized out>=
 ) at /usr/src/sys/kern/sys_generic.c:467</div></div><div class=3D"gmail_ext=
 ra"><div class=3D"gmail_extra">

 #11 0xc064c4bc in sys_write (td=3D<value optimized out>, uap=3D<va=
 lue optimized out>) at /usr/src/sys/kern/sys_generic.c:382</div></div><d=
 iv class=3D"gmail_extra"><div class=3D"gmail_extra">#12 0xc08614d3 in sysca=
 ll (frame=3D<value optimized out>) at subr_syscall.c:134</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#13 0xc084cca1 =
 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:270</div></div=
 ><div class=3D"gmail_extra"><div class=3D"gmail_extra">#14 0x281975b7 in ??=
  ()</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">Previous frame =
 inner to this frame (corrupt stack?)</div></div><div class=3D"gmail_extra">=
 <div class=3D"gmail_extra">Current language: =A0auto; currently minimal</di=
 v></div>

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">(kgdb) fr 3</div></di=
 v><div class=3D"gmail_extra"><div class=3D"gmail_extra">#3 =A00xc06de639 in=
  rtfree (rt=3D<value optimized out>) at /usr/src/sys/net/route.c:464<=
 /div></div>

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">464<span style=3D"whi=
 te-space:pre-wrap">				</span>panic("rtfree 2");</div></div><div =
 class=3D"gmail_extra"><div class=3D"gmail_extra">(kgdb) print *rt</div></di=
 v>
 <div class=3D"gmail_extra"><div class=3D"gmail_extra">$1 =3D {rt_nodes =3D =
 {{rn_mklist =3D 0xc3b4ab30, rn_parent =3D 0x1, rn_bit =3D 0, rn_bmask =3D 0=
  '\0', rn_flags =3D 0 '\0', rn_u =3D {rn_leaf =3D {</div></=
 div><div class=3D"gmail_extra">

 <div class=3D"gmail_extra">=A0 =A0 =A0 =A0 =A0 rn_Key =3D 0xc0882687 "=
 shutdown_post_sync", rn_Mask =3D 0x1030000 <Address 0x1030000 out o=
 f bounds>, rn_Dupedkey =3D 0x0}, rn_node =3D {</div></div><div class=3D"=
 gmail_extra"><div class=3D"gmail_extra">

 =A0 =A0 =A0 =A0 =A0 rn_Off =3D -1064819065, rn_L =3D 0x1030000, rn_R =3D 0x=
 0}}}, {rn_mklist =3D 0x0, rn_parent =3D 0x4, rn_bit =3D -18048, rn_bmask =
 =3D -94 '?',=A0</div></div><div class=3D"gmail_extra"><div class=3D=
 "gmail_extra">=A0 =A0 =A0 rn_flags =3D 195 '?', rn_u =3D {rn_leaf =
 =3D {rn_Key =3D 0xc3a545e0 "", rn_Mask =3D 0xc3a4e440 " ??(?=
 ??\020'", rn_Dupedkey =3D 0xc3a4e880},=A0</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">=A0 =A0 =A0 =A0=
  rn_node =3D {rn_Off =3D -1012578848, rn_L =3D 0xc3a4e440, rn_R =3D 0xc3a4e=
 880}}}}, rt_gateway =3D 0x74756873, rt_flags =3D 1853321060,=A0</div></div>=
 <div class=3D"gmail_extra">

 <div class=3D"gmail_extra">=A0 rt_refcnt =3D 1936683103, rt_ifp =3D 0x79735=
 f74, rt_ifa =3D 0x636e, rt_rmx =3D {rmx_mtu =3D 0, rmx_expire =3D 0, rmx_pk=
 sent =3D 0, rmx_weight =3D 0},=A0</div></div><div class=3D"gmail_extra"><di=
 v class=3D"gmail_extra">

 =A0 rt_fibnum =3D 0, rt_mtx =3D {lock_object =3D {lo_name =3D 0x0, lo_flags=
  =3D 0, lo_data =3D 0, lo_witness =3D 0x0}, mtx_lock =3D 0}}</div></div></b=
 lockquote><div class=3D"gmail_extra"><div class=3D"gmail_extra"><br></div><=
 div class=3D"gmail_extra">

 <br></div><div class=3D"gmail_extra">rn_Key with value of =93shutdown_post_=
 sync=94 ?</div><div class=3D"gmail_extra"><br></div><div class=3D"gmail_ext=
 ra">It=92s visible also in the raw_usend() frame:</div><div class=3D"gmail_=
 extra"><br>

 </div></div><blockquote style=3D"margin:0px 0px 0px 40px;border:none;paddin=
 g:0px"><div class=3D"gmail_extra"><div class=3D"gmail_extra">(kgdb) fr 5</d=
 iv></div><div class=3D"gmail_extra"><div class=3D"gmail_extra">#5 =A00xc06d=
 e18f in raw_usend (so=3D<value optimized out>, flags=3D0, m=3D<val=
 ue optimized out>, nam=3D0x0, control=3D<value optimized out>,=A0<=
 /div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">=A0 =A0 td=3D0x=
 c3bd2000) at /usr/src/sys/net/raw_usrreq.c:238</div></div><div class=3D"gma=
 il_extra"><div class=3D"gmail_extra">238<span style=3D"white-space:pre-wrap=
 ">		</span>return ((*so->so_proto->pr_output)(m, so));</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">(kgdb) print *m=
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra">$2 =3D {m=
 _hdr =3D {mh_next =3D 0xc3b4ab30, mh_nextpkt =3D 0x1, mh_data =3D 0x0, mh_l=
 en =3D -1064819065, mh_type =3D 0, mh_flags =3D 66304, mh_pad =3D 0},=A0</d=
 iv>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">=A0 M_dat =3D {=
 MH =3D {MH_pkthdr =3D {rcvif =3D 0x0, tags =3D {slh_first =3D 0x4}, len =3D=
  -1012745856, flowid =3D 3282388448,=A0</div></div><div class=3D"gmail_extr=
 a"><div class=3D"gmail_extra">

 =A0 =A0 =A0 =A0 csum_flags =3D 14097648373312316480, fibnum =3D 26739, cosq=
 os =3D 117 'u', rsstype =3D 116 't', l2hlen =3D 100 'd&=
 #39;, l3hlen =3D 111 'o',=A0</div></div><div class=3D"gmail_extra">=
 <div class=3D"gmail_extra">

 =A0 =A0 =A0 =A0 l4hlen =3D 119 'w', l5hlen =3D 110 'n', PH_=
 per =3D {eigth =3D "_post_sy", sixteen =3D {28767, 29551, 24436, =
 31091}, thirtytwo =3D {1936683103,=A0</div></div><div class=3D"gmail_extra"=
 ><div class=3D"gmail_extra">

 =A0 =A0 =A0 =A0 =A0 =A0 <a href=3D"tel:2037604212" value=3D"+12037604212" t=
 arget=3D"_blank">2037604212</a>}, sixtyfour =3D {8751443454668533855}, unin=
 tptr =3D {1936683103}, ptr =3D 0x736f705f}, PH_loc =3D {</div></div><div cl=
 ass=3D"gmail_extra"><div class=3D"gmail_extra">
 =A0 =A0 =A0 =A0 =A0 eigth =3D "nc\000\000\000\000\000", sixteen =
 =3D {25454, 0, 0, 0}, thirtytwo =3D {25454, 0}, sixtyfour =3D {25454}, unin=
 tptr =3D {25454},=A0</div>
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">=A0 =A0 =A0 =A0=
  =A0 ptr =3D 0x636e}}, MH_dat =3D {MH_ext =3D {ref_cnt =3D 0x0, ext_buf =3D=
  0x0, ext_size =3D 0, ext_type =3D 0, ext_flags =3D 0, ext_free =3D 0,=A0</=
 div></div><div class=3D"gmail_extra">

 <div class=3D"gmail_extra">=A0 =A0 =A0 =A0 =A0 ext_arg1 =3D 0x0, ext_arg2 =
 =3D 0x0},=A0</div></div><div class=3D"gmail_extra"><div class=3D"gmail_extr=
 a">=A0 =A0 =A0 =A0 MH_databuf =3D '\0' <repeats 56 times>, &q=
 uot;file", '\0' <repeats 20 times>, "\006\000\000\0=
 00\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\=
 000\000\000\000\000\000\00000Y?", '\0' <repeats 12 times&gt=
 ;, "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000=
 \203??\000\000\000\000\000???", '\0' <repeats 23 times>}=
 },=A0</div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra">=A0 =A0 M_datab=
 uf =3D "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_p=
 ost_sync", '\0' <repeats 62 times>, "file", &#=
 39;\0' <repeats 20 times>, "\006\000\000\000\020\000\000\000=
 ??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\000\000\000\000\00=
 0\000\00000Y?", '\0' <repeats 12 times>, "`2Y?\000\=
 000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\000\000\000=
 \000\000???", '\0' <repeats 23 times>}}</div>

 </div></blockquote><div class=3D"gmail_extra"><div class=3D"gmail_extra"><b=
 r></div><div class=3D"gmail_extra">This is 10.0-PRERELEASE r259547M (with a=
 pplied the recent nd6_nbr.c rtfree patch, which I =A0thought earlier might =
 be the cause of the panics I'm seeing).</div>

 <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">The machine=
  is Soekris Net5501-70 with this kernel config :</div><div class=3D"gmail_e=
 xtra"><br></div></div><blockquote style=3D"margin:0px 0px 0px 40px;border:n=
 one;padding:0px">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra">cpu<span style=3D"white-space:pre-wrap">		</span>I586_CPU</div></div>=
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">

 cpu<span style=3D"white-space:pre-wrap">		</span>I686_CPU</div></div></div>=
 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra">ident<span style=3D"white-space:pre-wrap">		</span>MARS</div></div>
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">options <span style=3D"white-space:pre-wrap">	</span>CPU_GEODE<=
 /div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div=
  class=3D"gmail_extra">

 options =A0 =A0 CPU_SOEKRIS</div></div></div><div class=3D"gmail_extra"><di=
 v class=3D"gmail_extra"><div class=3D"gmail_extra"><br></div></div></div><d=
 iv class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_ext=
 ra">options<span style=3D"white-space:pre-wrap">		</span>HZ=3D2000</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options<span style=3D"white-space:pre-wrap">		</span>DEVI=
 CE_POLLING</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_=
 extra">

 <div class=3D"gmail_extra">options<span style=3D"white-space:pre-wrap">		</=
 span>BPF_JITTER</div></div></div><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra"><div class=3D"gmail_extra"><br></div></div></div><div class=3D"=
 gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">makeoptions<span styl=
 e=3D"white-space:pre-wrap">	</span>DEBUG=3D-g<span style=3D"white-space:pre=
 -wrap">		</span># Build kernel with gdb(1) debug symbols</div></div></div>
 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra"><br></div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_=
 extra"><div class=3D"gmail_extra">options <span style=3D"white-space:pre-wr=
 ap">	</span>SCHED_ULE<span style=3D"white-space:pre-wrap">		</span># ULE sc=
 heduler</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>PREE=
 MPTION<span style=3D"white-space:pre-wrap">		</span># Enable kernel thread =
 preemption</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>INET=
 <span style=3D"white-space:pre-wrap">			</span># InterNETworking</div></div=
 >
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">options <span style=3D"white-space:pre-wrap">	</span>INET6<span=
  style=3D"white-space:pre-wrap">			</span># IPv6 communications protocols</=
 div>
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>TCP_=
 OFFLOAD<span style=3D"white-space:pre-wrap">		</span># TCP offload</div></d=
 iv>
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">options <span style=3D"white-space:pre-wrap">	</span>FFS<span s=
 tyle=3D"white-space:pre-wrap">			</span># Berkeley Fast Filesystem</div></d=
 iv>
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">options <span style=3D"white-space:pre-wrap">	</span>SOFTUPDATE=
 S<span style=3D"white-space:pre-wrap">		</span># Enable FFS soft updates su=
 pport</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>UFS_=
 DIRHASH<span style=3D"white-space:pre-wrap">		</span># Improve performance =
 on big directories</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>PROC=
 FS<span style=3D"white-space:pre-wrap">			</span># Process filesystem (requ=
 ires PSEUDOFS)</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>PSEU=
 DOFS<span style=3D"white-space:pre-wrap">		</span># Pseudo-filesystem frame=
 work</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>GEOM=
 _PART_GPT<span style=3D"white-space:pre-wrap">		</span># GUID Partition Tab=
 les.</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>GEOM=
 _LABEL<span style=3D"white-space:pre-wrap">		</span># Provides labelization=
 </div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>COMP=
 AT_FREEBSD4<span style=3D"white-space:pre-wrap">		</span># Compatible with =
 FreeBSD4</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>COMP=
 AT_FREEBSD5<span style=3D"white-space:pre-wrap">		</span># Compatible with =
 FreeBSD5</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>COMP=
 AT_FREEBSD6<span style=3D"white-space:pre-wrap">		</span># Compatible with =
 FreeBSD6</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>COMP=
 AT_FREEBSD7<span style=3D"white-space:pre-wrap">		</span># Compatible with =
 FreeBSD7</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>SCSI=
 _DELAY=3D500<span style=3D"white-space:pre-wrap">		</span># Delay (in ms) b=
 efore probing SCSI</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>KTRA=
 CE<span style=3D"white-space:pre-wrap">			</span># ktrace(1) support</div>
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>STAC=
 K<span style=3D"white-space:pre-wrap">			</span># stack(9) support</div></d=
 iv>
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">options <span style=3D"white-space:pre-wrap">	</span>SYSVSHM<sp=
 an style=3D"white-space:pre-wrap">			</span># SYSV-style shared memory</div=
 >
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>SYSV=
 MSG<span style=3D"white-space:pre-wrap">			</span># SYSV-style message queu=
 es</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>SYSV=
 SEM<span style=3D"white-space:pre-wrap">			</span># SYSV-style semaphores</=
 div>
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>_KPO=
 SIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions</div></div></=
 div>

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra">options <span style=3D"white-space:pre-wrap">	</span>PRINTF_BUFR_SIZE=
 =3D128<span style=3D"white-space:pre-wrap">	</span># Prevent printf output =
 being interspersed.</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>KBD_=
 INSTALL_CDEV<span style=3D"white-space:pre-wrap">	</span># install a CDEV e=
 ntry in /dev</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>HWPM=
 C_HOOKS<span style=3D"white-space:pre-wrap">		</span># Necessary kernel hoo=
 ks for hwpmc(4)</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>CAPA=
 BILITY_MODE<span style=3D"white-space:pre-wrap">		</span># Capsicum capabil=
 ity mode</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>CAPA=
 BILITIES<span style=3D"white-space:pre-wrap">		</span># Capsicum capabiliti=
 es</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>PROC=
 DESC<span style=3D"white-space:pre-wrap">		</span># Support for process des=
 criptors</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>INCL=
 UDE_CONFIG_FILE =A0 =A0 # Include this file in kernel</div></div></div><div=
  class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><br></div></div></div=
 ><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_=
 extra"># Debugging support. =A0Always need this:</div></div></div><div clas=
 s=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">options <span style=
 =3D"white-space:pre-wrap">	</span>KDB<span style=3D"white-space:pre-wrap">	=
 		</span># Enable kernel debugger support.</div></div></div><div class=3D"g=
 mail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">options <span style=
 =3D"white-space:pre-wrap">	</span>KDB_TRACE<span style=3D"white-space:pre-w=
 rap">		</span># Print a stack trace for a panic.</div></div></div><div clas=
 s=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">options<span style=3D=
 "white-space:pre-wrap">		</span>KDB_UNATTENDED</div></div></div><div class=
 =3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra"><br>=
 </div>
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options<span style=3D"white-space:pre-wrap">		</span>TEXT=
 DUMP_PREFERRED</div></div></div><div class=3D"gmail_extra"><div class=3D"gm=
 ail_extra">

 <div class=3D"gmail_extra">options<span style=3D"white-space:pre-wrap">		</=
 span>TEXTDUMP_VERBOSE</div></div></div><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra"><div class=3D"gmail_extra"><br></div></div></div><div cla=
 ss=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>pci</div></div></div><div class=3D"gmail_ext=
 ra"><div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=
 =3D"white-space:pre-wrap">		</span>ata<span style=3D"white-space:pre-wrap">=
 		</span># Legacy ATA/SATA controllers</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</span>ATA_=
 STATIC_ID<span style=3D"white-space:pre-wrap">	</span># Static device numbe=
 ring</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra"><br></div></div></div><div class=3D"gmail_extra"><div cla=
 ss=3D"gmail_extra"><div class=3D"gmail_extra"># ATA/SCSI peripherals</div><=
 /div></div>

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra">device<span style=3D"white-space:pre-wrap">		</span>scbus<span style=
 =3D"white-space:pre-wrap">		</span># SCSI bus (required for ATA/SCSI)</div>
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</span>da<sp=
 an style=3D"white-space:pre-wrap">		</span># Direct Access (disks)</div></d=
 iv>
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">device<span style=3D"white-space:pre-wrap">		</span>pass<span s=
 tyle=3D"white-space:pre-wrap">		</span># Passthrough device (direct ATA/SCS=
 I access)</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra"><br></div></div></div><div class=3D"gmail_extra"><div cla=
 ss=3D"gmail_extra"><div class=3D"gmail_extra"># Add suspend/resume support =
 for the i8254.</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</span>pmtim=
 er</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><=
 div class=3D"gmail_extra">

 <br></div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"=
 ><div class=3D"gmail_extra"># Serial (COM) ports</div></div></div><div clas=
 s=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra">dev=
 ice<span style=3D"white-space:pre-wrap">		</span>uart<span style=3D"white-s=
 pace:pre-wrap">		</span># Generic UART driver</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra"><br></div></div></div><div class=3D"gmail_extra"><div cla=
 ss=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"white-sp=
 ace:pre-wrap">		</span>miibus<span style=3D"white-space:pre-wrap">		</span>=
 # MII bus support</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</span>vr<sp=
 an style=3D"white-space:pre-wrap">		</span># VIA Rhine, Rhine II</div></div=
 >
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra"><br></div></div></div><div class=3D"gmail_extra"><div class=3D"=
 gmail_extra"><div class=3D"gmail_extra"># Wireless NIC cards</div></div></d=
 iv><div class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>wlan<span style=3D"white-space:pre-wrap">		<=
 /span># 802.11 support</div></div></div><div class=3D"gmail_extra"><div cla=
 ss=3D"gmail_extra">

 <div class=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</=
 span>IEEE80211_DEBUG<span style=3D"white-space:pre-wrap">	</span># enable d=
 ebug msgs</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra">

 <div class=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</=
 span>IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's</div></div><=
 /div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gm=
 ail_extra">

 options <span style=3D"white-space:pre-wrap">	</span>IEEE80211_SUPPORT_MESH=
 <span style=3D"white-space:pre-wrap">	</span># enable 802.11s draft support=
 </div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra">
 <div class=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</s=
 pan>wlan_wep<span style=3D"white-space:pre-wrap">	</span># 802.11 WEP suppo=
 rt</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra">
 <div class=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</s=
 pan>wlan_ccmp<span style=3D"white-space:pre-wrap">	</span># 802.11 CCMP sup=
 port</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"=
 >
 <div class=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</s=
 pan>wlan_tkip<span style=3D"white-space:pre-wrap">	</span># 802.11 TKIP sup=
 port</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"=
 >
 <div class=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</s=
 pan>wlan_amrr<span style=3D"white-space:pre-wrap">	</span># AMRR transmit r=
 ate control algorithm</div></div></div><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">

 <div class=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</s=
 pan>ath<span style=3D"white-space:pre-wrap">		</span># Atheros NICs</div></=
 div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=
 =3D"gmail_extra">

 device<span style=3D"white-space:pre-wrap">		</span>ath_pci<span style=3D"w=
 hite-space:pre-wrap">		</span># Atheros pci/cardbus glue</div></div></div><=
 div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_ex=
 tra">

 device<span style=3D"white-space:pre-wrap">		</span>ath_hal<span style=3D"w=
 hite-space:pre-wrap">		</span># pci/cardbus chip support</div></div></div><=
 div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_ex=
 tra">

 options <span style=3D"white-space:pre-wrap">	</span>AH_SUPPORT_AR5416<span=
  style=3D"white-space:pre-wrap">	</span># enable AR5416 tx/rx descriptors</=
 div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div =
 class=3D"gmail_extra">

 options <span style=3D"white-space:pre-wrap">	</span>AH_AR5416_INTERRUPT_MI=
 TIGATION<span style=3D"white-space:pre-wrap">	</span># AR5416 interrupt mit=
 igation</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_ext=
 ra">

 <div class=3D"gmail_extra">options <span style=3D"white-space:pre-wrap">	</=
 span>ATH_ENABLE_11N<span style=3D"white-space:pre-wrap">	</span># Enable 80=
 2.11n support for AR5416 and later</div></div></div><div class=3D"gmail_ext=
 ra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>ath_rate_sample<span style=3D"white-space:pr=
 e-wrap">	</span># SampleRate tx rate control for ath</div></div></div><div =
 class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><br></div></div></div=
 ><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_=
 extra"># Pseudo devices.</div></div></div><div class=3D"gmail_extra"><div c=
 lass=3D"gmail_extra">

 <div class=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</s=
 pan>loop<span style=3D"white-space:pre-wrap">		</span># Network loopback</d=
 iv></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div c=
 lass=3D"gmail_extra">

 device<span style=3D"white-space:pre-wrap">		</span>random<span style=3D"wh=
 ite-space:pre-wrap">		</span># Entropy device</div></div></div><div class=
 =3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra">devi=
 ce<span style=3D"white-space:pre-wrap">		</span>ether<span style=3D"white-s=
 pace:pre-wrap">		</span># Ethernet support</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</span>vlan<=
 span style=3D"white-space:pre-wrap">		</span># 802.1Q VLAN support</div></d=
 iv>
 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">device<span style=3D"white-space:pre-wrap">		</span>tun<span st=
 yle=3D"white-space:pre-wrap">		</span># Packet tunnel.</div></div></div><di=
 v class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>md<span style=3D"white-space:pre-wrap">		</s=
 pan># Memory "disks"</div></div></div><div class=3D"gmail_extra">
 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>gif<span style=3D"white-space:pre-wrap">		</=
 span># IPv6 and IPv4 tunneling</div></div></div><div class=3D"gmail_extra">
 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device =A0 =A0 =A0gre=
 </div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><di=
 v class=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</span=
 >faith<span style=3D"white-space:pre-wrap">		</span># IPv6-to-IPv4 relaying=
  (translation)</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">device<span style=3D"white-space:pre-wrap">		</span>firmw=
 are<span style=3D"white-space:pre-wrap">	</span># firmware assist module</d=
 iv>
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">device =A0 =A0 =A0if_bridge</div></div></div><div class=
 =3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra"><br>=
 </div></div></div>

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra">options =A0 =A0 VIMAGE</div></div></div><div class=3D"gmail_extra"><d=
 iv class=3D"gmail_extra"><div class=3D"gmail_extra">options =A0 =A0 ROUTETA=
 BLES=3D8</div></div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">options =A0 =A0 RADIX_MPATH</div></div></div><div class=3D"gmai=
 l_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra"><br></div></=
 div></div><div class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">options =A0 =A0 SW_WA=
 TCHDOG</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extr=
 a"><div class=3D"gmail_extra"><br></div></div></div><div class=3D"gmail_ext=
 ra"><div class=3D"gmail_extra">

 <div class=3D"gmail_extra">device =A0 =A0 =A0crypto</div></div></div><div c=
 lass=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra">=
 device =A0 =A0 =A0cryptodev</div></div></div><div class=3D"gmail_extra"><di=
 v class=3D"gmail_extra">

 <div class=3D"gmail_extra">device =A0 =A0 =A0glxsb</div></div></div><div cl=
 ass=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra"><=
 br></div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra">=
 <div class=3D"gmail_extra">

 options =A0 =A0 BOOTVERBOSE=3D1</div></div></div><div class=3D"gmail_extra"=
 ><div class=3D"gmail_extra"><div class=3D"gmail_extra"><br></div></div></di=
 v><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail=
 _extra">#device =A0 =A0 =A0pf</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">#device =A0 =A0 =A0pflog</div></div></div><div class=3D"g=
 mail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra">#device =
 =A0 =A0 =A0pfsync</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra">device =A0 =A0 =A0 carp</div></div></div><div class=3D"gm=
 ail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra">device =A0=
  =A0 =A0 enc</div></div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra">device =A0 =A0 =A0 lagg</div></div></div><div class=3D"gmail_ex=
 tra"><div class=3D"gmail_extra"><div class=3D"gmail_extra">device =A0 =A0 =
 =A0 epair</div></div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"g=
 mail_extra"><br></div></div></div><div class=3D"gmail_extra"><div class=3D"=
 gmail_extra"><div class=3D"gmail_extra">#options =A0 =A0 ALTQ</div></div></=
 div><div class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">#options =A0 =A0 ALTQ=
 _CBQ</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"=
 ><div class=3D"gmail_extra">#options =A0 =A0 ALTQ_RED</div></div></div><div=
  class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">#options =A0 =A0 ALTQ=
 _RIO</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"=
 ><div class=3D"gmail_extra">#options =A0 =A0 ALTQ_HFSC</div></div></div><di=
 v class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">#options =A0 =A0 ALTQ=
 _PRIQ</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra=
 "><div class=3D"gmail_extra"><br></div></div></div><div class=3D"gmail_extr=
 a"><div class=3D"gmail_extra">

 <div class=3D"gmail_extra">options =A0 =A0 IPFIREWALL</div></div></div><div=
  class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_extra=
 ">options =A0 =A0 IPFIREWALL_DEFAULT_TO_ACCEPT</div></div></div><div class=
 =3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">options =A0 =A0 IPFIR=
 EWALL_NAT</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra"><div class=3D"gmail_extra">options =A0 =A0 LIBALIAS</div></div></div>=
 <div class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">options =A0 =A0 IPDIV=
 ERT</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra">=
 <div class=3D"gmail_extra">options =A0 =A0 DUMMYNET</div></div></div><div c=
 lass=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><br></div></div></div=
 ><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div class=3D"gmail_=
 extra">device<span style=3D"white-space:pre-wrap">		</span>bpf<span style=
 =3D"white-space:pre-wrap">		</span># Berkeley packet filter</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div clas=
 s=3D"gmail_extra"><br></div></div></div><div class=3D"gmail_extra"><div cla=
 ss=3D"gmail_extra"><div class=3D"gmail_extra"># USB support</div></div></di=
 v><div class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">options <span style=
 =3D"white-space:pre-wrap">	</span>USB_DEBUG<span style=3D"white-space:pre-w=
 rap">	</span># enable debug msgs</div></div></div><div class=3D"gmail_extra=
 ">
 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>uhci<span style=3D"white-space:pre-wrap">		<=
 /span># UHCI PCI->USB interface</div></div></div><div class=3D"gmail_ext=
 ra">
 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>ohci<span style=3D"white-space:pre-wrap">		<=
 /span># OHCI PCI->USB interface</div></div></div><div class=3D"gmail_ext=
 ra">
 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>ehci<span style=3D"white-space:pre-wrap">		<=
 /span># EHCI PCI->USB interface (USB 2.0)</div></div></div><div class=3D=
 "gmail_extra">

 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>usb =A0 =A0 <span style=3D"white-space:pre-w=
 rap">	</span># USB Bus (required)</div></div></div><div class=3D"gmail_extr=
 a">
 <div class=3D"gmail_extra"><div class=3D"gmail_extra">device<span style=3D"=
 white-space:pre-wrap">		</span>umass<span style=3D"white-space:pre-wrap">		=
 </span># Disks/Mass storage - Requires scbus and da</div></div></div></bloc=
 kquote>

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div><br></div><div>A=
 lso src.conf and make.conf :</div><div><div><br></div></div></div></div><bl=
 ockquote style=3D"margin:0px 0px 0px 40px;border:none;padding:0px"><div cla=
 ss=3D"gmail_extra">

 <div class=3D"gmail_extra"><div><div>root at vpn_vrf:[VNET(x)]:/usr/src/sys # =
 cat /etc/src.conf=A0</div></div></div></div><div class=3D"gmail_extra"><div=
  class=3D"gmail_extra"><div>WITHOUT_ACCT=3Dyes</div></div></div><div class=
 =3D"gmail_extra">

 <div class=3D"gmail_extra"><div>WITHOUT_ACPI=3Dyes</div></div></div><div cl=
 ass=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITHOUT_AMD=3Dyes</div>=
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITH=
 OUT_APM=3Dyes</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITH=
 OUT_ASSERT_DEBUG=3Dyes</div></div></div><div class=3D"gmail_extra"><div cla=
 ss=3D"gmail_extra"><div>WITHOUT_AT=3Dyes</div></div></div><div class=3D"gma=
 il_extra">

 <div class=3D"gmail_extra"><div>WITHOUT_ATF=3Dyes</div></div></div><div cla=
 ss=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITHOUT_ATM=3Dyes</div><=
 /div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITHO=
 UT_AUDIT=3Dyes</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITH=
 OUT_BLUETOOTH=3Dyes</div></div></div><div class=3D"gmail_extra"><div class=
 =3D"gmail_extra"><div>WITHOUT_CALENDAR=3Dyes</div></div></div><div class=3D=
 "gmail_extra">

 <div class=3D"gmail_extra"><div>WITHOUT_CDDL=3Dyes</div></div></div><div cl=
 ass=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITHOUT_CTM=3Dyes</div>=
 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITH=
 OUT_DICT=3Dyes</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITH=
 OUT_FLOPPY=3Dyes</div></div></div><div class=3D"gmail_extra"><div class=3D"=
 gmail_extra"><div>WITHOUT_GAMES=3Dyes</div></div></div><div class=3D"gmail_=
 extra"><div class=3D"gmail_extra">

 <div>WITHOUT_HTML=3Dyes</div></div></div><div class=3D"gmail_extra"><div cl=
 ass=3D"gmail_extra"><div>WITHOUT_INFO=3Dyes</div></div></div><div class=3D"=
 gmail_extra"><div class=3D"gmail_extra"><div>WITHOUT_IPFILTER=3Dyes</div></=
 div></div>

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITHOUT_IPX=3Dye=
 s</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><d=
 iv>#WITHOUT_KERNEL_SYMBOLS=3Dyes</div></div></div><div class=3D"gmail_extra=
 "><div class=3D"gmail_extra">

 <div>WITHOUT_LEGACY_CONSOLE=3Dyes</div></div></div><div class=3D"gmail_extr=
 a"><div class=3D"gmail_extra"><div>WITHOUT_LOCALES=3Dyes</div></div></div><=
 div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITHOUT_LPR=3Dyes=
 </div></div>

 </div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITHOUT_MA=
 IL=3Dyes</div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_ex=
 tra"><div>WITHOUT_NDIS=3Dyes</div></div></div><div class=3D"gmail_extra"><d=
 iv class=3D"gmail_extra">

 <div>WITHOUT_QUOTAS=3Dyes</div></div></div><div class=3D"gmail_extra"><div =
 class=3D"gmail_extra"><div>WITHOUT_ROUTED=3Dyes</div></div></div><div class=
 =3D"gmail_extra"><div class=3D"gmail_extra"><div>WITHOUT_SENDMAIL=3Dyes</di=
 v></div></div>

 <div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>WITH_SVN=3Dyes</=
 div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>=
 WITHOUT_ZFS=3Dyes</div></div></div><div class=3D"gmail_extra"><div class=3D=
 "gmail_extra">

 <div><br></div></div></div><div class=3D"gmail_extra"><div class=3D"gmail_e=
 xtra"><div>root at vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf</div></=
 div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>CFLAGS=
 =3D-O2</div>

 </div></div><div class=3D"gmail_extra"><div class=3D"gmail_extra"><div>COPT=
 FLAGS=3D -O -pipe</div></div></div><div class=3D"gmail_extra"><div class=3D=
 "gmail_extra"><div>CPUTYPE=3Dgeode</div></div></div><div class=3D"gmail_ext=
 ra"><div class=3D"gmail_extra">

 <div>KERNCONF=3DMARS</div></div></div><div class=3D"gmail_extra"><div class=
 =3D"gmail_extra"><div>NO_MODULES=3Dyes</div></div></div><div class=3D"gmail=
 _extra"><div class=3D"gmail_extra"><div>BOOTWAIT=3D0</div></div></div><div =
 class=3D"gmail_extra">

 <div class=3D"gmail_extra"><div>DOC_LANG=3Den_US.ISO8859-1</div></div></div=
 ></blockquote><span class=3D""><font color=3D"#888888"><div class=3D"gmail_=
 extra"><div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra"><br>=
 </div><div class=3D"gmail_extra">
 --Nikolay</div><div class=3D"gmail_extra">
 <br></div></div></font></span></div>
 </blockquote></div><br></div><div class=3D"gmail_extra">Also, originally I =
 thought that the panic is when a multi path route is being deleted, however=
  again from the coredump it seems that the panic happens when openvpn delet=
 es the host route it installs for the remote openvpn server pointed to the =
 default gw (before openvpn installs the new default gw pointing to the vpn =
 tunnel) :<div>
 <br></div><div>(kgdb) fr 12</div><div><div>(kgdb) x/12sb td->td_proc-&gt=
 ;p_args</div><div>0xc4269780:<span class=3D"" style=3D"white-space:pre">	</=
 span>=A0"\001"</div><div>0xc4269782:<span class=3D"" style=3D"whi=
 te-space:pre">	</span>=A0""</div>
 <div>0xc4269783:<span class=3D"" style=3D"white-space:pre">	</span>=A0&quot=
 ;"</div><div>0xc4269784:<span class=3D"" style=3D"white-space:pre">	</=
 span>=A0"B"</div><div>0xc4269786:<span class=3D"" style=3D"white-=
 space:pre">	</span>=A0""</div>
 <div>0xc4269787:<span class=3D"" style=3D"white-space:pre">	</span>=A0&quot=
 ;"</div><div>0xc4269788:<span class=3D"" style=3D"white-space:pre">	</=
 span>=A0"/sbin/route"</div><div>0xc4269794:<span class=3D"" style=
 =3D"white-space:pre">	</span>=A0"delete"</div>
 <div>0xc426979b:<span class=3D"" style=3D"white-space:pre">	</span>=A0&quot=
 ;-net"</div><div>0xc42697a0:<span class=3D"" style=3D"white-space:pre"=
 >	</span>=A0"78.90.222.xxx"</div><div>0xc42697ad:<span class=3D""=
  style=3D"white-space:pre">	</span>=A0"10.255.255.0"</div>
 <div>0xc42697ba:<span class=3D"" style=3D"white-space:pre">	</span>=A0&quot=
 ;255.255.255.255"</div><div>(kgdb)=A0</div></div><div><br></div><div>I=
 'm trying to reproduce this on a VirtualBox instance now, however so fa=
 r no luck (no OpenVPN running, just adding and removing routes).</div>
 <div><br></div><div><br></div><div>--Nikolay</div></div></div>

 --001a11c1ba200f4f8504eee88df5--