misc/185092: panic: rtfree 2 (using RADIX_MPATH in a VNET jail)
Nikolay Denev
ndenev at gmail.com
Thu Jan 2 16:14:39 UTC 2014
On Wed, Jan 1, 2014 at 5:39 PM, Nikolay Denev <ndenev at gmail.com> wrote:
> Ok, killing openvpn with -9 leaves the routes around, and particularly
> interesting are the following routes :
>
> 78.90.222.xx 10.255.255.0 UGHS 0 5841 epair0 =>
> 78.90.222.xx/32 10.255.255.0 UGS 0 0 epair0
>
> Now, if I do :
>
> route delete 78.90.222.xx 10.255.255.0
>
> The route, with the H flag is deleted. If I repeat the command the
> second route is deleted as well, even if the second command specifies
> a netmask no panic.
>
> However the first delete command specifies the /32 mask like this :
>
> route delete 78.90.222.xx 10.255.255.0 255.255.255.255
>
> Then I get "rtfree 2" kernel panic immediately.
>
> This seems to be happening as I'm manually installing static routes in
> the vnet jail for the VPN remote endpoints , however OpenVPN adds such
> routes too however differently, which results in two routing entries.
>
> For example :
>
> route add $IP $GW
> and
> route add $IP $GW 255.255.255.255
>
> add to different route entries, one is /32 network, the other is a host route.
>
>
>
> --Nikolay
>
> On Wed, Jan 1, 2014 at 1:21 PM, Nikolay Denev <ndenev at gmail.com> wrote:
>> On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <ndenev at gmail.com> wrote:
>>>
>>> On Sun, Dec 22, 2013 at 1:10 PM, <FreeBSD-gnats-submit at freebsd.org> wrote:
>>>>
>>>> Thank you very much for your problem report.
>>>> It has the internal identification `misc/185092'.
>>>> The individual assigned to look at your
>>>> report is: freebsd-bugs.
>>>>
>>>> You can access the state of your problem report at any time
>>>> via this link:
>>>>
>>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=185092
>>>>
>>>> >Category: misc
>>>> >Responsible: freebsd-bugs
>>>> >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail)
>>>> >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013
>>>
>>>
>>> I'm trying to understand exactly what is happening here, and examining a
>>> core dump with kgdb I'm getting some output that confuses me :
>>>
>>> (kgdb) bt
>>> #0 doadump (textdump=-1011569920) at pcpu.h:233
>>> #1 0xc06069b2 in kern_reboot (howto=260) at
>>> /usr/src/sys/kern/kern_shutdown.c:447
>>> #2 0xc0606d0e in panic (fmt=<value optimized out>) at
>>> /usr/src/sys/kern/kern_shutdown.c:754
>>> #3 0xc06de639 in rtfree (rt=<value optimized out>) at
>>> /usr/src/sys/net/route.c:464
>>> #4 0xc06e188d in route_output (m=<value optimized out>) at
>>> /usr/src/sys/net/rtsock.c:951
>>> #5 0xc06de18f in raw_usend (so=<value optimized out>, flags=0, m=<value
>>> optimized out>, nam=0x0, control=<value optimized out>,
>>> td=0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
>>> #6 0xc066eca9 in sosend_generic (so=0xc3e9c1a8, uio=<value optimized
>>> out>, top=<value optimized out>, control=0x0,
>>> flags=<value optimized out>, td=<value optimized out>) at
>>> /usr/src/sys/kern/uipc_socket.c:1271
>>> #7 0xc066efc7 in sosend (so=0xc3e9c1a8, addr=0x0, uio=0xd9b9cc10,
>>> top=0x0, control=0x0, flags=0, td=0xc3bd2000)
>>> at /usr/src/sys/kern/uipc_socket.c:1315
>>> #8 0xc0654af4 in soo_write (fp=0xc3c0c818, uio=0xd9b9cc10,
>>> active_cred=0xc3f1dd00, flags=0, td=0xc3bd2000)
>>> at /usr/src/sys/kern/sys_socket.c:103
>>> #9 0xc064c866 in dofilewrite (td=0xc3bd2000, fd=3, fp=0xc3c0c818,
>>> auio=0xd9b9cc10, offset=-1, flags=0) at file.h:303
>>> #10 0xc064c566 in kern_writev (td=0xc3bd2000, fd=3, auio=<value optimized
>>> out>) at /usr/src/sys/kern/sys_generic.c:467
>>> #11 0xc064c4bc in sys_write (td=<value optimized out>, uap=<value
>>> optimized out>) at /usr/src/sys/kern/sys_generic.c:382
>>> #12 0xc08614d3 in syscall (frame=<value optimized out>) at
>>> subr_syscall.c:134
>>> #13 0xc084cca1 in Xint0x80_syscall () at
>>> /usr/src/sys/i386/i386/exception.s:270
>>> #14 0x281975b7 in ?? ()
>>> Previous frame inner to this frame (corrupt stack?)
>>> Current language: auto; currently minimal
>>> (kgdb) fr 3
>>> #3 0xc06de639 in rtfree (rt=<value optimized out>) at
>>> /usr/src/sys/net/route.c:464
>>> 464 panic("rtfree 2");
>>> (kgdb) print *rt
>>> $1 = {rt_nodes = {{rn_mklist = 0xc3b4ab30, rn_parent = 0x1, rn_bit = 0,
>>> rn_bmask = 0 '\0', rn_flags = 0 '\0', rn_u = {rn_leaf = {
>>> rn_Key = 0xc0882687 "shutdown_post_sync", rn_Mask = 0x1030000
>>> <Address 0x1030000 out of bounds>, rn_Dupedkey = 0x0}, rn_node = {
>>> rn_Off = -1064819065, rn_L = 0x1030000, rn_R = 0x0}}},
>>> {rn_mklist = 0x0, rn_parent = 0x4, rn_bit = -18048, rn_bmask = -94 '?',
>>> rn_flags = 195 '?', rn_u = {rn_leaf = {rn_Key = 0xc3a545e0 "",
>>> rn_Mask = 0xc3a4e440 " ??(???\020'", rn_Dupedkey = 0xc3a4e880},
>>> rn_node = {rn_Off = -1012578848, rn_L = 0xc3a4e440, rn_R =
>>> 0xc3a4e880}}}}, rt_gateway = 0x74756873, rt_flags = 1853321060,
>>> rt_refcnt = 1936683103, rt_ifp = 0x79735f74, rt_ifa = 0x636e, rt_rmx =
>>> {rmx_mtu = 0, rmx_expire = 0, rmx_pksent = 0, rmx_weight = 0},
>>> rt_fibnum = 0, rt_mtx = {lock_object = {lo_name = 0x0, lo_flags = 0,
>>> lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}}
>>>
>>>
>>>
>>> rn_Key with value of “shutdown_post_sync” ?
>>>
>>> It’s visible also in the raw_usend() frame:
>>>
>>> (kgdb) fr 5
>>> #5 0xc06de18f in raw_usend (so=<value optimized out>, flags=0, m=<value
>>> optimized out>, nam=0x0, control=<value optimized out>,
>>> td=0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238
>>> 238 return ((*so->so_proto->pr_output)(m, so));
>>> (kgdb) print *m
>>> $2 = {m_hdr = {mh_next = 0xc3b4ab30, mh_nextpkt = 0x1, mh_data = 0x0,
>>> mh_len = -1064819065, mh_type = 0, mh_flags = 66304, mh_pad = 0},
>>> M_dat = {MH = {MH_pkthdr = {rcvif = 0x0, tags = {slh_first = 0x4}, len =
>>> -1012745856, flowid = 3282388448,
>>> csum_flags = 14097648373312316480, fibnum = 26739, cosqos = 117
>>> 'u', rsstype = 116 't', l2hlen = 100 'd', l3hlen = 111 'o',
>>> l4hlen = 119 'w', l5hlen = 110 'n', PH_per = {eigth = "_post_sy",
>>> sixteen = {28767, 29551, 24436, 31091}, thirtytwo = {1936683103,
>>> 2037604212}, sixtyfour = {8751443454668533855}, unintptr =
>>> {1936683103}, ptr = 0x736f705f}, PH_loc = {
>>> eigth = "nc\000\000\000\000\000", sixteen = {25454, 0, 0, 0},
>>> thirtytwo = {25454, 0}, sixtyfour = {25454}, unintptr = {25454},
>>> ptr = 0x636e}}, MH_dat = {MH_ext = {ref_cnt = 0x0, ext_buf =
>>> 0x0, ext_size = 0, ext_type = 0, ext_flags = 0, ext_free = 0,
>>> ext_arg1 = 0x0, ext_arg2 = 0x0},
>>> MH_databuf = '\0' <repeats 56 times>, "file", '\0' <repeats 20
>>> times>,
>>> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\00000Y?",
>>> '\0' <repeats 12 times>,
>>> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\000\000\000\000\000???",
>>> '\0' <repeats 23 times>}},
>>> M_databuf =
>>> "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync",
>>> '\0' <repeats 62 times>, "file", '\0' <repeats 20 times>,
>>> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\00000Y?",
>>> '\0' <repeats 12 times>,
>>> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\000\000\000\000\000???",
>>> '\0' <repeats 23 times>}}
>>>
>>>
>>> This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfree
>>> patch, which I thought earlier might be the cause of the panics I'm
>>> seeing).
>>>
>>> The machine is Soekris Net5501-70 with this kernel config :
>>>
>>> cpu I586_CPU
>>> cpu I686_CPU
>>> ident MARS
>>> options CPU_GEODE
>>> options CPU_SOEKRIS
>>>
>>> options HZ=2000
>>> options DEVICE_POLLING
>>> options BPF_JITTER
>>>
>>> makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
>>>
>>> options SCHED_ULE # ULE scheduler
>>> options PREEMPTION # Enable kernel thread preemption
>>> options INET # InterNETworking
>>> options INET6 # IPv6 communications protocols
>>> options TCP_OFFLOAD # TCP offload
>>> options FFS # Berkeley Fast Filesystem
>>> options SOFTUPDATES # Enable FFS soft updates support
>>> options UFS_DIRHASH # Improve performance on big directories
>>> options PROCFS # Process filesystem (requires PSEUDOFS)
>>> options PSEUDOFS # Pseudo-filesystem framework
>>> options GEOM_PART_GPT # GUID Partition Tables.
>>> options GEOM_LABEL # Provides labelization
>>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4
>>> options COMPAT_FREEBSD5 # Compatible with FreeBSD5
>>> options COMPAT_FREEBSD6 # Compatible with FreeBSD6
>>> options COMPAT_FREEBSD7 # Compatible with FreeBSD7
>>> options SCSI_DELAY=500 # Delay (in ms) before probing SCSI
>>> options KTRACE # ktrace(1) support
>>> options STACK # stack(9) support
>>> options SYSVSHM # SYSV-style shared memory
>>> options SYSVMSG # SYSV-style message queues
>>> options SYSVSEM # SYSV-style semaphores
>>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
>>> options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed.
>>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev
>>> options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4)
>>> options CAPABILITY_MODE # Capsicum capability mode
>>> options CAPABILITIES # Capsicum capabilities
>>> options PROCDESC # Support for process descriptors
>>> options INCLUDE_CONFIG_FILE # Include this file in kernel
>>>
>>> # Debugging support. Always need this:
>>> options KDB # Enable kernel debugger support.
>>> options KDB_TRACE # Print a stack trace for a panic.
>>> options KDB_UNATTENDED
>>>
>>> options TEXTDUMP_PREFERRED
>>> options TEXTDUMP_VERBOSE
>>>
>>> device pci
>>> device ata # Legacy ATA/SATA controllers
>>> options ATA_STATIC_ID # Static device numbering
>>>
>>> # ATA/SCSI peripherals
>>> device scbus # SCSI bus (required for ATA/SCSI)
>>> device da # Direct Access (disks)
>>> device pass # Passthrough device (direct ATA/SCSI access)
>>>
>>> # Add suspend/resume support for the i8254.
>>> device pmtimer
>>>
>>> # Serial (COM) ports
>>> device uart # Generic UART driver
>>>
>>> device miibus # MII bus support
>>> device vr # VIA Rhine, Rhine II
>>>
>>> # Wireless NIC cards
>>> device wlan # 802.11 support
>>> options IEEE80211_DEBUG # enable debug msgs
>>> options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's
>>> options IEEE80211_SUPPORT_MESH # enable 802.11s draft support
>>> device wlan_wep # 802.11 WEP support
>>> device wlan_ccmp # 802.11 CCMP support
>>> device wlan_tkip # 802.11 TKIP support
>>> device wlan_amrr # AMRR transmit rate control algorithm
>>> device ath # Atheros NICs
>>> device ath_pci # Atheros pci/cardbus glue
>>> device ath_hal # pci/cardbus chip support
>>> options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors
>>> options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation
>>> options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later
>>> device ath_rate_sample # SampleRate tx rate control for ath
>>>
>>> # Pseudo devices.
>>> device loop # Network loopback
>>> device random # Entropy device
>>> device ether # Ethernet support
>>> device vlan # 802.1Q VLAN support
>>> device tun # Packet tunnel.
>>> device md # Memory "disks"
>>> device gif # IPv6 and IPv4 tunneling
>>> device gre
>>> device faith # IPv6-to-IPv4 relaying (translation)
>>> device firmware # firmware assist module
>>> device if_bridge
>>>
>>> options VIMAGE
>>> options ROUTETABLES=8
>>> options RADIX_MPATH
>>>
>>> options SW_WATCHDOG
>>>
>>> device crypto
>>> device cryptodev
>>> device glxsb
>>>
>>> options BOOTVERBOSE=1
>>>
>>> #device pf
>>> #device pflog
>>> #device pfsync
>>> device carp
>>> device enc
>>> device lagg
>>> device epair
>>>
>>> #options ALTQ
>>> #options ALTQ_CBQ
>>> #options ALTQ_RED
>>> #options ALTQ_RIO
>>> #options ALTQ_HFSC
>>> #options ALTQ_PRIQ
>>>
>>> options IPFIREWALL
>>> options IPFIREWALL_DEFAULT_TO_ACCEPT
>>> options IPFIREWALL_NAT
>>> options LIBALIAS
>>> options IPDIVERT
>>> options DUMMYNET
>>>
>>> device bpf # Berkeley packet filter
>>>
>>> # USB support
>>> options USB_DEBUG # enable debug msgs
>>> device uhci # UHCI PCI->USB interface
>>> device ohci # OHCI PCI->USB interface
>>> device ehci # EHCI PCI->USB interface (USB 2.0)
>>> device usb # USB Bus (required)
>>> device umass # Disks/Mass storage - Requires scbus and da
>>>
>>>
>>> Also src.conf and make.conf :
>>>
>>> root at vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf
>>> WITHOUT_ACCT=yes
>>> WITHOUT_ACPI=yes
>>> WITHOUT_AMD=yes
>>> WITHOUT_APM=yes
>>> WITHOUT_ASSERT_DEBUG=yes
>>> WITHOUT_AT=yes
>>> WITHOUT_ATF=yes
>>> WITHOUT_ATM=yes
>>> WITHOUT_AUDIT=yes
>>> WITHOUT_BLUETOOTH=yes
>>> WITHOUT_CALENDAR=yes
>>> WITHOUT_CDDL=yes
>>> WITHOUT_CTM=yes
>>> WITHOUT_DICT=yes
>>> WITHOUT_FLOPPY=yes
>>> WITHOUT_GAMES=yes
>>> WITHOUT_HTML=yes
>>> WITHOUT_INFO=yes
>>> WITHOUT_IPFILTER=yes
>>> WITHOUT_IPX=yes
>>> #WITHOUT_KERNEL_SYMBOLS=yes
>>> WITHOUT_LEGACY_CONSOLE=yes
>>> WITHOUT_LOCALES=yes
>>> WITHOUT_LPR=yes
>>> WITHOUT_MAIL=yes
>>> WITHOUT_NDIS=yes
>>> WITHOUT_QUOTAS=yes
>>> WITHOUT_ROUTED=yes
>>> WITHOUT_SENDMAIL=yes
>>> WITH_SVN=yes
>>> WITHOUT_ZFS=yes
>>>
>>> root at vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf
>>> CFLAGS=-O2
>>> COPTFLAGS= -O -pipe
>>> CPUTYPE=geode
>>> KERNCONF=MARS
>>> NO_MODULES=yes
>>> BOOTWAIT=0
>>> DOC_LANG=en_US.ISO8859-1
>>>
>>>
>>>
>>> --Nikolay
>>>
>>
>> Also, originally I thought that the panic is when a multi path route is
>> being deleted, however again from the coredump it seems that the panic
>> happens when openvpn deletes the host route it installs for the remote
>> openvpn server pointed to the default gw (before openvpn installs the new
>> default gw pointing to the vpn tunnel) :
>>
>> (kgdb) fr 12
>> (kgdb) x/12sb td->td_proc->p_args
>> 0xc4269780: "\001"
>> 0xc4269782: ""
>> 0xc4269783: ""
>> 0xc4269784: "B"
>> 0xc4269786: ""
>> 0xc4269787: ""
>> 0xc4269788: "/sbin/route"
>> 0xc4269794: "delete"
>> 0xc426979b: "-net"
>> 0xc42697a0: "78.90.222.xxx"
>> 0xc42697ad: "10.255.255.0"
>> 0xc42697ba: "255.255.255.255"
>> (kgdb)
>>
>> I'm trying to reproduce this on a VirtualBox instance now, however so far no
>> luck (no OpenVPN running, just adding and removing routes).
>>
>>
>> --Nikolay
Some more testing shows that it's pretty easy to panic the machine.
One just have to try to delete twice a host route that has the same GW
as the default gw and the machine panics.
# ifconfig em0 10.0.0.155/24 up
# route add default 10.0.0.1
add net default: gateway: 10.0.0.1
# route add 8.8.8.8 10.0.0.1
add host 8.8.8.8: gateway: 10.0.0.1
# route delete 8.8.8.8 10.0.0.1
delete host 8.8.8.8: gateway: 10.0.0.1
# route delete 8.8.8.8 10.0.0.1
panic: rtfree 2
--Nikolay
More information about the freebsd-bugs
mailing list