FreeBSD 11.1-RELEASE: Kernel panic in ipv6_output() via tcp6_usr_connect()
Viktor Dukhovni
freebsd at dukhovni.org
Tue Oct 31 00:04:43 UTC 2017
I am using FreeBSD 11.1 as the O/S for my DANE/SMTP adoption scanner.
The system has an IPv4 static IPv4 and also a corresponding 6to4
address on stf0.
The system is stable when I run IPv4-only scans, but crashes quickly
as soon as I start a bulk scan that also connects to the IPv6 addresses
of remote SMTP servers. Indeed after getting the destination address
of the connection that caused the panic (see below) I can now reproduce
the problem at will with just:
$ nc 2a01:5b40:0:2201::1 25
The system has ZFS and two igb network interfaces. The inside network
is my RFC1918 home network, so I use ipfw NAT rules for that, and set:
ifconfig_igb0="inet ... -tso -txcsum"
since hardware checksums don't seem to play along with ipfw and NAT.
The scans run on the machine itself, not an internal node.
After figuring out how build a debug kernel, and switch off encrypted
swap, I got the following stack trace:
#0 doadump (textdump=<value optimized out>) at pcpu.h:222
#1 0xffffffff80a6b6f1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366
#2 0xffffffff80a6bbb0 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80a6b9e3 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:690
#4 0xffffffff80edf832 in trap_fatal (frame=0xfffffe1041cc7260, eva=16) at /usr/src/sys/amd64/amd64/trap.c:801
#5 0xffffffff80edf889 in trap_pfault (frame=0xfffffe1041cc7260, usermode=0) at pcpu.h:222
#6 0xffffffff80edf0c6 in trap (frame=0xfffffe1041cc7260) at /usr/src/sys/amd64/amd64/trap.c:421
#7 0xffffffff80ec3641 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#8 0xffffffff80a49b50 in m_freem (mb=0x10) at /usr/src/sys/kern/kern_mbuf.c:952
#9 0xffffffff80c82b84 in ip6_output (m0=0xfffff80049f8e000, opt=<value optimized out>, ro=0xfffff80329e90530, flags=<value optimized out>,
im6o=<value optimized out>, ifpp=0x0, inp=<value optimized out>) at /usr/src/sys/netinet6/ip6_output.c:1040
#10 0xffffffff80c53269 in tcp_output (tp=0xfffff8027335c820) at /usr/src/sys/netinet/tcp_output.c:1403
#11 0xffffffff80c60f1b in tcp6_usr_connect (so=0xfffff80171dcc6c0, nam=<value optimized out>, td=0xfffff8017154c000) at /usr/src/sys/netinet/tcp_usrreq.c:642
#12 0xffffffff80af9bef in kern_connectat (td=<value optimized out>, dirfd=-100, fd=<value optimized out>, sa=0xfffff803296f3200)
at /usr/src/sys/kern/uipc_syscalls.c:584
#13 0xffffffff80af9aa7 in sys_connect (td=0xfffff8017154c000, uap=0xfffffe1041cc7930) at /usr/src/sys/kern/uipc_syscalls.c:549
#14 0xffffffff80ee0394 in amd64_syscall (td=0xfffff8017154c000, traced=0) at subr_syscall.c:135
#15 0xffffffff80ec392b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
#16 0x0000000803beee5a in ?? ()
Peeking at frame 8, it seems that "mb" is not a plausible mbuf pointer:
(kgdb) fr 8
#8 0xffffffff80a49b50 in m_freem (mb=0x10) at /usr/src/sys/kern/kern_mbuf.c:952
952 MBUF_PROBE1(m__freem, mb);
(kgdb) l
947 */
948 void
949 m_freem(struct mbuf *mb)
950 {
951
952 MBUF_PROBE1(m__freem, mb);
953 while (mb != NULL)
954 mb = m_free(mb);
955 }
(kgdb) p mb
$1 = (struct mbuf *) 0x10
Up a frame up I see:
(kgdb) up
#9 0xffffffff80c82b84 in ip6_output (m0=0xfffff80049f8e000, opt=<value optimized out>, ro=0xfffff80329e90530, flags=<value optimized out>,
im6o=<value optimized out>, ifpp=0x0, inp=<value optimized out>) at /usr/src/sys/netinet6/ip6_output.c:1040
1040 m_freem(m0);
(kgdb) l
1035 * Remove leading garbages.
1036 */
1037 sendorfree:
1038 m = m0->m_nextpkt;
1039 m0->m_nextpkt = 0;
1040 m_freem(m0);
1041 for (m0 = m; m; m = m0) {
1042 m0 = m->m_nextpkt;
1043 m->m_nextpkt = 0;
1044 if (error == 0) {
(kgdb) p *m0
$4 = {{m_next = 0x10, m_slist = {sle_next = 0x10}, m_stailq = {stqe_next = 0x10}}, {m_nextpkt = 0x0, m_slistpkt = {sle_next = 0x0}, m_stailqpkt = ...
Walking further up:
(kgdb) up
#10 0xffffffff80c53269 in tcp_output (tp=0xfffff8027335c820) at /usr/src/sys/netinet/tcp_output.c:1403
1403 error = ip6_output(m, tp->t_inpcb->in6p_outputopts,
(kgdb) l
1398 /* Save packet, if requested. */
1399 tcp_pcap_add(th, m, &(tp->t_outpkts));
1400 #endif
1401
1402 /* TODO: IPv6 IP6TOS_ECT bit on */
1403 error = ip6_output(m, tp->t_inpcb->in6p_outputopts,
1404 &tp->t_inpcb->inp_route6,
1405 ((so->so_options & SO_DONTROUTE) ? IP_ROUTETOIF : 0),
1406 NULL, NULL, tp->t_inpcb);
1407
(kgdb) up
#11 0xffffffff80c60f1b in tcp6_usr_connect (so=0xfffff80171dcc6c0, nam=<value optimized out>, td=0xfffff8017154c000) at /usr/src/sys/netinet/tcp_usrreq.c:642
642 error = tp->t_fb->tfb_tcp_output(tp);
(kgdb) l
637 (so->so_options & SO_NO_OFFLOAD) == 0 &&
638 (error = tcp_offload_connect(so, nam)) == 0)
639 goto out;
640 #endif
641 tcp_timer_activate(tp, TT_KEEP, TP_KEEPINIT(tp));
642 error = tp->t_fb->tfb_tcp_output(tp);
643
644 out:
645 TCPDEBUG2(PRU_CONNECT);
646 TCP_PROBE2(debug__user, tp, PRU_CONNECT);
(kgdb) up
#12 0xffffffff80af9bef in kern_connectat (td=<value optimized out>, dirfd=-100, fd=<value optimized out>, sa=0xfffff803296f3200)
at /usr/src/sys/kern/uipc_syscalls.c:584
584 error = soconnect(so, sa, td);
(kgdb) p *so
$2 = {so_count = 1, so_type = 1, so_options = 0, so_linger = 0, so_state = 260, so_qstate = 0, so_pcb = 0xfffff80329e903a0, so_vnet = 0x0,
so_proto = 0xffffffff8198a2c0, so_head = 0x0, so_incomp = {tqh_first = 0x0, tqh_last = 0xfffff80171dcc6f0}, so_comp = {tqh_first = 0x0,
tqh_last = 0xfffff80171dcc700}, so_list = {tqe_next = 0x0, tqe_prev = 0x0}, so_qlen = 0, so_incqlen = 0, so_qlimit = 0, so_timeo = 0, so_error = 0,
so_sigio = 0x0, so_oobmark = 0, so_rcv = {sb_sel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0}, si_note = {kl_list = {slh_first = 0x0},
kl_lock = 0xffffffff80a23fb0 <knlist_mtx_lock>, kl_unlock = 0xffffffff80a23ff0 <knlist_mtx_unlock>,
kl_assert_locked = 0xffffffff80a24030 <knlist_mtx_assert_locked>, kl_assert_unlocked = 0xffffffff80a24040 <knlist_mtx_assert_unlocked>,
kl_lockarg = 0xfffff80171dcc790, kl_autodestroy = 0}, si_mtx = 0x0}, sb_mtx = {lock_object = {lo_name = 0xffffffff8143469a "so_rcv",
lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, sb_sx = {lock_object = {lo_name = 0xffffffff814346ab "so_rcv_sx",
lo_flags = 36896768, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, sb_state = 0, sb_mb = 0x0, sb_mbtail = 0x0, sb_lastrecord = 0x0, sb_sndptr = 0x0,
sb_fnrdy = 0x0, sb_sndptroff = 0, sb_acc = 0, sb_ccc = 0, sb_hiwat = 65536, sb_mbcnt = 0, sb_mcnt = 0, sb_ccnt = 0, sb_mbmax = 524288, sb_ctl = 0,
sb_lowat = 1, sb_timeo = 0, sb_flags = 2048, sb_upcall = 0, sb_upcallarg = 0x0, sb_aiojobq = {tqh_first = 0x0, tqh_last = 0xfffff80171dcc848},
sb_aiotask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff80ad2450 <soaio_rcv>, ta_context = 0xfffff80171dcc6c0}},
so_snd = {sb_sel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0}, si_note = {kl_list = {slh_first = 0x0}, kl_lock = 0xffffffff80a23fb0 <knlist_mtx_lock>,
kl_unlock = 0xffffffff80a23ff0 <knlist_mtx_unlock>, kl_assert_locked = 0xffffffff80a24030 <knlist_mtx_assert_locked>,
kl_assert_unlocked = 0xffffffff80a24040 <knlist_mtx_assert_unlocked>, kl_lockarg = 0xfffff80171dcc8c8, kl_autodestroy = 0}, si_mtx = 0x0}, sb_mtx = {
lock_object = {lo_name = 0xffffffff81434693 "so_snd", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, sb_sx = {lock_object = {
lo_name = 0xffffffff814346a1 "so_snd_sx", lo_flags = 36896768, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, sb_state = 0, sb_mb = 0x0,
sb_mbtail = 0x0, sb_lastrecord = 0x0, sb_sndptr = 0x0, sb_fnrdy = 0x0, sb_sndptroff = 0, sb_acc = 0, sb_ccc = 0, sb_hiwat = 32768, sb_mbcnt = 0,
sb_mcnt = 0, sb_ccnt = 0, sb_mbmax = 262144, sb_ctl = 0, sb_lowat = 2048, sb_timeo = 0, sb_flags = 2048, sb_upcall = 0, sb_upcallarg = 0x0,
sb_aiojobq = {tqh_first = 0x0, tqh_last = 0xfffff80171dcc980}, sb_aiotask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0,
ta_func = 0xffffffff80ad2d00 <soaio_snd>, ta_context = 0xfffff80171dcc6c0}}, so_cred = 0xfffff80049ca4900, so_label = 0x0, so_peerlabel = 0x0,
so_gencnt = 21655, so_emuldata = 0x0, so_accf = 0x0, osd = {osd_nslots = 0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, so_fibnum = 0,
so_user_cookie = 0, so_pspare = 0xfffff80171dcca08, so_ispare = 0xfffff80171dcca18}
(kgdb) p/x (*sa->sa_data)@22
$7 = {0x0, 0x19, 0x0, 0x0, 0x0, 0x0, 0x2a, 0x1, 0x5b, 0x40, 0x0, 0x0, 0x22, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1}
So the connection seems to be port 25 (expected), flow info 0, at
2a01:5b40:0:2201::1 (which is mx01.domeneshop.no, also not surprising).
Anything further I can report? It seems I'll have to disable IPv6 for
now...
--
Viktor.
More information about the freebsd-net
mailing list