kern/168190: [pf] panic when using pf and route-to (maybe: bad fragment handling?)

Mon May 21 18:49:36 UTC 2012

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Content-ID: <alpine.BSF.2.00.1205212044250.74709 at unqrf.nqzva.sez2>

On Mon, 21 May 2012, Daniel Hartmeier wrote:

> On Mon, May 21, 2012 at 02:20:04PM +0000, Joerg Pulz wrote:
>
>>  ext_if="bge0"
>>  int_if="bge1"
>>  vpn_net="10.1.1.0/24"
>>  srv_net="172.16.1.0/24"
>>  gw_addr="172.16.1.254"
>>
>>  scrub in all
>>
>>  pass out on $ext_if route-to ($int_if $gw_addr) from $vpn_net to any keep state
>>  pass out on $int_if route-to ($int_if $gw_addr) from $vpn_net to $srv_net keep state
>
> So something from $vpn_net comes in, gets routed to the default gateway
> (on $ext_if side), attempts to pass out on $ext_if, matches the first
> rule, route-to applies, packet gets re-routed to $gw_addr, passes out
> on $int_if, matches the second rule, double route-to.
>
> All you need to do is prevent the second rule from applying for packets
> where the first rule matched, like with tags:
>
>  pass out on $ext_if route-to ($int_if $gw_addr) from $vpn_net to any keep state tag from_vpn
>  pass out on $int_if route-to ($int_if $gw_addr) from $vpn_net to $srv_net keep state
>  pass out on $int_if from $vpn_net to $srv_net keep state tagged from_vpn
>
> i.e. you add 'tag from_vpn' to the first rule, so packets matching it
> get tagged, then you add a third rule without route-to that applies to
> tagged packets, which wins last-match for such packets.
>
> Or, instead of adding a third rule, add '! tagged from_vpn' to the
> second rule, if tagged packets can still pass out on $int_if by another
> rule.

Daniel,

thanks again.
There is one main question left for me.
How could a packet which is directed to a host in $srv_net (and $srv_net 
is the same broadcast domain as $int_if) ever leave through $ext_if and 
make the first route-to match and therefor lead to a double route-to?

Regarding your other mail with the reproduce instructions

> BTW, if the theory is correct, you should be able to reproduce the
> problem by sending a packet in from $vpn_net to a host in $srv_net
> of size 334 (=306+28), e.g. with
>
>  $ ping -s 306 172.16.1.1

I tried this, but it did no harm to the system at all. Packets where 
flowing as they should and answers came back.

PF ruleset is still unchanged, no tagging.
Meanwhile the system panic'ed again, and this time there is no double 
route-to involved at all. kgdb(1) output below.
If your assumption in your first response is right:

> It looks like the byte order of ip_len is wrong, htons(334) = 19969,
> triggering fragmentation (334 < if_mtu, but 19969 > if_mtu).

Then it should read htons(175) = 44800
(175 < if_mtu, but 44800 > if_mtu")

But where is it coming from if double route-to can't be the culprit in 
this case?

Any thoughts on this?

Kind regards
Joerg

### kgdb.out2

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: m_copym, offset > size of mbuf chain
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
panic() at panic+0x182
m_copym() at m_copym+0x280
ip_fragment() at ip_fragment+0x1e5
pf_route() at pf_route+0x75c
pf_test() at pf_test+0xc29
pf_check_out() at pf_check_out+0x3a
pfil_run_hooks() at pfil_run_hooks+0xd2
ip_output() at ip_output+0x655
ip_forward() at ip_forward+0x175
ip_input() at ip_input+0x5fd
swi_net() at swi_net+0x15a
intr_event_execute_handlers() at intr_event_execute_handlers+0x66
ithread_loop() at ithread_loop+0xaf
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
- --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
KDB: enter: panic
Dumping 628 out of 4077 MB:..3%..11%..21%..31%..41%..51%..62%..72%..82%..92%

Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipmi.ko
#0  doadump (textdump=0) at pcpu.h:224
224		__asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) up 10
#10 0xffffffff806e9079 in m_copym (m=0x0, off0=1500, len=1480, wait=1)
     at /usr/src/sys/kern/uipc_mbuf.c:541
541			KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
(kgdb) list
536		KASSERT(len >= 0, ("m_copym, negative len %d", len));
537		MBUF_CHECKSLEEP(wait);
538		if (off == 0 && m->m_flags & M_PKTHDR)
539			copyhdr = 1;
540		while (off > 0) {
541			KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
542			if (off < m->m_len)
543				break;
544			off -= m->m_len;
545			m = m->m_next;
(kgdb) p off
$1 = 1325
(kgdb) up
#11 0xffffffff8077fe1f in ip_fragment (ip=0xfffffe0173b1fc74,
     m_frag=0xffffff8000241658, mtu=) at /usr/src/sys/netinet/ip_output.c:816
816			m->m_next = m_copym(m0, off, len, M_DONTWAIT);
(kgdb) list
811				len = ip->ip_len - off;
812				m->m_flags |= M_LASTFRAG;
813			} else
814				mhip->ip_off |= IP_MF;
815			mhip->ip_len = htons((u_short)(len + mhlen));
816			m->m_next = m_copym(m0, off, len, M_DONTWAIT);
817			if (m->m_next == NULL) {	/* copy failed */
818				m_free(m);
819				error = ENOBUFS;	/* ??? */
820				IPSTAT_INC(ips_odropped);
(kgdb) p *m0
$2 = {m_hdr = {mh_next = 0xfffffe0005b70000, mh_nextpkt = 0x0,
     mh_data = 0xfffffe0173b1fc74 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
     pad = "ÞÞÀÞ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
         header = 0x0, len = 175, flowid = 0, csum_flags = 1,
         csum_data = 59883, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
         tags = {slh_first = 0xfffffe00058c3840}}, MH_dat = {MH_ext = {
           ext_buf = 0xd29300ec0045 <Address 0xd29300ec0045 out of bounds>,
           ext_free = 0xd29300af0045, ext_arg1 = 0xba41965002280437,
           ext_arg2 = 0xaf00004557b3bb81, ext_size = 11615,
           ref_cnt = 0x240119ac02079b0a, ext_type = 1740374787},
         MH_databuf = "E\000ì\000\223Ò\000\000E\000¯\000\223Ò\000\0007\004(\002P\226Aº\201»³WE\000\000¯_-\000\000\177\001\034G\n\233\a\002¬\031\001$\003\003¼g\000\000\000\000E\000\000\223\215:\000\000>\0210F¬\031\001$\n\233\a\002\0005äv\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"}},
     M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000¯\000\000\000\000\000\000\000\001\000\000\000ëé\000\000\000\000\000\000ÞÀÞ@8\214\005\000þÿÿE\000ì\000\223Ò\000\000E\000¯\000\223Ò\000\0007\004(\002P\226Aº\201»³WE\000\000¯_-\000\000\177\001\034G\n\233\a\002¬\031\001$\003\003¼g\000\000\000\000E\000\000\223\215:\000\000>\0210F¬\031\001$\n\233\a\002\0005äv\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"...}}
(kgdb) p off
$3 = 1500
(kgdb) p len
$4 = 1480
(kgdb) p hlen
$5 = 20
(kgdb) up
#12 0xffffffff8032842a in pf_route (m=0xffffff80002418e8,
     r=0xfffffe0005d87750, dir=) at /usr/src/sys/contrib/pf/net/pf.c:6138
6138		error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum);
(kgdb) list
6133		/*
6134		 * XXX: is cheaper + less error prone than own function
6135		 */
6136		NTOHS(ip->ip_len);
6137		NTOHS(ip->ip_off);
6138		error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum);
6139	#else
6140		error = ip_fragment(m0, ifp, ifp->if_mtu);
6141	#endif
6142		if (error) {
(kgdb) p *ip
$6 = {ip_hl = 5 '\005', ip_v = 4 '\004', ip_tos = 0 '\0', ip_len = 44800,
   ip_id = 11615, ip_off = 0, ip_ttl = 127 '\177', ip_p = 1 '\001',
   ip_sum = 18204, ip_src = {s_addr = 34052874}, ip_dst = {s_addr = 604051884}}
(kgdb) p *m0
$7 = {m_hdr = {mh_next = 0xfffffe0005b70000, mh_nextpkt = 0x0,
     mh_data = 0xfffffe0173b1fc74 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
     pad = "ÞÞÀÞ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
         header = 0x0, len = 175, flowid = 0, csum_flags = 1,
         csum_data = 59883, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
         tags = {slh_first = 0xfffffe00058c3840}}, MH_dat = {MH_ext = {
           ext_buf = 0xd29300ec0045 <Address 0xd29300ec0045 out of bounds>,
           ext_free = 0xd29300af0045, ext_arg1 = 0xba41965002280437,
           ext_arg2 = 0xaf00004557b3bb81, ext_size = 11615,
           ref_cnt = 0x240119ac02079b0a, ext_type = 1740374787},
         MH_databuf = "E\000ì\000\223Ò\000\000E\000¯\000\223Ò\000\0007\004(\002P\226Aº\201»³WE\000\000¯_-\000\000\177\001\034G\n\233\a\002¬\031\001$\003\003¼g\000\000\000\000E\000\000\223\215:\000\000>\0210F¬\031\001$\n\233\a\002\0005äv\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"}},
     M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000¯\000\000\000\000\000\000\000\001\000\000\000ëé\000\000\000\000\000\000ÞÀÞ@8\214\005\000þÿÿE\000ì\000\223Ò\000\000E\000¯\000\223Ò\000\0007\004(\002P\226Aº\201»³WE\000\000¯_-\000\000\177\001\034G\n\233\a\002¬\031\001$\003\003¼g\000\000\000\000E\000\000\223\215:\000\000>\0210F¬\031\001$\n\233\a\002\0005äv\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"...}}
(kgdb) p *m1
$8 = {m_hdr = {mh_next = 0xfffffe0005b70000, mh_nextpkt = 0x0,
     mh_data = 0xfffffe0173b1fc74 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
     pad = "ÞÞÀÞ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
         header = 0x0, len = 175, flowid = 0, csum_flags = 1,
         csum_data = 59883, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
         tags = {slh_first = 0xfffffe00058c3840}}, MH_dat = {MH_ext = {
           ext_buf = 0xd29300ec0045 <Address 0xd29300ec0045 out of bounds>,
           ext_free = 0xd29300af0045, ext_arg1 = 0xba41965002280437,
           ext_arg2 = 0xaf00004557b3bb81, ext_size = 11615,
           ref_cnt = 0x240119ac02079b0a, ext_type = 1740374787},
         MH_databuf = "E\000ì\000\223Ò\000\000E\000¯\000\223Ò\000\0007\004(\002P\226Aº\201»³WE\000\000¯_-\000\000\177\001\034G\n\233\a\002¬\031\001$\003\003¼g\000\000\000\000E\000\000\223\215:\000\000>\0210F¬\031\001$\n\233\a\002\0005äv\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"}},
     M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\000¯\000\000\000\000\000\000\000\001\000\000\000ëé\000\000\000\000\000\000ÞÀÞ@8\214\005\000þÿÿE\000ì\000\223Ò\000\000E\000¯\000\223Ò\000\0007\004(\002P\226Aº\201»³WE\000\000¯_-\000\000\177\001\034G\n\233\a\002¬\031\001$\003\003¼g\000\000\000\000E\000\000\223\215:\000\000>\0210F¬\031\001$\n\233\a\002\0005äv\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000ÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞÞÀÞ"...}}
(kgdb)

### kgdb.out2

- -- 
The beginning is the most important part of the work.
 				-Plato
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (FreeBSD)

iD8DBQFPuo3vSPOsGF+KA+MRAu+AAKDGT0lDqaIcYO4Q6Lx37oUX64GeCgCffGhf
AtfXrgD94GTXHsX7roaKfAI=
=wEqQ
-----END PGP SIGNATURE-----