pf and ipv6 pmtu discovery not working

Thu May 1 13:13:38 UTC 2014

Hi,

for some time now i'm trying to nail down a phenomenon of hanging ipv6
transfers, it looks to me like there is a problem of getting ipv6 path 
mtu discovery right when using FreeBSD and pf.

My setup is an outside router (pisix, an rpi running freebsd connecting
to sixxs) connected via a firewall to a "freebsd server" (ernie). The
"outside router" has an MTU of 1280 bytes, everything after it to the 
inside has an MTU of 1500 bytes. On the server i have pf running. 
Its a 10-stable as of 28th april amd64 box. 

Now when the "freebsd server" sends a packet via the "outside router" 
which is larger than 1280 Bytes to the internet, the "outside router" 
sends an ICMP packet too big to the server, which in turn lowers its
MTU as shown in this tcpdump (which was taken with "set skip on fxp0"
in /etc/pf.conf, where fxp0 is the interface where this takes place):

10:45:16.071634 IP6 ernie.30711 > ber01s09-in-x18.1e100.net.https: Flags [P.], seq 518:1849, ack 167, win 2053, options [nop,nop,TS val 915142 ecr 2428082038], length 1331
10:45:16.073732 IP6 pisix > ernie: ICMP6, packet too big, mtu 1280, length 1240
10:45:16.073989 IP6 ernie.30711 > ber01s09-in-x18.1e100.net.https: Flags [.], seq 518:1726, ack 167, win 2053, options [nop,nop,TS val 915144 ecr 2428082038], length 1208
10:45:16.074119 IP6 ernie.30711 > ber01s09-in-x18.1e100.net.https: Flags [P.], seq 1726:1849, ack 167, win 2053, options [nop,nop,TS val 915144 ecr 2428082038], length 123

When the "set skip on fxp0" line in pf.conf is removed, the above
described pmtu scenario does not work anymore:

14:02:11.106566 IP6 ernie.17576 > ber01s09-in-x18.1e100.net.https: Flags [.], seq 1890:3288, ack 13944, win 2053, options [nop,nop,TS val 12730177 ecr 2439884647], length 1398
14:02:11.109301 IP6 pisix > ernie: ICMP6, packet too big, mtu 1280, length 1240
14:02:11.788529 IP6 ernie.17576 > ber01s09-in-x18.1e100.net.https: Flags [.], seq 1890:3288, ack 13944, win 2053, options [nop,nop,TS val 12730859 ecr 2439884647], length 1398
14:02:11.791271 IP6 pisix > ernie: ICMP6, packet too big, mtu 1280, length 1240
14:02:12.965534 IP6 ernie.17576 > ber01s09-in-x18.1e100.net.https: Flags [.], seq 1890:3288, ack 13944, win 2053, options [nop,nop,TS val 12732036 ecr 2439884647], length 1398
14:02:12.968302 IP6 pisix > ernie: ICMP6, packet too big, mtu 1280, length 1240
14:02:15.093370 IP6 ernie.17576 > ber01s09-in-x18.1e100.net.https: Flags [.], seq 1890:3288, ack 13944, win 2053, options [nop,nop,TS val 12734164 ecr 2439884647], length 1398

I found a similar thread of a FreeBSD bug in this area and tried to
disable net.inet.tcp.rfc1323, this makes no difference.

All sorts of rearranging rules or stripping ruled down to a bare
minimum in pf.conf made no difference.

Either disable pf at all or add "set skip on fxp0" make pmtu work.

Finally i added some printf's in /usr/src/sys/netinet6/icmp6.c, but
when pf is enabled, the icmp's don't even get there, so they must
be filtered out earlier.

At this point i found out about pfctl -x loud. For the pf enabled 
state, the console said for a similar situation:

pf: BAD ICMP 2:0 2001:xxxx:yyyy:100::253 -> 2001:xxxx:yyyy:200::1 state: TCP out wire: 2a00:1450:4008:801::101f[443] 2001:xxxx:yyyy:200::1[58987] stack: - [lo=2065898190 high=2065912306 win=2053 modulator=0 wscale=7] [lo=2904051605 high=2904314389 win=240 modulator=0 wscale=6] 4:4 seq=1059571332

Then i made the following modification to the file
/usr/src/sys/netpfil/pf/pf.c:

4665c4665
<                       if (!((*state)->state_flags & PFSTATE_SLOPPY) &&
---
>                       if ((icmptype != ICMP6_PACKET_TOO_BIG) && (!((*state)->state_flags & PFSTATE_SLOPPY) &&
4667c4667
<                           !SEQ_GEQ(seq, src->seqlo - (dst->max_win << dws)))) {
---
>                           !SEQ_GEQ(seq, src->seqlo - (dst->max_win << dws))))) {

and voila, pmtu works with pf enabled.

It seems the behaviour must have somthing to do with the if statement

if (!((*state)->state_flags & PFSTATE_SLOPPY) &&
     (!SEQ_GEQ(src->seqhi, seq) ||
      !SEQ_GEQ(seq, src->seqlo - (dst->max_win << dws)))) 

in /usr/src/sys/netpfil/pf/pf.c

Setting "keep state (sloppy)" in a line in pf.conf letting v6 icmp’s
thru made no difference. Same with adding „no state“.

I'm a bit stuck here right now. Is there perhaps someone with a bit
more insight in this type of pf-mechanics who might want to have a
look at this ?

Hellmuth
-- 
Hellmuth Michaelis
Hallstraße 20
25462 Rellingen

Mobil: +49-160-9645 5696
Telefon: +49-4101-85299-20
Telefax: +49-4101-85299-21

web: www.hellmuth-michaelis.de
mail: hm at hellmuth-michaelis.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20140501/ce38a043/attachment.sig>