pf and fragmented packets

[Rui Paulo]

Hi,

I found two problems with pf where fragmented packets behind a NAT don't get properly transmitted/translated.  This affects things like the PS3, PS Vita and probably other consoles.

The first problem is when I send a fragmented ICMP/UDP packet and pf routes packets to the WAN interface _without_ changing the IP address and port.  To see this in action, you can install fragroute and then use 'fragtest frag www.google.com'.  In this case, my rule set has "scrub on $ext_if all fragment reassemble". 

Here's the packet dump on the WAN interface (notice the use of RFC 1918 addresses):

00:27:24.992023 IP (tos 0x0, ttl 63, id 40521, offset 0, flags [+], proto ICMP (1), length 28, bad cksum 0 (->78a1)!)
    10.0.1.87 > 74.125.239.34: ICMP echo request, id 48597, seq 1, length 8
00:27:24.992115 IP (tos 0x0, ttl 63, id 40521, offset 8, flags [+], proto ICMP (1), length 28, bad cksum 0 (->78a0)!)
    10.0.1.87 > 74.125.239.34: ip-proto-1
00:27:24.992189 IP (tos 0x0, ttl 63, id 40521, offset 16, flags [+], proto ICMP (1), length 28, bad cksum 0 (->789f)!)
    10.0.1.87 > 74.125.239.34: ip-proto-1
00:27:24.992263 IP (tos 0x0, ttl 63, id 40521, offset 24, flags [none], proto ICMP (1), length 28, bad cksum 0 (->989e)!)
    10.0.1.87 > 74.125.239.34: ip-proto-1

If I enable "reassemble tcp fragment reassemble", I get this:

00:28:43.989497 IP (tos 0x0, ttl 63, id 63913, offset 0, flags [none], proto ICMP (1), length 52, bad cksum 0 (->1fdf)!)
    24.6.16.155 > 74.125.239.34: ICMP echo request, id 27701, seq 1, length 32

It looks like "reassemble tcp" does the trick.  However, this is not TCP, so I'm guessing it's just a side effect.  This is also not a sensible workaround, because it doesn't work when the packets are too big.  That leads us to...

The second problem happens with large UDP packets.  If I change the rule "scrub on $ext_if all fragment reassemble" to "scrub on $ext_if all reassemble tcp fragment reassemble", I can see the UDP packets going out correctly translated, but if I send a large UDP packet (> MTU), pf sends the reassembled packet as a large packet which exceeds the MTU. 

Here's a packet trace from my PS Vita.  First on the internal interface:

00:35:06.673636 IP (tos 0x0, ttl 64, id 25171, offset 0, flags [+], proto UDP (17), length 1500)
    10.0.1.125.50929 > 198.107.156.154.3478: UDP, length 2108
00:35:06.673987 IP (tos 0x0, ttl 64, id 25171, offset 1480, flags [none], proto UDP (17), length 656)
    10.0.1.125 > 198.107.156.154: ip-proto-17

And the translated packet:

00:35:06.674096 IP (tos 0x0, ttl 63, id 25171, offset 0, flags [none], proto UDP (17), length 2136, bad cksum 0 (->859b)!)
    24.6.16.155.56632 > 198.107.156.154.3478: [udp sum ok] UDP, length 2108

This is just getting dropped by the interface because it's too big.  

I could share my complete rule set if that helps, but it's really easy to test this with fragtest.  The second test is not as simple because you either need a PS Vita or you will need to modify fragtest.c so that it sends a large packet.

I think this is a serious problem since it impacts the use of FreeBSD as a router.  Any ideas?  

--
Rui Paulo