kern/121743: ipfw in-kernel nat loses fragmented packets

[Alexander Zagrebin]

>Number:         121743
>Category:       kern
>Synopsis:       ipfw in-kernel nat loses fragmented packets
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Mar 15 19:00:05 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Alexander Zagrebin
>Release:        7.0-RELEASE
>Organization:
-
>Environment:
FreeBSD <hidden> 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sat Mar 15 19:18:40 MSK 2008     alex@<hidden>:/usr/src/sys/i386/compile/KERNEL  i386
>Description:
When trying to use ipfw in-kernel nat, I observed, that it loses
fragmented packets (see section "How to repeat the problem").
The problem is in current ipfw code (sys/netinet/ip_fw2.c).
After processing packet with LibAliasIn, ipfw analyses retcode and
drops the packet if retcode != PKT_ALIAS_OK.
But LibAliasIn, when processing the first fragment of the packet, returns
PKT_ALIAS_FOUND_HEADER_FRAGMENT instead. This code is not error, therefore ipfw should consider this fact, when deciding to pass or drop a packet.
(See the patch at "Fix to the problem if known")

Also,
libalias(3) (see section "FRAGMENT HANDLING") suggests the method of fragmented packets processing via LibAliasSaveFragment, LibAliasGetFragment, LibAliasFragmentIn, but neither ipfw nat code, nor user-space natd, doesn't use it now. It can be important, if packet's fragments will reordered during a way.
>How-To-Repeat:
My internal network is 192.168.1.0/24
External (public) network is 10.0.0.0/8
External interface (xl0) has address 10.255.255.2

Add to kernel config:
options         IPFIREWALL
options         IPFIREWALL_NAT
options         LIBALIAS

Add to ipfw config (xl0 is the external interface):
..
nat 1 config if xl0 log reset same_ports
add 999 count log ip from any to any via xl0
add 1000 nat 1 ip from any to any via xl0
add 1001 count log ip from any to any via xl0
..

To log packets after nat:
# sysctl net.inet.ip.fw.one_pass=0

Try to ping the external host (10.0.0.1 in my case) from an internal address,
and use packets with a size greater than MTU:

# ping -S 192.168.1.1 -s 2000 <some_external_host>
PING <some_external_host> from 192.168.1.1: 2000 data bytes
^C
--- 10.0.0.1 ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss

So, ping fails.

See /var/log/security (my comments are marked with >>>):
..
>Fix:
--- sys/netinet/ip_fw2.c.orig   2008-02-28 11:28:09.000000000 +0300
+++ sys/netinet/ip_fw2.c        2008-03-15 18:41:52.000000000 +0300
@@ -3568,7 +3568,8 @@
                                else
                                        retval = LibAliasOut(t->lib, c,
                                            MCLBYTES);
-                               if (retval != PKT_ALIAS_OK) {
+                               if (retval != PKT_ALIAS_OK &&
+                                   retval != PKT_ALIAS_FOUND_HEADER_FRAGMENT) {
                                        /* XXX - should i add some logging? */
                                        m_free(mcl);
                                badnat:

>Release-Note:
>Audit-Trail:
>Unformatted:
 >>> Our outbound ICMP echo request before NAT
 Mar 13 10:39:00 gw kernel: ipfw: 999 Count ICMP:8.0 192.168.1.1 10.0.0.1 out via xl0
 >>> Our outbound ICMP echo request after NAT
 Mar 13 10:39:00 gw kernel: ipfw: 1001 Count ICMP:8.0 10.255.255.2 10.0.0.1 out via xl0
 >>> ICMP echo reply (fragment 1) before NAT
 Mar 13 10:39:00 gw kernel: ipfw: 999 Count ICMP:0.0 10.0.0.1 10.255.255.2 in via xl0 (frag 20433:1480 at 0+)
 >>> ICMP echo reply (fragment 2) before NAT
 Mar 13 10:39:00 gw kernel: ipfw: 999 Count ICMP 10.0.0.1 10.255.255.2 in via xl0 (frag 20433:528 at 1480)
 >>> (!) ICMP echo reply (fragment 1) IS LOST!
 >>> ICMP echo reply (fragment 2) after NAT
 Mar 13 10:39:00 gw kernel: ipfw: 1001 Count ICMP 10.0.0.1 192.168.1.1 in via xl0 (frag 20433:528 at 1480)
 ..
 Mar 13 10:39:01 gw kernel: ipfw: 999 Count ICMP:8.0 192.168.1.1 10.0.0.1 out via xl0
 Mar 13 10:39:01 gw kernel: ipfw: 1001 Count ICMP:8.0 10.255.255.2 10.0.0.1 out via xl0
 Mar 13 10:39:01 gw kernel: ipfw: 999 Count ICMP:0.0 10.0.0.1 10.255.255.2 in via xl0 (frag 21967:1480 at 0+)
 Mar 13 10:39:01 gw kernel: ipfw: 999 Count ICMP 10.0.0.1 10.255.255.2 in via xl0 (frag 21967:528 at 1480)
 Mar 13 10:39:01 gw kernel: ipfw: 1001 Count ICMP 10.0.0.1 192.168.1.1 in via xl0 (frag 21967:528 at 1480)
 ..