kern/157796: IPFW in-kernel NAT nat loopback / Default Router Changes Unexpectedly

Sun Jun 12 16:10:15 UTC 2011

>Number:         157796
>Category:       kern
>Synopsis:       IPFW in-kernel NAT nat loopback / Default Router Changes Unexpectedly
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jun 12 16:10:14 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Ozkan KIRIK
>Release:        FreeBSD 8.2-RELEASE
>Organization:
Mersin University @ Turkey
>Environment:
FreeBSD ipfw.ipfw 8.2-RELEASE FreeBSD 8.2-RELEASE #9: Wed Mar  2 11:40:32 EET 2011     root at ipfw.ipfw:/usr/obj/usr/src/sys/ipfw amd64
>Description:
When a packet injects into ipfw in-kernel nat more then once,
operating systems behave abnormally.

Let's inspect the ruleset below:

ipfw table 12 add 10.0.14.1/32 33
ipfw table 13 add 193.X.128.30/32 33

ipfw nat 33 config redirect_addr 10.0.14.1 193.X.128.30
ipfw nat 799 config ip 3.3.3.3 reverse

55000 nat tablearg ip from table(12) to not 3.3.3.3 via em3
55000 nat tablearg ip from any to table(13) via em3
55000 nat 799 ip from any to table(13) not via em3
55000 nat tablearg ip from 3.3.3.3 to table(13)
55000 nat tablearg ip from table(12) to 3.3.3.3
55000 nat 799 ip from table(13) to 3.3.3.3

This ruleset, makes static nat.
To access from a client ( i.e. 10.0.14.5 ) to 193.X.128.30, 
I decided to make source address translation to incoming requests.

All incoming static nat requests is to be exposed to reverse nat. 
( source address translation as 3.3.3.3 )
All of this ruleset works, but under 5-6 Mbps static nat traffic load
( and total throughput about 150-200 Mbps ), operating system's
default router changes unexpectedly. When i try to monitor changes by "route -n monitor" command, I cannot see any events about default gateway.

New default gateway shown as the local ip address of static nat ( 10.0.14.1 ).
If you apply bandwidth limiting with dummynet, this problem occurs
more frequently. Dummynet catalyze the problem.

when i remove the reverse nat rules as :
55000 nat tablearg ip from table(12) to any
55000 nat tablearg ip from any to table(13)

everything is fine. ( default gateway doesnt change. ). Altough
dummynet is active, problem doesnt seen.
I think there is no problem with dummynet.

I tried different rulesets for different aims, and then i understood
that if you inject packets into in-kernel nat more then once system
behaves unexpectedly.

Another example attached below:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
       ether 00:23:8b:89:e4:9c
       inet6 fe80::223:8bff:fe89:e49c%em0 prefixlen 64 scopeid 0x1
       inet 193.X.128.1 netmask 0xffffff00 broadcast 193.X.128.255
       inet 193.X.128.4 netmask 0xffffffff broadcast 193.X.128.4
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
       ether 00:23:8b:89:e4:9d
       inet6 fe80::223:8bff:fe89:e49d%em1 prefixlen 64 scopeid 0x2
       inet 192.168.254.254 netmask 0xffffff00 broadcast 192.168.254.255
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
       ether 00:23:8b:89:e4:9e
       inet6 fe80::223:8bff:fe89:e49e%em2 prefixlen 64 scopeid 0x4
       inet 10.200.202.254 netmask 0xffffff00 broadcast 10.200.202.255
       inet 1.1.184.254 netmask 0xffffff00 broadcast 1.1.184.255
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

bce0.14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=103<RXCSUM,TXCSUM,TSO4>
       ether 00:1b:78:38:55:f8
       inet6 fe80::223:8bff:fe89:e49c%bce0.14 prefixlen 64 scopeid 0xb
       inet 10.0.14.254 netmask 0xffffff00 broadcast 10.0.14.255
       inet 1.1.3.254 netmask 0xffffff00 broadcast 1.1.3.255
       nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
       media: Ethernet autoselect (1000baseSX <full-duplex>)
       status: active
       vlan: 14 parent interface: bce0

ipfw nat 800 config same_ports reset redirect_addr 192.168.254.4 193.X.128.4

45000 nat 800 ip from 192.168.254.4 to any not via em2 // DMZ: Web_Server
45000 nat 800 ip from any to 193.X.128.4 not via em2 // DMZ: Web_Server

With this configuration, when you try to access from a client (
10.0.14.5 ) to 193.X.128.4 the system hangs immediately.
>How-To-Repeat:
ipfw table 12 add 10.0.14.1/32 33
ipfw table 13 add 193.X.128.30/32 33

ipfw nat 33 config redirect_addr 10.0.14.1 193.X.128.30
ipfw nat 799 config ip 3.3.3.3 reverse

55000 nat tablearg ip from table(12) to not 3.3.3.3 via em3
55000 nat tablearg ip from any to table(13) via em3
55000 nat 799 ip from any to table(13) not via em3
55000 nat tablearg ip from 3.3.3.3 to table(13)
55000 nat tablearg ip from table(12) to 3.3.3.3
55000 nat 799 ip from table(13) to 3.3.3.3

make a request from a client who is at same network with the static natted server about 4-5 Mbit/s. ( ie: client: 10.0.14.5 )

default router changes unexpectedly ( frequency : twice per minute )
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: