ipfw problems using divert and fwd at the same time with 6.3-release

[Paul Thornton]

Hi folks,

I'm having trouble using NAT and forward in the same ipfw ruleset.  It 
appears that the forward "wins" over the NAT regardless of ordering in 
the ipwf ruleset.  I'm hoping that I'm missing something obvious; but is 
there a way to use these two together?

Some background - I'm testing in the lab a potential setup to provide 
limited network access to a few subnets in 10.X address space, but if 
you aren't going to an "approved" address then you get forwarded to a 
web page running on port 8000 on the same box.  This box is running 
6.3-RELEASE-p9 and has two em interfaces.

In this setup, 10.81.0.0/16 are my subnets.  They are presented to the 
machine as a bunch of VLANs physically on em1, one /24 subnet per VLAN. 
  The machine also does DHCP and DNS for each of these VLANs, and is the 
default gateway.

em0 is the external IP address for the machine, currently 192.91.199.5
The machine has no problem accessing the 'net.  If I remove the "clever" 
divert rules and the fwd rule, and make it a vanilla NAT setup, the 
client has no problem accessing the 'net.

In this setup, I expect to be able to browse to www.prt.org (on 
217.65.161.4) and that a machine in the 10.81.129.0/24 subnet has 
unrestricted NATted access to the 'net.  Any other attempt at browsing 
should hit the forward and display the "no access" page from the server 
on the gateway machine.

Using the following ruleset:

> [root at xrg1 /var/tmp]# ipfw show
> 00010     0       0 allow ip from any to any via lo0
> 00020     0       0 deny ip from any to 127.0.0.1
> 00022     0       0 deny ip from 127.0.0.1 to any
> 00050     0       0 allow udp from any 67-68 to 255.255.255.255 dst-port 67-68
> 00052     0       0 allow udp from 10.81.0.0/16 67-68 to me dst-port 67-68
> 00054     0       0 allow udp from me 67-68 to 10.81.0.0/16 dst-port 67-68
> 00056     0       0 allow udp from 10.81.0.0/16 to me dst-port 53
> 00058     0       0 allow udp from me 53 to 10.81.0.0/16
> 00060     0       0 allow icmp from 10.81.0.0/16 to me
> 00062     0       0 allow icmp from me to 10.81.0.0/16
> 00100     0       0 allow ip from 192.91.199.5 to any
> 02000     0       0 divert 8668 ip from 10.81.0.0/16 to 217.65.161.4 dst-port 80 via em0
> 05000     0       0 divert 8668 ip from 10.81.129.0/24 to any via em0
> 06000     0       0 divert 8668 ip from any to me via em0
> 08000     0       0 fwd 127.0.0.1,8000 tcp from 10.81.0.0/16 to any dst-port 80
> 32000     0       0 allow ip from any to any

If I browse "www.prt.org" on the client machine (10.81.2.246) I hit the 
fwd rule and I get my "Sorry you can't view this" webpage from the local 
server, and neither of the NAT rules are hit.
(DNS on the client correctly resolves to 217.65.161.4) :

> [root at xrg1 /var/tmp]# ipfw show
> 00010     0       0 allow ip from any to any via lo0
> 00020     0       0 deny ip from any to 127.0.0.1
> 00022     0       0 deny ip from 127.0.0.1 to any
> 00050     0       0 allow udp from any 67-68 to 255.255.255.255 dst-port 67-68
> 00052     0       0 allow udp from 10.81.0.0/16 67-68 to me dst-port 67-68
> 00054     0       0 allow udp from me 67-68 to 10.81.0.0/16 dst-port 67-68
> 00056     2     119 allow udp from 10.81.0.0/16 to me dst-port 53
> 00058     2     356 allow udp from me 53 to 10.81.0.0/16
> 00060     0       0 allow icmp from 10.81.0.0/16 to me
> 00062     0       0 allow icmp from me to 10.81.0.0/16
> 00100     3     214 allow ip from 192.91.199.5 to any
> 02000     0       0 divert 8668 ip from 10.81.0.0/16 to 217.65.161.4 dst-port 80 via em0
> 05000     0       0 divert 8668 ip from 10.81.129.0/24 to any via em0
> 06000     3     601 divert 8668 ip from any to me via em0
> 08000    43    4796 fwd 127.0.0.1,8000 tcp from 10.81.0.0/16 to any dst-port 80
> 32000    58   55935 allow ip from any to any

If I remove rule 8000, then I can browse to www.prt.org as expected, and 
I hit the divert rules:

> 00010     0       0 allow ip from any to any via lo0
> 00020     0       0 deny ip from any to 127.0.0.1
> 00022     0       0 deny ip from 127.0.0.1 to any
> 00050     0       0 allow udp from any 67-68 to 255.255.255.255 dst-port 67-68
> 00052     0       0 allow udp from 10.81.0.0/16 67-68 to me dst-port 67-68
> 00054     0       0 allow udp from me 67-68 to 10.81.0.0/16 dst-port 67-68
> 00056     7     460 allow udp from 10.81.0.0/16 to me dst-port 53
> 00058     7    1247 allow udp from me 53 to 10.81.0.0/16
> 00060     0       0 allow icmp from 10.81.0.0/16 to me
> 00062     0       0 allow icmp from me to 10.81.0.0/16
> 00100    45    3375 allow ip from 192.91.199.5 to any
> 02000    38    5096 divert 8668 ip from 10.81.0.0/16 to 217.65.161.4 dst-port 80 via em0
> 05000     0       0 divert 8668 ip from 10.81.129.0/24 to any via em0
> 06000    75   37498 divert 8668 ip from any to me via em0
> 32000   273  142906 allow ip from any to any

The natd config is trivial - I'm just launching it with:
  natd -port 8668 -same_ports -verbose interface em0

Does anyone have any ideas?  I've spent the whole weekend trying various 
things (like extra permits of the "special" traffic before the fwd line) 
but it makes no difference - the fwd still wins over everything.

Many thanks,

Paul.