Pf, rtable, and rdr...bug?

[Dave Hayes]

[ Resending this to the PF list in hopes of some insight. Thanks. ]

Hello everyone. I'm having a problem with using rdr in an existing pf that uses rtable. I'm running 10.1-STABLE #0 r282154 and I believe this is a bug, but it could also be something I haven't spotted.

I have a firewall with three interfaces. The ip addresses have been changed to protect the innocent. :)

 - a slow net      (1.2.3.0/24)     interface: em0 @ 1.2.3.10
 - a fast net      (4.5.6.0/24)     interface: em1 @ 4.5.6.10
 - an internal net (192.168.4.0/24) interface: em2 @ 192.168.4.10

I route the internal net traffic over the fast cable net, and allow the internet net to access machines on the slower work net. Both default routes for the slow and fast net are .1 addresses (e.g. 1.2.3.1 and 4.5.6.1). I use an alias on both the slow and fast net (.42) to route the traffic from so I can see what's going on. I have net.fibs="2" in loader.conf and two different default routes set up for each fib. The default "default route" (fib 0) is 1.2.3.1.

Here's my pf ruleset that works, paraphrased.

    $slow_net = "1.2.3.0/24"
    $slow_if = "em0"
    $slow_nat_ip = "1.2.3.42"

    $fast_net = "4.5.6.0/24"
    $fast_if = "em1"
    $fast_nat_ip = "4.5.6.42"
     $int_net = "192.168.4.0/24"
    $int_if = "em2"
    $int_ip = "192.168.4.10"   # I don't alias this side

    table <private> const { 10/8, 172.16/12, 192.168/16 }

    nat log in $fast_if inet from $int_if:network to ! $slow_net -> $fast_nat_ip
    nat log on $slow_if inet from $int_if:network to $slow_net -> $slow_nat_ip
        block in log all
    antispoof log quick for { $slow_if $fast_if $int_if }
    pass in log quick on $int_if inet from $int_net to !$slow_if:network modulate state rtable 1
    pass in log quick on $int_if inet from $int_net to $slow_if:network modulate state rtable 0
    pass log on $slow_if inet from ! <private> to any modulate state
    pass out log inet from any to any modulate state

So I tried to use rdr to forward some ports from the to a machine on the internal net:

    $webserver = "192.168.4.22"
    ....
    rdr on $fast_if inet proto tcp from any to port 80 -> $webserver
    
This doesn't work. When I turn on tcpdump on all three interfaces, I see the packets coming in from the fast net to the internal net. The responses are appearing on the slow net, with the IP addresses of the fast net. So if I see this from em1:

   14:34:11.887357 IP 10.11.12.13:18600 > 4.5.6.42:80 ...

I then see the response...but on em0:

   14:34:12.087283 IP 4.5.6.42:80 > 10.11.12.13:18600 ...

Why doesn't this response packet go out the proper interface?

Thanks in advance for any insight. If I don't hear from anyone, I'm going to assume this is a bug and file a bug report. 
-- 
Dave Hayes - Consultant - Altadena CA, USA - dave at jetcafe.org 
>>>> *The opinions expressed above are entirely my own* <<<<

A path and a gateway have no meaning or use once the
objective is in sight.