NAT & RDR rules for jailed proxy services

[Carsten Larsen]

Hi there,

Beeblebrox wrote:
> I'd like to run some proxies in a jail (with only one service per jail).
> Internal Net and localhost traffic will be diverted to the jailed proxies.
> Also, jail traffic AND traffic for non-jailed services should be NATted on
> the router NIC (ExtIf).
>
> *The setup:*  Three interfaces: $ExtIf (192.168.1.10/24), $IntIf
> (192.168.2.1/26), lo0 (127.0.0.1/8).  One cloned interface $JailIf to be
> used as Jail NIC. No IP or netmask assigned to lo2. lo2 aliases:
> 192.168.2.97/32, 192.168.2.98/32, 192.168.2.99/32, 192.168.2.100/32 (four
> jails).
> * Jails need to send and receive traffic to a) each other b) {lo0, $IntIF}
> c) $ExtIF
> * Each jail has in <jail>/etc/rc.conf: defaultrouter="ExtIF's IP"
> * Traffic for jailed services originating from {lo0, $IntIF} should be
> redirected to the $JailIf. We should also make sure that no packets try to
> escape through $ExtIF without our consent.
>
> *Relevant pf.conf entries:*
> # ExtIf, IntIf, JailIf defined, also each jail and gateway IP's
>   jdns="192.168.2.97"   # Jail for unbound-DNS & dnscrypt-proxy using 443
>   jprvx="192.168.2.99"   # Jail for Privoxy
>   jhttp="192.168.2.100"  # Jail for squid-like http cache
>   gate="192.168.1.10"    # IP of ExtIf. forwarding gateway IP is 192.168.1.1
>
> # Define jailed ports & service numbers
>   JailTCP="{53,80,443,8080,8118}"
>
> # NAT & RDR rules  ## NO SKIP on lo0 ##
> # On $JailIf, NAT all traffic defined as to be hosted, and only that traffic
>   nat on $JailIf proto {tcp,udp} from !($JailIf) to any port $JailTCP ->
> $JailIf
>
> # Since (all packets) in the $JailTCP set have already been forwarded to
> $JailIf, NAT everything else arriving on the gateway normally.
>   nat on $ExtIf from any to $ExtIf -> $gate
>
The NAT logic should simply be:
nat on $ExtIf from any to !($ExtIf) -> ($ExtIf)

> # I need all DNS lookup queries to got to the jail running unbound. NAT
> already does this I think, so not needed any longer? # rdr on $JailIf proto
> {tcp,udp} from any port domain -> $jdns
>
> ## NOW re-direct by port, traffic exiting each jail to the gateway IP
> # Unbound DNS on .97
>   rdr on $JailIf proto {tcp,udp} from $jdns to any port {domain,443} -> $gate
>
> # Privoxy on .99
>   rdr on $JailIf proto {tcp} from $jprvx to any port {80,443} -> $gate
>
> # HTTP Cache on .100 = Not needed? Since jhttp (cache) -> jprvx (privoxy)
> traffic will be defined from http-cache config file.
> # rdr on $JaIf proto tcp from any to $JaIf port 8118 -> $jhttp
>
> # Ntpd time server for the LAN
>   rdr on $IntIf proto udp from $IntNet to $IntIf port ntp -> $IntIf
>
> *The problem:*
> 1) NAT and redirect rules fail. Traffic originating from inside jails does
> not reach the external network. For example, the DNS jail sends traffic out,
> but it looks like it's not routed to the gate. tcpdump shows:
> 192.168.2.97.57472 > 208.67.220.220.443: [udp sum ok] UDP, length 45
> Where <IP>:443 is the traffic generated by dnscrypt-proxy. Also, unrelated
> (?).
To my experience it is easier to do filtering while NAT'ing:
nat on $ExtIf from any to !($ExtIf) port https -> ($ExtIf)

If you need to account for traffic then add tags and use those tags in 
filter rules. For example.
nat on $ExtIf from any to !($ExtIf) port https tag NAT_HTTPS -> ($ExtIf)
..
pass on $ExtIf inet tagged NAT_HTTPS

> ethertype IPv4 (0x0800), length 74: (tos 0x1c, ttl 64, id 61312, offset 0,
> flags [DF], proto TCP (6), length 60, *bad cksum* 0 (->72d6)!)
> 2) The aim of the first NAT rule, is to capture on $ExtIf any rogue packets
> trying to escape without passing through the jail gauntlet. A solution would
> need to take that concern into account. Maybe a "block ou" filter for
> packets NOT originating from $JailIf would be simpler?
For jails I use rules in this form:
nat on $wan_if from $j_www to !$wan_ip port HTTP tag NAT_WWW_3 -> 
$wan_ip port $nat_ports

Also for accounting to work make sure you apply tagged filter rules in 
top of the rule set and to apply the keyword *quick* to these rules.

> 3) I try to monitor traffic on lo2 or pflog0 using tcpdump -nettvv and the
> above ruleset. Nothing shows up when I try to access the HTTP cache
> (192.168.2.100) although the cache responds on the browser (nothing to the
> outside, just cache internal page).
>
> I'm basically stuck in a logic loop and have failed to find my error.
>
> Regards.
>
>
>
> -----
> FreeBSD-11-current_amd64_root-on-zfs_RadeonKMS
> --
> View this message in context: http://freebsd.1045724.n5.nabble.com/NAT-RDR-rules-for-jailed-proxy-services-tp5869777.html
> Sent from the freebsd-pf mailing list archive at Nabble.com.
> _______________________________________________
> freebsd-pf at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to "freebsd-pf-unsubscribe at freebsd.org"
>