NAT & RDR rules for jailed proxy services

[Beeblebrox]

I'd like to run some proxies in a jail (with only one service per jail).
Internal Net and localhost traffic will be diverted to the jailed proxies.
Also, jail traffic AND traffic for non-jailed services should be NATted on
the router NIC (ExtIf).

*The setup:*  Three interfaces: $ExtIf (192.168.1.10/24), $IntIf
(192.168.2.1/26), lo0 (127.0.0.1/8).  One cloned interface $JailIf to be
used as Jail NIC. No IP or netmask assigned to lo2. lo2 aliases:
192.168.2.97/32, 192.168.2.98/32, 192.168.2.99/32, 192.168.2.100/32 (four
jails).
* Jails need to send and receive traffic to a) each other b) {lo0, $IntIF}
c) $ExtIF
* Each jail has in <jail>/etc/rc.conf: defaultrouter="ExtIF's IP"
* Traffic for jailed services originating from {lo0, $IntIF} should be
redirected to the $JailIf. We should also make sure that no packets try to
escape through $ExtIF without our consent.

*Relevant pf.conf entries:*
# ExtIf, IntIf, JailIf defined, also each jail and gateway IP's
 jdns="192.168.2.97"   # Jail for unbound-DNS & dnscrypt-proxy using 443
 jprvx="192.168.2.99"   # Jail for Privoxy
 jhttp="192.168.2.100"  # Jail for squid-like http cache
 gate="192.168.1.10"    # IP of ExtIf. forwarding gateway IP is 192.168.1.1

# Define jailed ports & service numbers
 JailTCP="{53,80,443,8080,8118}"

# NAT & RDR rules  ## NO SKIP on lo0 ##
# On $JailIf, NAT all traffic defined as to be hosted, and only that traffic
 nat on $JailIf proto {tcp,udp} from !($JailIf) to any port $JailTCP ->
$JailIf

# Since (all packets) in the $JailTCP set have already been forwarded to
$JailIf, NAT everything else arriving on the gateway normally.
 nat on $ExtIf from any to $ExtIf -> $gate

# I need all DNS lookup queries to got to the jail running unbound. NAT
already does this I think, so not needed any longer? # rdr on $JailIf proto
{tcp,udp} from any port domain -> $jdns

## NOW re-direct by port, traffic exiting each jail to the gateway IP
# Unbound DNS on .97
 rdr on $JailIf proto {tcp,udp} from $jdns to any port {domain,443} -> $gate

# Privoxy on .99
 rdr on $JailIf proto {tcp} from $jprvx to any port {80,443} -> $gate

# HTTP Cache on .100 = Not needed? Since jhttp (cache) -> jprvx (privoxy)
traffic will be defined from http-cache config file.
# rdr on $JaIf proto tcp from any to $JaIf port 8118 -> $jhttp

# Ntpd time server for the LAN
 rdr on $IntIf proto udp from $IntNet to $IntIf port ntp -> $IntIf

*The problem:*
1) NAT and redirect rules fail. Traffic originating from inside jails does
not reach the external network. For example, the DNS jail sends traffic out,
but it looks like it's not routed to the gate. tcpdump shows:
192.168.2.97.57472 > 208.67.220.220.443: [udp sum ok] UDP, length 45
Where <IP>:443 is the traffic generated by dnscrypt-proxy. Also, unrelated
(?).
ethertype IPv4 (0x0800), length 74: (tos 0x1c, ttl 64, id 61312, offset 0,
flags [DF], proto TCP (6), length 60, *bad cksum* 0 (->72d6)!)
2) The aim of the first NAT rule, is to capture on $ExtIf any rogue packets
trying to escape without passing through the jail gauntlet. A solution would
need to take that concern into account. Maybe a "block ou" filter for
packets NOT originating from $JailIf would be simpler?
3) I try to monitor traffic on lo2 or pflog0 using tcpdump -nettvv and the
above ruleset. Nothing shows up when I try to access the HTTP cache
(192.168.2.100) although the cache responds on the browser (nothing to the
outside, just cache internal page).

I'm basically stuck in a logic loop and have failed to find my error.

Regards.



-----
FreeBSD-11-current_amd64_root-on-zfs_RadeonKMS
--
View this message in context: http://freebsd.1045724.n5.nabble.com/NAT-RDR-rules-for-jailed-proxy-services-tp5869777.html
Sent from the freebsd-pf mailing list archive at Nabble.com.