Using PF ALTQ to schedule NAT jail traffic

[Bryan Alves]

I'm using a FreeBSD box to do the routing, NAT, and firewall duties for my
home network.  There are also a couple of jails located on the machine,
providing various services.  One of these jails is running a torrent client
24/7.  I isolated the torrent client to a jail in an attempt to more
effectively schedule packets going to and from it.

However, I'm having problems properly tagging packets that are originating
from the jail destined for the internet.

First, my ifconfig:

-------------------------
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:1b:21:29:b0:b0
        inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
        inet 192.168.10.200 netmask 0xffffffff broadcast 192.168.10.200
        inet 192.168.10.201 netmask 0xffffffff broadcast 192.168.10.201
        inet 192.168.10.202 netmask 0xffffffff broadcast 192.168.10.202
        inet 192.168.10.203 netmask 0xffffffff broadcast 192.168.10.203
        inet 192.168.10.204 netmask 0xffffffff broadcast 192.168.10.204
        media: Ethernet 1000baseTX <full-duplex>
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:1b:21:29:b0:ab
        inet 173.48.75.167 netmask 0xffffff00 broadcast 173.48.75.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
------------------------------


And now my current PF rules:

---------------------------------------------
ext_if          = "em1"

int_if          = "em0"
localnet        = $int_if:network
torrent_ports   = "57100:57199"
web_ports       = "81"
gateway         = "192.168.10.1"
httpd_jail      = "192.168.10.200"
samba_jail      = "192.168.10.201"
slimserver_jail = "192.168.10.202"
torrent_jail    = "192.168.10.203"
desktop         = "192.168.10.111"

set loginterface $ext_if
scrub on $ext_if

altq on $ext_if bandwidth 4550Kb hfsc queue { q_high, q_med, q_low }
  queue q_high        bandwidth 25% priority 6 qlimit 250 hfsc
  queue q_med         bandwidth 45% priority 4 qlimit 250 hfsc (default)
  queue q_low         bandwidth 30% priority 3 qlimit 500 hfsc

nat on $ext_if from $localnet to any -> ($ext_if)

#Port Forwards
rdr on $ext_if proto tcp from any to any port ssh            -> $gateway
rdr on $ext_if proto tcp from any to any port $web_ports     -> $httpd_jail
rdr on $ext_if proto tcp from any to any port $torrent_ports ->
$torrent_jail

#Nat Reflection
rdr on $int_if proto tcp from $localnet to $ext_if port ssh -> $gateway
rdr on $int_if proto tcp from $localnet to $ext_if port $web_ports ->
$httpd_jail
no nat on $int_if proto tcp from $int_if to $localnet
nat on $int_if proto tcp from $localnet to $gateway port ssh -> $int_if
nat on $int_if proto tcp from $localnet to $httpd_jail port $web_ports ->
$int_if

antispoof for $ext_if
#block in quick on $ext_if from <blacklisted> to any
block log all

pass on lo0

#Allow all LAN traffic
pass in on $int_if from $localnet to any keep state
pass out on $int_if from any to $localnet keep state
#Tag special traffic so it gets queued right
pass in on $int_if from $desktop      to any tag DESKTOP
pass in on $int_if from $torrent_jail to any tag TORRENT
pass in on $int_if from $httpd_jail   to any tag HTTPD

#In on ext_if
pass in on $ext_if proto tcp          from any to $httpd_jail   port
$web_ports     keep state     queue (q_high)
pass in on $ext_if proto { tcp, udp } from any to $torrent_jail port
$torrent_ports keep state     queue (q_low)
pass in on $ext_if proto tcp          from any to any           port
ssh            modulate state queue (q_high)
pass in on $ext_if proto gre          from any to
any                               keep state     queue (q_high)
pass in on $ext_if proto tcp          from any to any           port
pptp           keep state     queue (q_high)

#Out on ext_if
pass out on $ext_if proto tcp           all modulate state queue (q_med)
pass out on $ext_if proto { udp, icmp } all keep state     queue (q_med)
pass out on $ext_if proto gre           all keep state     queue (q_high)
#Queue tagged traffic
pass out on $ext_if proto { tcp, udp } from any to any tagged TORRENT queue
(q_low)
pass out on $ext_if from any to any tagged DESKTOP queue (q_high)
pass out on $ext_if from any to any tagged HTTPD   queue (q_high)
------------------------------------------------

The behavior of tagging NAT traffic so that it gets queued on the outside
interface works correctly in the case of the DESKTOP tag, which is a
different physical machine.  No packets pass through the firewall and get
tagged with either TORRENT or HTTPD, however.  Torrent traffic from
192.168.10.203 which originates from the jail passes through on the "pass
out on $ext_if proto tcp all modulate state queue (q_med) rule instead,
since it wasn't tagged on it's way through the NAT.  The 2nd "in on ext_if"
rule properly catches packets and puts them into the right queue, and
statefully the responses from these pass through it as well.  It's the
packets that non-statefully originate from the jail that I can't catch.

I suspect this has something to do with jail using lo0 to manage the
traffic, but I'm not sure how to fix it.  I tried to tcpdump the lo0, em1,
and em0 interfaces to find this traffic, but I can't find the traffic at all

How can I modify my rules and/or aliases so that I can properly tag all
traffic that originates from a specific jail?