HFSC and NAT Problems

Tue Dec 18 09:01:43 PST 2007

Hi,

I'm having an issue with HFSC where anything coming over my internal
interface NATed to the outside is not being placed in the default
queue on the external IF. I have a DMZ that's not NATed and packets
coming out of the DMZ are being placed in the queue, so I have reason
to believe that NAT may be the culprit.

Here is my ruleset (IP addresses have been changed to protect the innocent):
# pf.conf
# 12/14/2007 JMS

#### Macros: define common values, so they can be referenced and changed easily.

# Interfaces
internet     = "fxp0"
dmz1         = "fxp1"
dmz2         = "fxp2"
lan          = "fxp3"

# Networks
lan_net         = "192.168.0.0/24"
outside_net   = "x.x.x.x/xx"
dmz1_net      = "x.x.x.x/xx"
dmz2_net     = "x.x.x.x/xx"
multicast    = "224.0.0.251"

# Key Machines
asterisk     = "192.168.0.10"
external_ip  = "x.x.x.x"
dns_servers  = "{x.x.x.x x.x.x.x}"
rogue        = "x.x.x.x"

# Macros
dmz2_svcs       = "{www https 8080 smtp pop3 imap pop3s imaps ssh ftp >1024}"
ldap            = "{ldap ldaps}"
sql             = "{1433 3306}"
netbios_tcp     = "{139}"
netbios_udp     = "{137 138}"
netbios_tcp_udp = "{445 135}"
voip            = "{5059:5082 8000:20000 4569}"
rogue_svcs      = "{687 625 311 8079:9000}"

#### End Macros

# Tables: similar to macros, but more flexible for many addresses.
table <charlotte> { x.x.x.x x.x.x.x x.x.x.x x.x.x.x }

### Options

set   block-policy return
set   loginterface $internet
set   skip on lo0
scrub in all

#### End Options

#### Queueing and traqffic shaping

altq on $internet hfsc bandwidth 1.5Mb queue { std, voip, tcpack }
        queue std    bandwidth 33%            priority 1 hfsc (default)
        queue voip   bandwidth 34%            priority 7 hfsc (realtime 60%)
        queue tcpack bandwidth 33%            priority 6 hfsc (red realtime 20%)

#### End Queueing

### NAT/RDR Rules
nat on $internet from $lan_net to any -> ($internet)

nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr pass on $lan proto tcp from any to any port 21 -> 127.0.0.1 port 8021
rdr pass on $internet proto { udp tcp } from ! $lan_net to
$external_ip port $voip tag VOIP -> $asterisk

#### End NAT rules

#### Filtering rules

## Rules for all interfaces
nchor "ftp-proxy/*"
# POLICY: Block all incoming by default, only filter incoming to make
life easier.
block in log all
antispoof quick for { lo0 $lan }
block in log quick on ! $lan proto tcp from any to any port 8021
# Allow mDNS reflector
pass in quick on ! $internet proto udp from any to $multicast port 5353

## -------------------------------------------------------------------------
## Internet (fxp0)
## -------------------------------------------------------------------------

# Skip Logging, faster response to ident
block in quick on $internet proto {tcp udp} from any to $external_ip port 41899
block return-rst in quick on $internet proto tcp from any to any port 113

# Allow traffic from this machine (I thought the pass out catch all
rule would be sufficient,
# but for reasons that I don't understand it does not work)
pass out quick on $internet from $external_ip to any keep state queue(std)

# SQL For both DMZs
pass in quick on $internet proto tcp from any to {$dmz1_net $dmz2_net}
port $sql keep state

# Incoming from net into DMZ2 Allowed Services
pass in quick on {$internet $dmz2} proto { tcp udp } from any to
$dns_servers port domain keep state
pass in quick on {$internet $dmz2} proto tcp from any to $dmz2_net
port $dmz2_svcs keep state
pass in quick on {$internet $dmz2} proto tcp from any to $rogue port
$rogue_svcs keep state

# Outgoing SQL connections from DMZ2
pass in quick on {$internet $dmz2} proto tcp from $dmz2_net to any
port $sql keep state

## Push ident through faster
pass out quick on $internet proto tcp from any port 113 to any flags
R/RSFUP queue(std)

## Process tagged for VOIP packets and everything else gets pushed
into the std queue
pass out quick on $internet tagged VOIP keep state queue(voip)

### TESTING - Try to force traffic into the queue
pass out quick on $internet proto tcp from any to any queue(std, tcpack)
pass out quick on $internet tagged STD keep state queue(std, tcpack)
pass out quick on $internet from any to any keep state queue(std tcpack)

## -------------------------------------------------------------------------
## LAN (fxp3)
## -------------------------------------------------------------------------

## Tag SIP and IAX2 with VOIP tag for later queueing
pass in quick on $lan proto udp from $lan_net to any port $voip tag
VOIP keep state

# TESTING - Try to tag packets to force into std queue. Probably does
not work with NAT
pass in quick on $lan from $lan_net to any tag STD keep state

## Trust the lan (for now)
pass in quick on $lan from {! $dmz1_net ! $dmz2_net} to any keep state

## -------------------------------------------------------------------------
## DMZ1 (fxp1)
## -------------------------------------------------------------------------
pass in quick on $dmz1 proto { tcp udp } from any to $dns_servers port
domain keep state
pass in quick on $dmz1 proto tcp from any to $dmz1_net port $sql keep state

## -------------------------------------------------------------------------
## DMZ2 (fxp2)
## -------------------------------------------------------------------------

## Skip logging on these
block in quick on $dmz2 proto udp from $dmz2_net to 208.254.161.255 port 137
# Also see macro on incoming for fxp0
pass in quick on $dmz2 proto {tcp udp} from $dns_servers to any port
domain keep state
pass in quick on $dmz2 proto { tcp udp } from $lan_net to $dmz2_net
port 161 keep state
pass in quick on $dmz2 proto { tcp udp } from $dmz2_net to $lan_net
port {135 137} keep state
pass in quick on $dmz2 proto icmp from any to $dmz2_net keep state

# Quickly pass DNS
pass out quick on $dmz2 proto { tcp udp } from any to any port 53 keep state

# Only filter incoming to make things easier
pass out keep state

Here's some output from pfctl -vvvsq with the T1 line maxed out with
clients downloading files from the lan.

queue root_fxp0 bandwidth 1.50Mb priority 0 {std, voip, tcpack}
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.0 packets/s, 0 b/s ]
queue  std bandwidth 495Kb hfsc( default )
  [ pkts:     626828  bytes:  102182679  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:   101.9 packets/s, 66.23Kb/s ]
queue  voip bandwidth 510Kb priority 7 hfsc( realtime 900Kb )
  [ pkts:       1580  bytes:     170382  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.2 packets/s, 433.60 b/s ]
queue  tcpack bandwidth 495Kb priority 6 hfsc( red realtime 300Kb )
  [ pkts:      66393  bytes:    4435542  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
  [ measured:     0.1 packets/s, 28.80 b/s ]

The trafffic for TCP acks seems to be OK, while the bandwidth for the
std queue seems WAY low.