Traffic mysteriously dropping

[Christopher McGee]

Christopher McGee wrote:

> I have 2 firewalls using all "em" network cards.  They have 2 onboard 
> Intel Gigabit interfaces and 1 quad port intel pro1000MT in each 
> firewall.  They are currently using both of the onboard interfaces and 
> 2 of the interfaces from the pci cards.  The firewalls are running 
> carp and pfsync for failover.  They are managing traffic for a gigabit 
> link and they usually don't push more than 150-200 Mbit/s and that is 
> rare.  Some http traffic is mysteriously just disappearing, even at 
> times when the firewalls are not busy(only 3-4 Mbit/s of traffic).  
> I've tested this, and the traffic is reaching the firewall(inbound to 
> our network) and hits pf and seems to be passing but then just never 
> makes it out the other interfaces(although pf does not log any blocked 
> packets).  The client will resend SYN packets until the connection 
> eventually just times out.  This timeout is happening on approximately 
> 1 out of 25 connections.
> Here is how I fixed this temporarily:
> I moved the rule for the http traffic to the FIRST rule of pf.conf and 
> make it a quick rule and bidirectional(stateless), it works and 
> doesn't seem to drop any connections.
>
> I have a fairly extensive ruleset, 378 rules to be exact when they are 
> all loaded.  I am using if-bound states.  If I make these rules 
> stateful, or move them down even one or 2 lines in the list of rules, 
> they start dropping connections again.  Hopefully someone can help 
> with this.
>
> Chris


A quick follow up since I realize I left out a little detail.  I have 
tried this on 5.4-RELEASE-p8 and 6.0-RELEASE-p6.  I've been trying to 
get altq working properly also, but it's been disabled until I work out 
the above problem.

The problem I've had with altq is trying to implement hfsc on the 6.0 
firewall.  I thought it was a pretty simple configuration.  I want to 
limit outgoing traffic to 100Mbit/s and have one queue higher priority, 
with a guaranteed 3 Mb of bandwidth, and a second lower priority queue 
with no guaranteed bandwidth.  The 2 queues should share the 97Mb of 
spare bandwidth evenly when the firewalls are busy, and queue2 should 
not be allowed to exceed 95Mb ever.  This is what I put together but it 
errors:

altq on $ext_if bandwidth 100Mb hfsc queue { queue1, queue2 }
queue queue1 priority 3 hfsc(realtime 3Mb linkshare 50% default red)
queue queue2 hfsc(upperlimit 95Mb linkshare 50% red)

I get the following error:
pfctl: the sum of the child bandwidth higher than parent "root_em0"

These 2 problems, are making pf, virtually unusable for our firewall 
needs.  Hopefully there is a fix for them.

Chris