Lots of weird PF behavior on 7.2-STABLE

[Peter Maxwell]

2009/12/15 Linda Messerschmidt <linda.messerschmidt at gmail.com>:
> On Tue, Dec 15, 2009 at 11:08 AM, Peter Maxwell <peter at allicient.co.uk> wrote:
>> I'm pretty sure you can run tcpdump against a packet capture from the
>> pflog interface on the pf box; that will include fields like
>> block/pass and rule number for each packet filtered.
>
> I have done that with "log" on all block rules.  The packets shown as
> missing are *not* present in the dump when I do, so as far as I can
> tell they are not being dropped by a filter rule.  Which makes sense,
> since none of the few block rules we have would touch packets in the
> middle of an established connection that was permitted.

Although it's not likely to be causing the problem (the default
"flags" on tcp rules are S/SA, which should exclude this possibility),
I'd check that the implicit pass rule isn't getting hit by the web
traffic.  Add in an explicit "pass all" rule at the start and set the
log keyword on it.  Make sure *none* of the web traffic is hitting
this rule.


>
> For comparison, endless streams of packets from those DNS guys we
> block *are* present in the tcpdump output, exactly as you describe, so
> I know pflog is working.
>
> So it really seems like something wrong in the internals of pf.  I
> just don't know how to pursue it further.


If the box isn't too loaded, you may try using "log (all)" on the pass
rules (so that ALL packets are logged, not just the initial SYN) -
that way at least you'd find out which rules those data packets are
hitting, which would likely pin down the problem quite a bit.. those
missing packets must have went somewhere ;-)  If it were me, that
would be my preferable option if it was available.

Barring that, I'd suggest simplifying your setup as much as possible -
is there too much traffic to remove the "route-to" and try it against
a single webserver?  If it's not possible, you could try setting up a
simple TCP service on internal hosts and get something that works,
then build up (ECHO or TIME are not bad for this).

I'd also suggest removing the "scrub" directive until you have it
working properly.  Is the "state-policy" floating or if-bound?

Does the problem affect other services in a similar manner, can you
replicate the exact same problem with the web servers with sshd, for
example?

What's annoying me is that I'm fairly sure I've seen this problem
before, but for the life of me I can't remember what caused it :-(


>
>> However one thing that I would strongly suggest is using proper packet
>> filter design: either decide upon a default deny then pass what you
>> want, or decide on a default pass and only deny what is bad.
>
> We are doing the latter; default pass and denying only what is bad.
> This isn't even really a firewall, it's for load balancing web
> connections.  We just threw a couple of block rules in there because
> it was a good place to stop a particular attack.  There are other
> "default deny" firewalls on other machines that handle all the traffic
> that isn't getting diverted by the load balance rule.
>
>> If you're having to use the "quick" keyword, you've probably*
>> done something wrong.
>
> Since we are using load balancing, the "pass" rule that wouldn't pass
> all the traffic we've just gone to the trouble to block would be
> outrageously complex.  Hence, quick.
>

pf uses the last rule in the ruleset that matches for a given packet,
so for a first instance setup I'd suggest putting an explicit "pass
all" as the first rule, then any pass rules that do load-balancing and
the like, then a list of block rules.  It makes it a lot easier to
read and debug.  Then once it's working as you want, you can move the
block rules up to the top and add in the "quick" keyword for
performance purposes.



> If a case can be made that the use of "quick" is causing our packets
> to disappear, we'd probably be willing to tackle trying to restructure
> the rules.
>
> Thanks!
>