rule not responding to incoming packets

[Mike M]

Damien,

 Further responses below.

On Thu, Jun 9, 2011 at 3:26 AM, Damien Fleuriot <ml at my.gd> wrote:

> Hey up Mike,
>
>
> Sorry about the delay, was busy at work ;)
>
> On 6/8/11 3:55 PM, Mike M wrote:
> > Hi Damien,
> >
> >   Thanks for helping out, I've provided responses beneath yours below.
> >
> > On Wed, Jun 8, 2011 at 11:25 PM, Damien Fleuriot <ml at my.gd
> > <mailto:ml at my.gd>> wrote:
> >
> >     Hey up Mike,
> >
> >
> >     My responses in between your own text.
> >
> >     On 6/8/11 9:58 AM, Mike M wrote:
> >     > Hi,
> >     >
> [snip]
> >
> >     > Another thing I've noticed is that the src limits seem to have an
> >     > effect (state table is typically 4k-7k entries), as without this in
> >     > place, the state table fills rapidly, rendering the box near
> unusable.
> >     >  Using 'synproxy state' also seems to have a similar effect.  I
> have
> >     > never observed any IP addresses within the <attacksource> table
> (via
> >     > 'pfctl -T show -t attacksource')
> >     >
> >
> >     Reduce your tcp timeout values so states are removed faster.
>

I understand that lowering these values helps remove the states quicker (and
thus helps prevent the table from filling up so easily), but I don't believe
this is the actual problem here (and the state table will still fill
regardless when the packet rate is high).

What I'm observing, is packets not being returned and thus connections in
the handshake not being fully established.

However, it's not happening for all connecting clients... I can connect to
the web server fine because I match the <whitelist> table rule, as can
others in the <staff> rule.

The problem is only prevalent for users matching the rule containing the
overload.  However, if I remove the 'overload' bits with the max src
settings, the state table fills instantly and renders the box unusable.
 However, the <attacksource> table never contains any entries, so how does
this make sense?


> >
> >
> > I think I've already done this but if you have specific sets or values
> > that you can suggest, I'll definitely take them on board.
> >
>
> These are the values we use on a production, high traffic website (3k
> hits/s)
>
> $ sudo pfctl -st
> tcp.first                     5s
> tcp.opening                   5s
> tcp.established              30s
> tcp.closing                   5s
> tcp.finwait                  10s
> tcp.closed                    5s
> tcp.tsdiff                   10s
> udp.first                    10s
> udp.single                   10s
> udp.multiple                 30s
> icmp.first                   10s
> icmp.error                    5s
> other.first                  10s
> other.single                 10s
> other.multiple               10s
> frag                         30s
> interval                      5s
> adaptive.start           156000 states
> adaptive.end             312000 states
> src.track                     0s
>
>
> >     > # -- adaptive timeouts
> >     > set timeout { tcp.first 20, adaptive.start 30000, adaptive.end
> >     1800000 }
> >
> >     If you're under DDoS, adjust your timers so that TCP syn packets time
> >     out much faster.
> >
> >     You can also set it only for your port 80 rule.
> >
> >
> > Any specific instruction for this? Open to suggestion!
> >
>
> This is an example of a timeout set only for a very specific rule:
> pass quick on $vpnif keep state ( tcp.established 36000 )
>
>
> >     > # -- disallow basic spoof
> >     > antispoof quick for { lo }
> >
> >     I do not see what you're hoping to achieve here.
> >     Also, you've set skip on lo, so you're adding rules that won't ever
> be
> >     applied.
> >
> >
> > This one was added on suggestion I had read in forums, if it is
> > unnecessary or not useful, I'll happily remove it?
> >
>
> Please do, this antispoof won't help you.
>
> It would if you applied it on your PHYSICAL interfaces, but on loopback
> ? no sense at all.
>

Roger that, will do.


>
>
> >     > block quick from <attacksource>
> >
> >     If these are spoofed IPs generated randomly, you'll saturate your
> table
> >     and you'll make your firewall work a lot for not much...
> >
> >
> > The idea behind this was from the 'overload' statement in my public
> > 80/TCP rule below -- if I can identify those attacking sources, I should
> > be able to drop those packets as quickly and efficiently as possible,
> > no?  What I've observed however, is that *zero* source IP's have been
> > added to the <attacksource> table so I'm not sure it's doing its job.
> > This is part of my reason to mail this list, I really don't know if this
> > is a bug, or the behaviour I'm seeing is part of my configuration.  If
> > the latter, I'm getting undesired results so am interested in getting
> > suggestions to mitigate this DDoS and remain serviceable to my public
> > clients.
> >
>
> Obviously they're not hitting the limit you set in your pass rule for
> port 80 then.
>

But how is it, that when I remove this bit from the rule, the state table
fills instantly?  It seems to have some sort of effect, but not the one I'd
expect.  This, is confusing.


>
>
> >     >
> >     > # -- HTTP public
> >     > pass in proto tcp from any to $h_pub port 80 flags S/SA keep state
> >     > (max-src-conn 100, max-src-conn-rate 20/5, overload <attacksource>
> >     > flush global)
> >
> >     flags S/SA is optional as it is the default.
> >     Why did you set $if_pub and $if_priv if you're not using them ?
> >
> >
> > Yeah, understand the bit about the flags but this is handled when the
> > conf is processed no?  So I assume the inclusion adds no extra
> > processing load.
> >
>
> Indeed, makes your ruleset file easier to read to the human eye though.
>
>
> > As far as the $if_priv var goes, yeah this is not utilised but again,
> > wouldn't imagine the lack of use would create a noticeable load?  It's
> > merely there for future use.
> >
>
> You may safely use them like so:
>
> pass in on $if_pub inet proto tcp from any to $if_pub port 80 keep state
> (source track...)
>
> Notice that when you say "to $if_pub", PF automagically takes the IPs
> assigned to your public interface.
>
> In case of multiple IPs, PF creates one rule per IP.
>
> >
> >     1/ I would suggest enabling logging on your default drop rule and
> run:
> >     tcpdump -nei pflog0 ip and port 80
> >
> >
> > Adding 'log' to my rules and monitoring pflog0 creates a substantial
> > load on the box (and fills up the hard drive quickly in /var/log... so
> > I'm not all that keen to add 'log' statements in my rules to do this.
> >  I've been looking at em0 with tcpdump to do my diagnostics.  Any reason
> > why this wouldn't suffice?
> >
>
> Indeed, the idea is to not leave this running for very long...
>
> Find below the output produced by tcpdump'ing pflog0 with the -e flag
> set (-e adds rule numbers):
>
> # tcpdump -nei pflog0 ip
> tcpdump: WARNING: pflog0: no IPv4 address assigned
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size
> 96 bytes
> 19:21:41.070165 rule 47/0(match): pass in on bce0: 109.0.65.3.12628 >
> 88.190.222.254.53: [|domain]
> 19:21:42.819873 rule 24/0(match): block in on bce0: 88.190.13.231.5353 >
> 224.0.0.251.5353: [|domain]
> 19:21:43.821475 rule 24/0(match): block in on bce0: 88.190.13.231.5353 >
> 224.0.0.251.5353: [|domain]
> ^C
> 3 packets captured
> 9 packets received by filter
> 0 packets dropped by kernel
>
>
> Notice how the log tells me what rule the packet matched (and ultimately
> passed or dropped).
>

I've actually already done this and I can see the rule in question, is the
one being matched.  But there is no other block rule or anything else that
helps me understand why packets don't go back out, and thus ending up with a
fully established connection.  If it's the kernel causing it, why can
sources in the <whitelist> and <staff> table establish connections without
problem? Is what I'm seeing, some sort of bug?



>
> Later on, I can check my rules like so: pfctl -vvsr
>
> This outputs rules with their number, for example rule 47 is:
> @47 pass in log on bce0 inet proto udp from any to 88.190.222.254 port =
> domain keep state (if-bound)
>  [ Evaluations: 4130668   Packets: 107887    Bytes: 10484013    States:
> 0     ]
>  [ Inserted: uid 0 pid 20941 ]
>
> >
> >     You'll see what rule matches when dropping your packets to port 80.
> >
> >     Chances are it will be your default drop rule, if so, this means your
> >     port 80 rule is not allowed to create any more state entries.
> >
> >
> > I really don't understand why the box isn't responding to packets that
> > are matched by the public rule for 80/TCP when those matched by
> > <whitelist> or <staff> tables are.  This tells me that the DDoS load is
> > not the cause of the problem, but rather the pf rule itself.  Can anyone
> > suggest why what I'm seeing, is actually happening?
> >
>
> See above my response about "pfctl -vvsr", it will give you counters for
> each of your rules.
>
> > I can't see any drop rules occurring when 'log' is enabled, it's just
> > that the incoming packets don't have anything go back 'out'.  Is this
> > kernel related?
> >
>
> Then that would mean your rule works and connections are sent to your
> backend web server (same machine ? diff machine ?).
>


The webserver is the same machine.  Routing back works, if I stick the
source IP's that are failing with this rule, in the <whitelist> or <staff>
tables, everything works fine.  This leads me to believe it's not a kernel
setting (which is not dependant on IP), but rather a problem with this
particular rule.  But how to fix it?


>
> You will want to ensure your backend web server can actually reach your
> clients' public IPs.
>
> >
> >     2/ Double check that your remote client test IP isn't in either
> >     <blacklist> or <attacksource>
> >
> >
> >
> > Again, there are *zero* entries in <blacklist> or <attacksource> -- what
> > strikes me as weird, is that even though the overload entry exists in
> > that rule, no IP's are actually inside <attacksource> (via 'pfctl -T
> > show -t attacksource') -- but when I remove the overload statement and
> > reload pf, the state table fills up rapidly and renders the box useless.
> >  This tells me that this overload condition is having an effect, but I
> > don't understand why the table is empty.
> >
>
> See above about "pfctl -vvsr"
>

Unfortunately this doesn't help me.  I can see the inbound rule is matched,
but there is no explanation for why the packets don't get sent back... the
handshake never completes so the connection isn't established... herein lies
my problem!

>
> > /still pulling hair out... please help! :>
> >
> > - Mike
>


Any ideas? :>

Cheers,

- Mike