Lots of weird PF behavior on 7.2-STABLE

[Peter Maxwell]

Hi Linda,

I'm pretty sure you can run tcpdump against a packet capture from the
pflog interface on the pf box; that will include fields like
block/pass and rule number for each packet filtered.  That way you at
least know what rule is dropping/passing your packets.  And if my
memory serves me right, pf uses a default pass rule - if it were me
I'd check that the SYN-ACK from the webserver isn't creating a second
state table entry using the default pass rule, or something equally as
annoying.

It is also possible I'm completely wrong, as it's been a while since
I've actually messed about with pf in any meaningful way.

However one thing that I would strongly suggest is using proper packet
filter design: either decide upon a default deny then pass what you
want, or decide on a default pass and only deny what is bad.  This
methodology applies to any firewall whether pf, checkpoint, cisco,
etc.  If you're having to use the "quick" keyword, you've probably*
done something wrong.  The default deny approach is usually
preferable, so you should have one block all rule at the top of the
ruleset then all other rules should be pass rules.

Best wishes,

Peter

* there are some situations where you may have to use quick, but not
particularly often.


2009/12/15 Linda Messerschmidt <linda.messerschmidt at gmail.com>:
> Hi all,
>
> I have a PF machine that is giving fits.  I see a lot of weird behavior.
>
> 1) TCP connections (mainly port 80) sometimes take 3 seconds to get
> started instead of being virtually instant.
> 2) Sometimes HTTP connections just stop responding.  (Client program
> times out waiting for response.)
> 3) Sometimes connections get weirdly dropped ("Connection reset by peer.")
> 4) Sometimes if I am ssh'd through the firewall, something will happen
> and my inbound packets will start getting dropped, but outbound
> packets still pass.  For example, if I'm at the shell prompt, it is
> non-responsive.  But if I log alongside a stuck connection and "write"
> to that tty, I will see it no problem.
> 5) States that have no right to still be there continue to pile up
> into the hundreds of thousands.
>
> I kind of get the feeling that all of these are related.  In
> particular, I think 2, 3, and 4.
>
> Of all of these, the only one I can document at the moment is #3.
>
> Here is a packet capture from the public (web client) interface:
>
> 20:00:02.038067 IP 1.2.3.4.61645 > 5.6.7.8.80: S
> 620577087:620577087(0) win 65535 <mss 1460,nop,wscale
> 9,sackOK,timestamp 953726452 0>
> 20:00:02.038328 IP 5.6.7.8.80 > 1.2.3.4.61645: S 40565958:40565958(0)
> ack 620577088 win 0 <mss 1460>
> 20:00:02.065678 IP 1.2.3.4.61645 > 5.6.7.8.80: . ack 1 win 65535
> 20:00:02.095158 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:02.378248 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:02.746163 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:03.282122 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:04.154112 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:05.698002 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:07.913721 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:12.145438 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:12.287038 IP 5.6.7.8.80 > 1.2.3.4.61645: F 1:1(0) ack 1 win 65535
> 20:00:20.408734 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:20.409874 IP 5.6.7.8.80 > 1.2.3.4.61645: R 40565959:40565959(0) win 0
>
> Here is a packet capture of the same session from the private (web
> server) interface:
>
> 20:00:02.038089 IP 1.2.3.4.61645 > 5.6.7.8.80: S
> 620577087:620577087(0) win 65535 <mss 1460,nop,wscale
> 9,sackOK,timestamp 953726452 0>
> 20:00:02.038311 IP 5.6.7.8.80 > 1.2.3.4.61645: S 40565958:40565958(0)
> ack 620577088 win 0 <mss 1460>
> 20:00:02.065694 IP 1.2.3.4.61645 > 5.6.7.8.80: . ack 1 win 65535
> 20:00:12.287026 IP 5.6.7.8.80 > 1.2.3.4.61645: F 1:1(0) ack 1 win 65535
> 20:00:20.408747 IP 1.2.3.4.61645 > 5.6.7.8.80: P 1:80(79) ack 1 win 65535
> 20:00:20.409859 IP 5.6.7.8.80 > 1.2.3.4.61645: R 40565959:40565959(0) win 0
>
> So that client -> server push packet is not making it through the
> firewall despite numerous retransmits, until 18 seconds later when the
> server has already given up on it.
>
> That connection hangs around in the state table for a long time as:
>
> all tcp 5.6.7.8:80 <- 1.2.3.4:61645       CLOSED:CLOSING
>
> This despite:
>
> set timeout tcp.closed 5
> set timeout tcp.closing 30
>
> To test, I stopped connections from 1.2.3.4 to 5.6.7.8.  At present,
> there are *zero* established connections between 1.2.3.4 and 5.6.7.8.
> None.  But:
>
> $ sudo pfctl -s state | fgrep 1.2.3.4: | fgrep :80 | wc
>    2243   13458  160932
>
> A few minutes later I broke this down by connection status:
> 1222 CLOSED:CLOSING
>  556 ESTABLISHED:ESTABLISHED
>  15 FIN_WAIT_2:CLOSING
>  27 SYN_SENT:FIN_WAIT_2
>
> That doesn't add up to 2243, so they *are* slowly dying off.  I did
> some poking around, and the CLOSED:CLOSING ones expire after fifteen
> minutes, which is the timeout for tcp.opening.  Um, OK.
>
> The 556 ESTABLISHED:ESTABLISHED states appear content to persist until
> they age off too, even though those connections are long gone.
>
> As far as the "3 second" thing, I noticed somebody here recently had a
> similar problem and made it go away by upping their states and
> dropping their timeouts.  Well, he dropped his timeouts to where ours
> are, and we're at:
>
> set limit states 2000000
>
> We are definitely not out of states; we're seeing these problems right
> now and due to my playing around with the tcp.established timeout,
> we're at 66412 states right now.  (Ordinarily it hovers around
> 350,000.)  The machine is a dual-core Core 2 6320 with 2GB of RAM and
> nothing to but load balance this traffic.  It shows as 95% idle all
> day.
>
> So sometimes pf loses packets related to connections that are still
> around, and sometimes it thinks connections are still around long
> after the packets are gone.
>
> I would be really, really grateful for any suggestions or help.  I am
> completely lost here and at my wits' end!
>
> I've included my pf.conf below.
>
>
> --------------------------------------------------------------------------------------------
>
> set limit states 2000000
> set timeout tcp.established 86400
> set timeout tcp.closed 5
> set timeout tcp.closing 30
>
> ExtIf = "em0"
> IntIf = "em1"
>
> table <NoRouteIPs> { 127.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24,
> 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }
> table <OurIPs> { ... }
> table <DNSServers> { ... }
> table <BalanceBlocks> { ... }
>
> scrub
>
> ##  Block Reserved Addresses
> block log quick on $ExtIf from <NoRouteIPs> to any
> block log quick on $ExtIf from any to <NoRouteIPs>
>
> ##  Block our own Addresses
> block in log quick on $ExtIf inet from <OurIPs> to any
>
> ##  Anti-DDOS
> table <AntiDDOS> persist
> block quick from <AntiDDOS> to any
> block quick from any to <AntiDDOS>
>
> ##  Block HTTP traffic to DNS servers
> block quick inet proto tcp from any to <DNSServers> port 80
>
> ##  Weird DNS people added 2009-06-18
> block drop log quick proto 255
> table <GTExperimentDNS> { 61.220.4.0/24 }
> block drop in quick proto { udp, tcp } from <GTExperimentDNS> to any port 53
>
> ## Load Balancing
> pass in on $ExtIf route-to { ($IntIf 3.4.5.6), ($IntIf 3.4.5.7),
> ($IntIf 3.4.5.8), ($IntIf 3.4.5.9) } round-robin proto tcp from any to
> <BalanceBlocks> port 80
> _______________________________________________
> freebsd-pf at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to "freebsd-pf-unsubscribe at freebsd.org"
>