pf randomly blocks specific packets?

[Peter Wullinger]

Nejc Škoberne wrote:
> pass in quick on $int_Trust from $addr_sysSvarun to any keep state
Note: You can remove "keep state". This is implicit for newer version of pf.
> pass quick on $int_Loop all
> pass quick on $int_Jails all
Note: These keep state, see above. You might want to add "no state" here,
to decrease state table usage.
> pass quick inet proto icmp all icmp-type $ICMPTypes keep state
> pass in on $int_Trust from any to any keep state
> pass out on $int_Trust from any to $addr_net_Local keep state
> pass out on $int_Trust from $int_Trust to any keep state
> pass out on lo1 from 192.168.223.10 to 192.168.223.10 keep state
> --------------------------------------------------------------------------------- 
>
>
> So as you can see there is a "pass quick on $int_Jails all" line. 
> Which, as far
> as I understand, should do exactly that. But, when I do
> "tcpdump -n -r /var/log/pflog", I get these:
>
> 10:22:56.353027 IP 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) 
> ack 1 win 8960 <nop,nop,timestamp 569935938 230047874>
 From the frequency of the logs, it looks like that there is heavy load 
on the server
(or a high connection latency).  If so, this may be a problem of state 
table exhaustion
or timeouts. pf may drop a "dangling, almost finished" connection before 
the final "FIN"
packet arrives and thus create such log entries as the final packet gets 
blocked, when the
corresponding state table entry is not present any more.

Do you monitor your state table size? If this is exhausted, other problems
are likely to occur, too.
>
> Which means, pf blocks these packets. Why would this be? The 
> consequence of
> blocking these packets are, that I get this in the hosts's Apache server:
>
> [Fri Jul 25 09:57:10 2008] [error] (1)Operation not permitted: proxy: 
> HTTP: attempt to connect to 192.168.223.10:80 (mail) failed

What is being blocked by PF are TCP packets with the FIN bit set. These 
are part of the connection
shutdown sequence. The error message here indicates, that connection 
establishment failed. I would find it strange,
if these are really related /directly/, but see below.

If I figure out the error message correctly, this means that the 
"connect()"system call failed. I
guess (would have to confirm by looking at the source code but after a 
short mailing list search
it seems to be likely), that a state table exhaustion can  also create 
"connect()" errors such as
these. "Cannot connect, because pf cannot create a state table entry for 
your connection".

To eliminate this possibility, you should monitor the size of your state 
table and possible increase the limits, if so.
Or insert some "no state" statements into your ruleset.

Regards,
    Peter