occasional "Operation not permitted" on state-mismatch

[Max Laier]

On Tuesday 18 December 2007, Atrox wrote:
> Atrox wrote:
> > Hello!
> >
> > I have some FreeBSD-boxes (2x6.3-PRERELEASE (installed on 08.Dec),
> > 1x6.2-RELEASE) with PF configured. They are connected with OpenVPN
> > LAN-to-LAN
> > and the problem is that a few times per hour connection drops between
> > computers from one LAN to another. At first I blamed OpenVPN, then I
> > blamed
> > bridge, but now I've realized that the problem is in PF.
> > So I've tried increasing TCP-timeouts and setting optimization
> > to "aggressive", but well, it's still the same.
> >
> > I monitor connections by sending TCP packets once per second to some
> > other host and wait for reply. I use Nagios-plugins' check_tcp for
> > that. The script
> > looks like:
> > =====
> > while [ 1 ]; do
> > 	pfctl -si |grep mismatch
> > 	/usr/local/libexec/nagios/check_tcp -H $host -p $port -t 2
> > 	pfctl -si |grep mismatch
> > 	sleep 1
> > done
> > =====
> >
> > So if I let this script into action, I see that in 2-3 minutes,
> > check_tcp gets "Operation not permitted" error and just in this
> > moment packet-mismatch
> > counter is increased by one (on machine with lesser traffic, I get
> > the timeout
> > in a few hours). That's on both 6.3-PRERELEASE as well as on
> > 6.2-RELEASE. I've
> > tried connections:
> > * along WAN to IPFW-enabled machines
> > * along WAN to PF-enabled machines
> > * along LAN to PF-enabled machines
> > * along LAN to Windows machines
> > * along VPN to PF-enabled machines
> > * along VPN to Windows machines
> >
> > Sometimes I get just some connection timeout: CRITICAL - Socket
> > timeout after
> > 2 seconds (I don't know what could cause that).
> >
> > I can see this behaviour in about every FreeBSD/PF machine I have.
> >
> > The basic PF-configuration looks like:
> > =====
> > set block-policy return
> > set loginterface $ext_if
> > set timeout tcp.closed 15
> > set optimization aggressive
> > scrub in all no-df
> >
> > block drop out quick on $ext_if from ($ext_if) to 0.0.0.0
> > block log all
> > pass quick on lo0 all
> > pass out all modulate state
> > pass out proto tcp all flags S/SA modulate state
> > pass on $int_if all modulate state
> > pass on $int_if proto tcp all flags S/SA modulate state
> > pass in on $ext_if proto tcp from any to ($ext_if) port $tcp_services
> > flags
> > S/SA modulate state
> > =====
> >
> > Is PF buggy or have I misconfigured smth?
>
> Today I installed an OpenBSD-4.2 box just to see whether PF does the
> same thing there. And yes, it does.
> pf.conf:
> =====
> ext_if = rl0
> set block-policy return
> set loginterface $ext_if
> scrub in all no-df
> block drop out quick on $ext_if from ($ext_if) to 0.0.0.0
> pass all modulate state
> pass quick on lo0 all
> =====
>
> I check TCP without "sleep 1" now, and I do it to FreeBSD box without
> firewall. state-mismatch gets increased by one, and I get either "No
> route to host" or "Socket timeout after 2 seconds".
>
> Am I still misconfiguring the thing?

No idea, you don't give nearly enough data for us to figure out what your 
setup looks like.  Regular tcp state mismatch usually hints that pf isn't 
seeing all packets of the conversation.  This can be caused by triangular 
routing, load balanceing or if_bridge (which is difficult to get right in 
some scenarios).

You should figure out the exact path your tcp packets are taking (back and 
forth) and make sure pf sees all of them.  Enabling additional pf logging 
(pfctl -xm) helps to figure out what kind of mismatch is happening.

-- 
/"\  Best regards,                      | mlaier at freebsd.org
\ /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.love2party.net/  | mlaier at EFnet
/ \  ASCII Ribbon Campaign              | Against HTML Mail and News
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: This is a digitally signed message part.
Url : http://lists.freebsd.org/pipermail/freebsd-pf/attachments/20071218/50a87031/attachment.pgp