pf randomly blocks specific packets?

Tue Jul 29 09:40:27 UTC 2008

Hello,

I have a FreeBSD 7.0 system with jails (and services in them). In one of the jails there
is an Apache server, which also runs on the host system (and forwards traffic using
mod_proxy to the jailed Apache).

Everything works as expected, I only have problems with pf which seems to block certain
packets randomly (not all of them).

This is how my rc.conf  on host system looks like (relevant parts):

---------------------------------------------------------------------------------
defaultrouter="172.20.2.1"
ifconfig_em0="inet 172.20.2.2 netmask 255.255.255.0"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
apache22_enable="YES"
cloned_interfaces="lo1"
ifconfig_lo1="192.168.223.1"
jail_enable="YES"
jail_sysvipc_allow="YES"
jail_list="mail"
jail_mail_rootdir="/usr/jail/j/mail"
jail_mail_hostname="mail"
jail_mail_ip="192.168.223.10"
jail_mail_interface="lo1 netmask 255.255.255.0"
jail_mail_devfs_enable="YES"
jail_mail_procfs_enable="YES"
jail_mail_devfs_ruleset="devfsrules_jail"
---------------------------------------------------------------------------------

This is how my pf.conf looks like:

---------------------------------------------------------------------------------
int_Trust = "em0"
int_Loop = "lo0"
int_Jails = "lo1"
int_jail_mail = "{" $int_Trust "}"

addr_net_Private = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"
addr_net_Local = "{" $int_Trust:network ", 172.20.2.0/24, 192.168.0.0/16 }"
addr_sysSvarun = "a.b.c.d"
addr_jail_mail = "192.168.223.10"

svc_TCP_HOST_Private = "{ ssh, iwebmin, itelnet }"
svc_TCP_HOST_Public = "{ ssh, http, https, domain }"
svc_UDP_HOST_Public = "{ domain }"
svc_TCP_jail_mail = "{ smtp, smtps, pop3, pop3s, imap, imaps }"

ICMPTypes = "echoreq"
AllProtocols = "{ tcp, udp, ipv6, icmp, esp, ipencap, gre }"
table <trusted> persist file "/usr/local/etc/trusted.addresses"
set loginterface $int_Trust
scrub on $int_Trust all no-df random-id reassemble tcp

nat on $int_Trust from $int_Jails:network to any -> $int_Trust
rdr pass on $int_jail_mail proto tcp from any to $int_jail_mail port \
   $svc_TCP_jail_mail -> $addr_jail_mail

block log all
pass in quick on $int_Trust from $addr_sysSvarun to any keep state
pass quick on $int_Loop all
pass quick on $int_Jails all
pass quick inet proto icmp all icmp-type $ICMPTypes keep state
pass in on $int_Trust from any to any keep state
pass out on $int_Trust from any to $addr_net_Local keep state
pass out on $int_Trust from $int_Trust to any keep state
pass out on lo1 from 192.168.223.10 to 192.168.223.10 keep state
---------------------------------------------------------------------------------

So as you can see there is a "pass quick on $int_Jails all" line. Which, as far
as I understand, should do exactly that. But, when I do
"tcpdump -n -r /var/log/pflog", I get these:

10:22:56.353027 IP 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 569935938 230047874>
10:23:06.744057 IP 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 569946122 230047874>
10:23:27.330096 IP 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 569966290 230047874>
10:23:47.918481 IP 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 569986458 230047874>
10:24:08.508126 IP 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570006626 230047874>
10:24:29.096918 IP 192.168.223.10.53777 > 192.168.223.10.80: R 1:1(0) ack 1 win 8960 <nop,nop,timestamp 570026794 230047874>
10:33:12.341285 IP 192.168.223.10.51214 > 192.168.223.10.80: F 1457218003:1457218003(0) ack 1764186631 win 8960 <nop,nop,timestamp 570539665 3540558548>
10:33:12.637811 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570539957 3540558548>
10:33:13.029827 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570540341 3540558548>
10:33:13.609705 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570540909 3540558548>
10:33:14.561443 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570541845 3540558548>
10:33:16.256344 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570543517 3540558548>
10:33:19.073348 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570546277 3540558548>
10:33:24.504722 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570551597 3540558548>
10:33:35.163039 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570562037 3540558548>
10:33:56.274140 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570582717 3540558548>
10:34:38.293842 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570623877 3540558548>
10:35:20.310801 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570665037 3540558548>
10:36:02.326561 IP 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570706197 3540558548>
10:36:44.339793 IP 192.168.223.10.51214 > 192.168.223.10.80: R 1:1(0) ack 1 win 8960 <nop,nop,timestamp 570747357 3540558548>

or, if I pass "-e -ttt" parameters to tcpdump as well, these:

rule 0/0(match): block out on lo1: 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 569935938 230047874>
rule 0/0(match): block out on lo1: 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 569946122 230047874>
rule 0/0(match): block out on lo1: 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 569966290 230047874>
rule 0/0(match): block out on lo1: 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 569986458 230047874>
rule 0/0(match): block out on lo1: 192.168.223.10.53777 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570006626 230047874>
rule 0/0(match): block out on lo1: 192.168.223.10.53777 > 192.168.223.10.80: R 1:1(0) ack 1 win 8960 <nop,nop,timestamp 570026794 230047874>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 1457218003:1457218003(0) ack 1764186631 win 8960 <nop,nop,timestamp 570539665 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570539957 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570540341 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570540909 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570541845 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570543517 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570546277 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570551597 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570562037 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570582717 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570623877 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570665037 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: F 0:0(0) ack 1 win 8960 <nop,nop,timestamp 570706197 3540558548>
rule 0/0(match): block out on lo1: 192.168.223.10.51214 > 192.168.223.10.80: R 1:1(0) ack 1 win 8960 <nop,nop,timestamp 570747357 3540558548>

Which means, pf blocks these packets. Why would this be? The consequence of
blocking these packets are, that I get this in the hosts's Apache server:

[Fri Jul 25 09:57:10 2008] [error] (1)Operation not permitted: proxy: HTTP: attempt to connect to 192.168.223.10:80 (mail) failed
[Fri Jul 25 09:57:10 2008] [error] ap_proxy_connect_backend disabling worker for (mail)
[Fri Jul 25 09:57:10 2008] [error] proxy: HTTP: disabled connection for (mail)

which disables connections to the jailed Apache for a while. Which is very
annoying. However, this (that Apache get's blocked) doesn't happen always,
I would say 10-20 times daily.

Any ideas?

Thanks,
Nejc