connection refused on heavy usage

[Gergely CZUCZY]

Good morning,

I've got a problem that disappeared by disabling pf.

From the beginning. I'm testing an http reverse proxy[pound], at
the moment. I've got two gateways in a pfsync+carp+pound configuration
and two web backends. I'm doing performance testing on the proxy with
apache benchmarks, this involves hordes of simultaneous connections in
and out. connections are recieved by pound on the gateway and it
connects to a given web-backend to make the actual request.

The problem is that, periodically it's unable to connect to some
backends, or just to one of them and renders it DEAD.
When this happens there's a "connect: operation not permitted" message
in the syslog. Nor I'm able to connect to the backends directly with
elinks from the gateway, it also says "operation not permitted". After
waiting a few seconds it works again.

So, the proxy can accept client's connections but it's unable to
connect forward to the actual web-backends. When I disabled pf with
pfctl -d these symptons stopped immedietly.

I tried playing around with different tcp timeout values, but that
failed to help.

My pf.conf is the following:
--- chop with axe here ---
if_ext="em0"
if_vvv="fxp0"
if_sync="em1"

ip_pub="192.168.4.55"
ip_vvv="10.0.0.254"

ip_vvv1="10.0.0.1"
ip_vvv2="10.0.0.2"
ip_vvv3="10.0.0.3"

table <vvv> {$ip_vvv1, $ip_vvv2, $ip_vvv3}

# Options: tune the behavior of pf, default values are given.
#set timeout { interval 5, frag 30 }
#set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
set timeout { tcp.closing 900, tcp.finwait 30, tcp.closed 60 }
#set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
#set timeout { icmp.first 20, icmp.error 10 }
#set timeout { other.first 60, other.single 30, other.multiple 60 }
set timeout { adaptive.start 70000, adaptive.end 120000 }
set limit { states 100000, frags 2000 }
#set loginterface none
set block-policy return
set require-order yes
set fingerprints "/etc/pf.os"
set debug misc

set skip on lo0

#scrub in all

rdr on $if_ext proto tcp from any to $ip_pub port 10001 -> $ip_vvv1 port 22
rdr on $if_ext proto tcp from any to $ip_pub port 10002 -> $ip_vvv2 port 22
rdr on $if_ext proto tcp from any to $ip_pub port 10003 -> $ip_vvv3 port 22

block in log on $if_ext all

pass in quick on {$if_ext,$if_vvv} proto vrrp
pass out quick on {$if_ext,$if_vvv} proto vrrp

pass out quick on $if_ext proto udp from any to 192.168.4.200 port 123 keep state

pass in quick on $if_ext proto tcp from any to $if_ext:0 port 22 flags S/SA synproxy state (no-sync)
pass in quick on $if_ext proto tcp from any to $ip_pub port 80 flags S/SA modulate state (no-sync) label "2"

pass out quick on $if_ext proto udp from $if_ext:0 to port 53 keep state (no-sync)
pass out quick on $if_ext proto udp from any to port 53 keep state

pass out quick on $if_ext proto tcp from $if_ext:0 to port 80 flags S/SA keep state (no-sync)
pass out quick on $if_ext proto tcp from any to port 80 flags S/SA keep state

pass in quick on $if_ext proto tcp from any to <vvv> port 22 flags S/SA synproxy state

pass out quick on $if_vvv proto tcp from ($if_vvv) to <vvv> port 80 flags S/SA keep state (no-sync)
--- chop with axe here ---

FreeBSD lvs1.in.publishing.hu 6.2-RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #1: Tue Jul 24 08:07:07 UTC 2007     toor at pointyhat.office:/usr/obj/usr/src/sys/LVS  i386

I'm played with the followings without any success:
set timeout { tcp.closing 900, tcp.finwait 30, tcp.closed 60 }
set timeout { adaptive.start 70000, adaptive.end 120000 }

What can cause this issue?
How could this be fixed?

[pound] http://www.apsis.ch/pound/

Sincerely,

Gergely Czuczy
mailto: gergely.czuczy at harmless.hu

-- 
Weenies test. Geniuses solve problems that arise.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 2563 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-pf/attachments/20070725/c766fff4/attachment.pgp