connection refused on heavy usage

Wed Jul 25 11:49:23 UTC 2007

Just catched something in the syslog:
Jul 25 13:46:14 lvs1 kernel: pf: BAD state: TCP 10.0.0.251:53051 10.0.0.251:53051 10.0.0.1:80 [lo=2626749835 high=2626816443 win=33304 modulator=0 wscale=1] [lo=2986152604 high=2986219211 win=33304 modulator=0 wscale=1] 9:9 S seq=2736349746 ack=2986152604 len=0 ackskew=0 pkts=23:26 dir=out,fwd
Jul 25 13:46:14 lvs1 kernel: pf: State failure on: 1       | 5
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not permitted
Jul 25 13:46:14 lvs1 kernel: pf: BAD state: TCP 10.0.0.251:53052 10.0.0.251:53052 10.0.0.1:80 [lo=368073977 high=368140585 win=33304 modulator=0 wscale=1] [lo=530543602 high=530610209 win=33304 modulator=0 wscale=1] 9:9 S seq=477665814 ack=530543602 len=0 ackskew=0 pkts=24:26 dir=out,fwd
Jul 25 13:46:14 lvs1 kernel: pf: State failure on: 1       | 5
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not permitted
Jul 25 13:46:14 lvs1 kernel: pf: BAD state: TCP 10.0.0.251:53053 10.0.0.251:53053 10.0.0.1:80 [lo=3069306042 high=3069372650 win=33304 modulator=0 wscale=1] [lo=1682247531 high=1682314138 win=33304 modulator=0 wscale=1] 9:9 S seq=3178900053 ack=1682247531 len=0 ackskew=0 pkts=23:26 dir=out,fwd
Jul 25 13:46:14 lvs1 kernel: pf: State failure on: 1       | 5
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not permitted
Jul 25 13:46:14 lvs1 last message repeated 40 times
Jul 25 13:46:14 lvs1 pound: no back-end "GET /phpinfo-lycos.html HTTP/1.0" from 192.168.4.21
Jul 25 13:46:14 lvs1 pound: no back-end "GET /phpinfo-lycos.html HTTP/1.0" from 192.168.4.21
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not permitted
Jul 25 13:46:14 lvs1 pound: no back-end "GET /phpinfo-lycos.html HTTP/1.0" from 192.168.4.21
Jul 25 13:46:14 lvs1 pound: backend 10.0.0.1:80 connect: Operation not permitted
Jul 25 13:46:14 lvs1 pound: no back-end "GET /phpinfo-lycos.html HTTP/1.0" from 192.168.4.21
Jul 25 13:46:17 lvs1 last message repeated 681 times
Jul 25 13:46:17 lvs1 pound: BackEnd 10.0.0.1:80 resurrect

As i see these tend to happen from time to time.

On Wed, Jul 25, 2007 at 01:38:24PM +0200, Gergely CZUCZY wrote:
> Good morning,
> 
> I've got a problem that disappeared by disabling pf.
> 
> From the beginning. I'm testing an http reverse proxy[pound], at
> the moment. I've got two gateways in a pfsync+carp+pound configuration
> and two web backends. I'm doing performance testing on the proxy with
> apache benchmarks, this involves hordes of simultaneous connections in
> and out. connections are recieved by pound on the gateway and it
> connects to a given web-backend to make the actual request.
> 
> The problem is that, periodically it's unable to connect to some
> backends, or just to one of them and renders it DEAD.
> When this happens there's a "connect: operation not permitted" message
> in the syslog. Nor I'm able to connect to the backends directly with
> elinks from the gateway, it also says "operation not permitted". After
> waiting a few seconds it works again.
> 
> So, the proxy can accept client's connections but it's unable to
> connect forward to the actual web-backends. When I disabled pf with
> pfctl -d these symptons stopped immedietly.
> 
> I tried playing around with different tcp timeout values, but that
> failed to help.
> 
> My pf.conf is the following:
> --- chop with axe here ---
> if_ext="em0"
> if_vvv="fxp0"
> if_sync="em1"
> 
> ip_pub="192.168.4.55"
> ip_vvv="10.0.0.254"
> 
> ip_vvv1="10.0.0.1"
> ip_vvv2="10.0.0.2"
> ip_vvv3="10.0.0.3"
> 
> table <vvv> {$ip_vvv1, $ip_vvv2, $ip_vvv3}
> 
> # Options: tune the behavior of pf, default values are given.
> #set timeout { interval 5, frag 30 }
> #set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
> set timeout { tcp.closing 900, tcp.finwait 30, tcp.closed 60 }
> #set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
> #set timeout { icmp.first 20, icmp.error 10 }
> #set timeout { other.first 60, other.single 30, other.multiple 60 }
> set timeout { adaptive.start 70000, adaptive.end 120000 }
> set limit { states 100000, frags 2000 }
> #set loginterface none
> set block-policy return
> set require-order yes
> set fingerprints "/etc/pf.os"
> set debug misc
> 
> set skip on lo0
> 
> #scrub in all
> 
> rdr on $if_ext proto tcp from any to $ip_pub port 10001 -> $ip_vvv1 port 22
> rdr on $if_ext proto tcp from any to $ip_pub port 10002 -> $ip_vvv2 port 22
> rdr on $if_ext proto tcp from any to $ip_pub port 10003 -> $ip_vvv3 port 22
> 
> block in log on $if_ext all
> 
> pass in quick on {$if_ext,$if_vvv} proto vrrp
> pass out quick on {$if_ext,$if_vvv} proto vrrp
> 
> pass out quick on $if_ext proto udp from any to 192.168.4.200 port 123 keep state
> 
> pass in quick on $if_ext proto tcp from any to $if_ext:0 port 22 flags S/SA synproxy state (no-sync)
> pass in quick on $if_ext proto tcp from any to $ip_pub port 80 flags S/SA modulate state (no-sync) label "2"
> 
> pass out quick on $if_ext proto udp from $if_ext:0 to port 53 keep state (no-sync)
> pass out quick on $if_ext proto udp from any to port 53 keep state
> 
> pass out quick on $if_ext proto tcp from $if_ext:0 to port 80 flags S/SA keep state (no-sync)
> pass out quick on $if_ext proto tcp from any to port 80 flags S/SA keep state
> 
> pass in quick on $if_ext proto tcp from any to <vvv> port 22 flags S/SA synproxy state
> 
> pass out quick on $if_vvv proto tcp from ($if_vvv) to <vvv> port 80 flags S/SA keep state (no-sync)
> --- chop with axe here ---
> 
> FreeBSD lvs1.in.publishing.hu 6.2-RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #1: Tue Jul 24 08:07:07 UTC 2007     toor at pointyhat.office:/usr/obj/usr/src/sys/LVS  i386
> 
> I'm played with the followings without any success:
> set timeout { tcp.closing 900, tcp.finwait 30, tcp.closed 60 }
> set timeout { adaptive.start 70000, adaptive.end 120000 }
> 
> What can cause this issue?
> How could this be fixed?
> 
> [pound] http://www.apsis.ch/pound/
> 
> Sincerely,
> 
> Gergely Czuczy
> mailto: gergely.czuczy at harmless.hu
> 
> -- 
> Weenies test. Geniuses solve problems that arise.

Sincerely,

Gergely Czuczy
mailto: gergely.czuczy at harmless.hu

-- 
Weenies test. Geniuses solve problems that arise.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 3360 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-pf/attachments/20070725/92b30109/attachment.pgp