connect(): Operation not permitted

Sat May 17 16:01:19 UTC 2008

First of all, for freebsd-pf subscribers, I posted my original problem  
(in the bottom) to freebsd-net earlier, but replies seems to point to  
PF so I'll CC there too..

On May 17, 2008, at 5:19 PM, Alex Trull wrote:

> Hi Johan and List,
>
> In my case a few months ago it was pahu. Don't give that fine fellow  
> an
> account on your precious system !

>
>
> But seriously, I had a pf-firewalled jail being being used for DNS
> testing, with large numbers of udp "connections" hanging around in pf
> state. While the default udp timeout settings in PF are lower than  
> those
> of the tcp timeouts, it is was still too high for it to to remove the
> states in time before hitting the default 10k state limit!
>
> If this is the case with you - run 'pfctl -s state | wc -l' - when  
> there
> is traffic load you may see that hitting 10k states if you've not  
> tuned
> that variable.
>
> What to do next - up the state limit or lower the state timeouts. I  
> did
> both, to be safe.
>
> in /etc/pf.conf these must be at the very top of the file:
>
> # options
> # 10k is insanely low, lets raise it..
> set limit { frags 16384, states 32768 }
> # timeouts - see 'pfctl -s timeouts' for options - you will want to
> # change the tcp ones rather than the udp ones for your smtp setup.
> # but these are mine, I set them for the dns traffic.
> set timeout { udp.first 15, udp.single 5, udp.multiple 30 }
>
>
> don't forget to:
>
> $ /etc/rc.d/pf check && /etc/rc.d/pf reload

Ok, looked over the PF states now, but I'm not quite sure thats what  
causing it. I have default limit on 10k states, normally I seem to  
have around ~800 states, and when I start my test script that tries to  
send as many mails as possible (using PHP's Pear::Mail, creating a  
connection, sending, disconnecting, creating new connection.. and so  
on), I can clearly see the PF state counter (pfctl -vsi) increase, but  
the script aborts with Operation not permitted way before I hit 10k,  
its rather around 3-4k..
If I then wait a few seconds and run the script again, I can see the  
number of states increase even more, and if I do this enough times I  
finally hit around 9700 states. But at this point (states exhausted),  
I don't get Operation not permitted, instead it just seems that the  
script blocks up a few seconds while states clear up, then continues  
running until it gets a Operation not permitted.

So, from the above results, I cant say that it looks like its the  
states?

Just tried to disable the altq rule now too, no changes (not that I  
expected one, since its on bce0 not lo0).

Another thing, which might be more approriate in freebsd-pf though..  
Why would it create states at all for this traffic, when my pf.conf  
rule is "pass on lo0 inet from $jail to $jail" (i have a block drop in  
rule to drop all traffic)? A check with pfctl -vsr reveals that the  
actual rule inserted is "pass on lo0 inet from 123.123.123.123 to  
123.123.123.123 flags S/SA keep state". Where did that "keep state"  
come from?

Thanks for ideas :)

>
>
> HTH,
>
> Alex
>
> On Sat, 2008-05-17 at 16:33 +0200, Johan Ström wrote:
>> Hello
>>
>> I got a FreeBSD 7 machine running mail services (among other things).
>> This machine recently replaced a FreeBSD 6.2 machine doing the same
>> tasks.
>> Now and then I need to send alot of mail to customers (mailing list),
>> and one thing i've noticed now after the change is that when I use a
>> lot of connections subsequently (high connection rate, even if they
>> are very shortlived) inside a jail (dunno if that has anything to do
>> with it though), I start to get Operation not permitted in return to
>> connect().
>> I've seen this in the PHP app that sends mail, when it tried to
>> connect to localhost, as well as from postfix when it have been  
>> trying
>> to connect to amavisd on localhost, but also from postfix when it has
>> tried to connect to remote SMTP servers.
>>
>> I do have PF for filtering, but there are no max-src-conn-rate limits
>> enabled for any rules that is used for this. However, from one of the
>> jail I do have a hfsc queue limiting the outgoing mail traffic from
>> one jailed IP. But I'm not sure that this would be the problem, since
>> I've also seen the problem when doing localhost connects in the jail,
>> and also in other jails on an entierly different IP that is not
>> affected.
>>
>> Does anyone have any clues about what I can look at and tune to fix
>> this?
>>
>> Thanks!
>>
>> --
>> Johan Ström
>> Stromnet
>> johan at stromnet.se
>> http://www.stromnet.se/
>>
>>
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org 
>> "