connect(): Operation not permitted

Kian Mohageri kian.mohageri at gmail.com
Fri Jul 4 21:30:50 UTC 2008


On Fri, Jul 4, 2008 at 4:32 AM, Jeremy Chadwick <koitsu at freebsd.org> wrote:
> On Thu, Jul 03, 2008 at 08:55:21AM -0700, Kian Mohageri wrote:
>> On Wed, Jul 2, 2008 at 5:39 PM, Stef <stef-list at memberwebs.com> wrote:
>> > Kian Mohageri wrote:
>> >> On Sun, May 18, 2008 at 3:33 AM, Johan Ström <johan at stromnet.se> wrote:
>> >>> On May 18, 2008, at 9:19 AM, Matthew Seaman wrote:
>> >>>
>> >>>> Johan Ström wrote:
>> >>>>
>> >>>>> drop all traffic)? A check with pfctl -vsr reveals that the actual rule
>> >>>>> inserted is "pass on lo0 inet from 123.123.123.123 to 123.123.123.123 flags
>> >>>>> S/SA keep state". Where did that "keep state" come from?
>> >>>> 'flags S/SA keep state' is the default now for tcp filter rules -- that
>> >>>> was new in 7.0 reflecting the upstream changes made between the 4.0 and
>> >>>> 4.1
>> >>>> releases of OpenBSD.  If you want a stateless rule, append 'no state'.
>> >>>>
>> >>>> http://www.openbsd.org/faq/pf/filter.html#state
>> >>> Thanks! I was actually looking around in the pf.conf manpage but failed to
>> >>> find it yesterday, but looking closer today I now saw it.
>> >>> Applied the no state (and quick) to the rule, and now no state is created.
>> >>> And the problem I had in the first place seems to have been resolved too
>> >>> now, even though it didn't look like a state problem.. (started to deny new
>> >>> connections much earlier than the states was full, altough maybee i wasnt
>> >>> looking for updates fast enough or something).
>> >>>
>> >>
>> >> I'd be willing to bet it's because you're reusing the source port on a
>> >> new connection before the old state expires.
>> >>
>> >> You'll know if you check the state-mismatch counter.
>> >>
>> >> Anyway, glad you found a resolution.
>> >
>> > I've been experiencing this "Operation not permitted" too. I've been
>> > trying to track down the problem for many months, but due to the
>> > complexity of my firewalls (scores of jails each with scores of rules),
>> > I wasn't brave enough to ask for help :)
>> >
>> > As a work around we started creating rules without state, whenever we
>> > would run into the problem.
>> >
>> > Thanks for the pointer about state-mismatch. The state-mismatch counter
>> > does is in fact high in my case (see below). How would I go about
>> > getting the pf state timeout and the reuse of ports for outbound
>> > connections to match? Or is this an intractable problem, that just needs
>> > to be worked around?
>>
>> Make sure your state-mismatch counter is increasing at the same times
>> you experience the problem (and isn't just high from some unrelated
>> issue).
>>
>> A similar/related problem was addressed in OpenBSD 4.3
>> (http://www.openbsd.org/plus43.html).
>>
>>   * In pf(4), allow state reuse if both sides are in FIN_WAIT_2 and a
>> new SYN arrives.
>>
>> I'm not sure if it's been imported yet.  If not, you could try tuning
>> your timeout values (see pf.conf(5)).
>>
>> The specific issue I was experienced was solved by shortening
>> tcp.closed, IIRC.  It's been a while though.
>
> When administrators see state-mismatch increasing, they get concerned.
> The common scapegoat is tcp.closed, which people don't even bother to
> describe (pf has an internal value of 10 seconds applied to that value,
> e.g. tcp.closed=5 means 15 seconds).
>
> You can set tcp.closed as low as you want, but chances are random
> Internet users will have equipment with IP stacks that re-use outbound
> sockets which haven't fully closed down within the aforementioned
> interval.  pf cannot fix this.
>
> For example, on our production/hosting systems, we see state-mismatch
> increase fairly often.  I just pfctl -F info'd our main webserver, and
> within about 15 minutes, state-mismatch was up to 22.  We use tcp.closed
> of 5 (which means 15 seconds).
>
> Workarounds such as "no state" suffice, but if you use rdr rules, you
> MUST track state, which means there's no way of winning in that case.
> For sake of example, OpenBSD spamd requires the use of rdr rules.
>
> Administrators then ask 3 questions:
>

For the sake of a helpful archive...

> 1) How do I determine whether or not state-mismatch increasing is a
>   sign of bad things, or due to peoples' broken IP stacks,

You can't.  Only way you know is probably when people complain, or you
notice scripts/page loads failing.

> 2) What happens to packets which cause state-mismatch to increment,
>   e.g. are they blocked, passed, or what?

Dropped.  In the case of a state-mismatch during TCP handshake, an RST
is sent.  That's why the failure happens immediately.

> 3) Why isn't state-mismatch described in detail in the documentation?
>

Good question.  I guess because it would be difficult to document all
of the reasons a state wouldn't match.  It would be easier to simply
document what a state _is_, but that's already in the archives.

-Kian


More information about the freebsd-net mailing list