connect(): Operation not permitted

[Jeremy Chadwick]

On Thu, Jul 03, 2008 at 08:55:21AM -0700, Kian Mohageri wrote:
> On Wed, Jul 2, 2008 at 5:39 PM, Stef <stef-list at memberwebs.com> wrote:
> > Kian Mohageri wrote:
> >> On Sun, May 18, 2008 at 3:33 AM, Johan Ström <johan at stromnet.se> wrote:
> >>> On May 18, 2008, at 9:19 AM, Matthew Seaman wrote:
> >>>
> >>>> Johan Ström wrote:
> >>>>
> >>>>> drop all traffic)? A check with pfctl -vsr reveals that the actual rule
> >>>>> inserted is "pass on lo0 inet from 123.123.123.123 to 123.123.123.123 flags
> >>>>> S/SA keep state". Where did that "keep state" come from?
> >>>> 'flags S/SA keep state' is the default now for tcp filter rules -- that
> >>>> was new in 7.0 reflecting the upstream changes made between the 4.0 and
> >>>> 4.1
> >>>> releases of OpenBSD.  If you want a stateless rule, append 'no state'.
> >>>>
> >>>> http://www.openbsd.org/faq/pf/filter.html#state
> >>> Thanks! I was actually looking around in the pf.conf manpage but failed to
> >>> find it yesterday, but looking closer today I now saw it.
> >>> Applied the no state (and quick) to the rule, and now no state is created.
> >>> And the problem I had in the first place seems to have been resolved too
> >>> now, even though it didn't look like a state problem.. (started to deny new
> >>> connections much earlier than the states was full, altough maybee i wasnt
> >>> looking for updates fast enough or something).
> >>>
> >>
> >> I'd be willing to bet it's because you're reusing the source port on a
> >> new connection before the old state expires.
> >>
> >> You'll know if you check the state-mismatch counter.
> >>
> >> Anyway, glad you found a resolution.
> >
> > I've been experiencing this "Operation not permitted" too. I've been
> > trying to track down the problem for many months, but due to the
> > complexity of my firewalls (scores of jails each with scores of rules),
> > I wasn't brave enough to ask for help :)
> >
> > As a work around we started creating rules without state, whenever we
> > would run into the problem.
> >
> > Thanks for the pointer about state-mismatch. The state-mismatch counter
> > does is in fact high in my case (see below). How would I go about
> > getting the pf state timeout and the reuse of ports for outbound
> > connections to match? Or is this an intractable problem, that just needs
> > to be worked around?
> 
> Make sure your state-mismatch counter is increasing at the same times
> you experience the problem (and isn't just high from some unrelated
> issue).
> 
> A similar/related problem was addressed in OpenBSD 4.3
> (http://www.openbsd.org/plus43.html).
> 
>   * In pf(4), allow state reuse if both sides are in FIN_WAIT_2 and a
> new SYN arrives.
> 
> I'm not sure if it's been imported yet.  If not, you could try tuning
> your timeout values (see pf.conf(5)).
> 
> The specific issue I was experienced was solved by shortening
> tcp.closed, IIRC.  It's been a while though.

When administrators see state-mismatch increasing, they get concerned.
The common scapegoat is tcp.closed, which people don't even bother to
describe (pf has an internal value of 10 seconds applied to that value,
e.g. tcp.closed=5 means 15 seconds).

You can set tcp.closed as low as you want, but chances are random
Internet users will have equipment with IP stacks that re-use outbound
sockets which haven't fully closed down within the aforementioned
interval.  pf cannot fix this.

For example, on our production/hosting systems, we see state-mismatch
increase fairly often.  I just pfctl -F info'd our main webserver, and
within about 15 minutes, state-mismatch was up to 22.  We use tcp.closed
of 5 (which means 15 seconds).

Workarounds such as "no state" suffice, but if you use rdr rules, you
MUST track state, which means there's no way of winning in that case.
For sake of example, OpenBSD spamd requires the use of rdr rules.

Administrators then ask 3 questions:

1) How do I determine whether or not state-mismatch increasing is a
   sign of bad things, or due to peoples' broken IP stacks,
2) What happens to packets which cause state-mismatch to increment,
   e.g. are they blocked, passed, or what?
3) Why isn't state-mismatch described in detail in the documentation?

Finally, the fix in OpenBSD 4.3 should really be backported to FreeBSD
ASAP.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |