kern/121668: connect randomly fails with EPERM with some pf rules

[Laurent Frigault]

>Number:         121668
>Category:       kern
>Synopsis:       connect randomly fails with EPERM with some pf rules
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Mar 13 16:50:01 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Laurent Frigault
>Release:        6.2-RELEASE-p10 , 7.0-RELEASE
>Organization:
>Environment:
FreeBSD troll.free.org 6.2-RELEASE-p10 FreeBSD 6.2-RELEASE-p10 #0: Wed Jan 16 14:22:17 CET 2008     lolo at troll.free.org:/usr/src/sys/i386/compile/SMP  i386

FreeBSD surt.free.org 7.0-RELEASE FreeBSD 7.0-RELEASE #1: Wed Feb 27 18:29:25 CET 2008     root at surt.free.org:/usr/src/sys/amd64/compile/GENERIC  amd64

>Description:
>From times to times, connect fails with EPERM when using pf statefully.

I discover this problem when investigating the cause of unexpected mysql connection failure between a web php script and and mysql server running on an other server. This leads me to a connect(2) problem failing without reason with EPERM (there is no EPERM failure cause in connect manual) .

ruleset1 (no state was the default before 7.0):
==============================================
scrub in all fragment reassemble
 
pass out quick on lo0 all no state
pass in quick on lo0 all no state
..
==============================================
ruleset 2

==============================================
scrub in all fragment reassemble

pass out quick on lo0 proto tcp from any to any port 9 flags S/SA keep state
pass out quick on lo0 all no state
pass in quick on lo0 all no state
==============================================

With ruleset 1 => no problem
With ruleset 2 => connect fails sometimes with EPERM

There is no reject information in pf logs which is logical because pf rules authorize those connections

>How-To-Repeat:
sysctl net.inet.tcp.nolocaltimewait=1
not needed, but helps to reproduce the problem with client and server on the same computer.

start inetd with discard/tcp service enabled :

inetd_enable="YES"
inetd_flags="-wl -R 0"

% grep ^discard /etc/inetd.conf 
discard stream  tcp     nowait  root    internal

pf rules:
scrub in all fragment reassemble

pass out quick on lo0 proto tcp from any to any port 9 flags S/SA keep state
pass out quick on lo0 all no state
pass in quick on lo0 all no state

Lauch the following perl script.

Sometimes, connect will wrongly fail with EPERM
==============================================================
#!/usr/bin/perl -w

use strict;

use Socket;
use Errno;

$|=1;

sub con($$$)
{
        my ($sin,$port,$proto) = @_;

        socket(Socket_Handle, PF_INET, SOCK_STREAM, $proto);
        if(connect(Socket_Handle,$sin))
        {
                print "ok\t";
                print Socket_Handle "hello\n";
                close (Socket_Handle);
        }
        else
        {
                print "$!\n";
        };
}
  
my $proto =  getprotobyname('tcp');
my $port = getservbyname('discard', 'tcp');
my $sin = sockaddr_in($port,inet_aton("127.1"));
   
for (my $cpt=0;$cpt<=2000;++$cpt)
{
        print "$cpt\t";
        con($sin,$port,$proto);
};
==============================================================

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: