occasional ECONNREFUSED when connect()ing.

[Deomid Ryabkov]

I have a strange case of occasional refused connect()'s.
The system is running 4.10-STABLE.

This was originally reported by one of our developers.
The test script he provided is a simple Perl script that repeatedly 
fetches an URL
from Apache running on this same server.
In the run of 2000 iterations, there are 5-10 random errors fetching the 
URL.
After further investigation and digging, I uncovered the true cause of 
the error:
connect() returning ECONNREFUSED.
To see if this was somehow related to Apache, I wrote my own simple 
server that
accept()'s a connection, read()'s what supposed to be request and 
write()'s a stereotypic reply,
thus resembling an HTTP request/reply conversation.
The number of sporadically occuring connect() errors has increased somewhat,
maybe due to quicker turnaround of my stub-server.
The question is - why do connections get refused at all?
I can think of no valid reasons...

here's the output of the client as i see it:

----------------------------- sample output of x.pl 
-----------------------------
71590, http://localhost:8000/junk, 2000: starting.
59 500 Can't connect to localhost:8000 (Timeout connect: Connection refused)
127 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
213 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
259 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
363 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
402 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
416 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
690 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
703 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
754 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
947 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
960 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
997 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
1163 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
1272 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
1433 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
1612 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
1887 500 Can't connect to localhost:8000 (Timeout connect: Connection 
refused)
----------------------------- cut -----------------------------
(the confusing "Timeout connect:" message is a result of my 
IO/Socket/INET.pm hacking.
without it, just a confusing "Timeout" was reported. i added printing of 
actual error string.)

a fragment of output of the same script running under strace:

----------------------------- strace of x.pl -----------------------------
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
connect(3, {sa_family=AF_INET, sin_port=htons(8000), 
sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)
----------------------------- cut -----------------------------

and the source of both server and client used:

----------------------------- accepter.c (server) 
-----------------------------
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include <stdio.h>

char *reply = "HTTP/1.0 200 OK\r\nConetnt-type: 
text/plain\r\n\r\nNothing special.\r\n";
int rlen;

int main(int argc, char **argv) {

        int i, cal;
        int s, ss, n;
        char *buf;
        struct sockaddr_in saddr, caddr;

        memset(&saddr, 0, sizeof saddr);
        saddr.sin_family = AF_INET;
        saddr.sin_addr.s_addr = htonl(INADDR_ANY);
        saddr.sin_port = htons(8000);

        ss = socket(PF_INET, SOCK_STREAM, 6);
        if (ss < 0) return 1;
        if (bind(ss, (struct sockaddr *) &saddr, sizeof saddr) < 0) 
return 2;
        if (listen(ss, 511) < 0) return 3;

        buf = (char *) malloc(4096);
        if (buf == NULL) return 4;

        rlen = strlen(reply);

        fprintf(stderr, "accepting.\n");

        for (cal = sizeof caddr;
             s = accept(ss, (struct sockaddr *) &caddr, &cal);
             cal = sizeof caddr) {

                n = read(s, buf, 4096);
                if (n > 0) {
                        write(s, reply, rlen);
                }
                close(s);
        }

        return (s < 0);
}
----------------------------- cut -----------------------------

----------------------------- x.pl (client) -----------------------------
#!/usr/bin/perl

$| = 1;

use LWP::UserAgent;

$url = 'http://localhost:8000/junk';

my $n = 2000;

print "$$, $url, $n: starting.\n";

for (my $i = 0; $i < $n; $i++) {
        my $ua = LWP::UserAgent->new;
        $ua->timeout(1000);
        $req = HTTP::Request->new("GET" => $url);
        $res = $ua->request($req);

        print $i." ".$res->status_line."\n" if (0 or $res->code == 500);
}

exit();
----------------------------- cut -----------------------------

--
Deomid Ryabkov aka Rojer
myself at rojer.pp.ru
rojer at sysadmins.ru
ICQ: 8025844

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3228 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20050218/685d5235/smime.bin