SO_REUSEPORT: strange kernel balancer behaviour

[Adrian Chadd]

Answering my own email:

SO_REUSEPORT on FreeBSD doesn't load balance incoming connections.

Test case:

* 8 threads
* each creates a TCP socket, listening on port 1667, with SO_REUSEPORT
* only the first thread ever sees incoming requests.

I think this load distribution feature is useful to implement, but it
shouldn't be called SO_REUSEPORT.

(Silly Linux, why would you do that too..)



-adrian

On 23 July 2013 07:39, Adrian Chadd <adrian at freebsd.org> wrote:
> On 23 July 2013 00:09, trafdev <trafdev at mail.ru> wrote:
>> It's like shared acceptor FD and N processes:
>
> [snip] looks like mine, but I use threads.
>
>> Accept conn callback is called in N processes on each connection, only one
>> wins,
>> others exit by errno == EAGAIN case. Overhead is almost zero.
>> Problem is that "wins" distribution is far from equal.
>
> Right. I'm not at that stage yet, but I can totally see that happening.
>
> Ok. Time to hit up the TCP stack people to weigh in on the recent lkml
> posts about this:
>
> http://lwn.net/Articles/542629/
>
> With SO_REUSEPORT, we should create one listen FD per thread, and let
> the OS balance how that gets distributed. Rather than one listen
> socket that is shared between all processes/threads. I'll try that
> locally and see if that works right. Would you mind trying it locally
> and see if it improves the distribution of work?
>
> Thanks,
>
>
> -adrian