Best practice for accepting TCP connections on multicore?

[Hooman Fazaeli]

On 6/7/2014 6:32 AM, Adrian Chadd wrote:
> Hi,
>
> In the RSS model I'm hacking on:
>
> * you query the kernel for the current RSS bucket -> CPU mapping.
> Right now it's one bucket -> one CPU but at some point it may be one
> bucket -> cpuset;
> * you spawn a thread for each RSS bucket;
> * you pin each thread to the RSS bucket current CPU id;
> * you create a listen socket in that thread, marked as BINDMULTI (ie,
> multiple things can listen on this and the kernel will load balance
> between them) and RSS_BUCKET (ie, please place this socket in the
> given RSS bucket, rather than the global/wildcard list);
> * then when a connection comes in, the kernel will first do a lookup
> for a matching wildcard socket in the per-RSS PCBGROUP, rather than
> the global wildcard table;
> * if it finds it, that socket gets the incoming connection.
>
> At some point I'll add some notification via kqueue or what not that
> the RSS buckets need rebalancing, and userland can then re-pin the
> per-bucket threads.
>
> At the moment the hacks I've done only support one listen socket per
> entry. My hope is that BINDMULTI will do some basic hash to load
> balance within a set of matching PCB entries - and it'll be combined,
> so if you do BINDMULTI without RSS, it'll just load balance between
> multiple sockets with no CPU affinity knowledge. If you do RSS, it'll
> distribute only CPU-local requests to a thread that's sitting in the
> right RSS bucket. if you enable both, you can use a thread pool for
> each RSS bucket CPU and (eventually, when I write it) it'll load
> balance among those.
>
> But for now I'm assuming one incoming thread per RSS bucket will be
> enough for people to experiment with.
>
> anyway, I guess I should email out the details:
>
> * http://github.com/erikarn/freebsd - the 'local/rss' branch has the
> RSS changes to dev/e1000/if_igb.c and netinet/
would you please point to the exact URL for the branch?
> * http://github.com/erikarn/freebsd-rss - has some RSS examples. Look
> at rss-http.
>
> I haven't yet tested this at > 1GE because all I have at home are igb
> and em NICs. If someone would like to donate ixgbe and T4 hardware,
> i'll gratefully take it and do up RSS patches for those drivers.
>
>

-- 

Best regards.
Hooman Fazaeli