svn commit: r263198 - in head/sys: amd64/conf conf net netinet netinet6 sys

[Adrian Chadd]

On 17 March 2014 01:26, Robert N. M. Watson <rwatson at freebsd.org> wrote:
>
> On 17 Mar 2014, at 07:54, David Malone <dwmalone at maths.tcd.ie> wrote:
>
>>>  (1) Merge a software implementation of the Toeplitz hash specified in
>>>      RSS implemented by David Malone.  This is used to allow suitable
>>>      pcbgroup placement of connections before the first packet is
>>>      received from the NIC.  Software hashing is generally avoided,
>>>      however, due to high cost of the hash on general-purpose CPUs.
>>
>> I could look at a faster software implementation, but I guess most of
>> the value is when hashing is done on the NIC.
>
> In my benchmarking (a couple of years ago) the software path never really turned up a lot. If you end up with a flow migrating from an RSS NIC to a non-RSS NIC, you fall back on the 'reservation' hash table, using the conventional hash and picking up contention, but you don't end up doing the software version of Toeplitz per-packet. The only reason we need a software implementation (currently) is to do an initial placement of a new outbound flow into the hash table prior to receiving a packet with the hardware-generated hash on it. For the inbound direction, we can pick it up from the first packet. (Although, actually, we also need to do it for inbound flows that come from non-RSS NICs -- or localhost or such).

If it's done right, then yes, we should only see a per-packet RSS hash on:

* NICs / paths that don't do it in hardware on the receive side;
* Doing per-packet sanity checking - ie, doing a software RSS as well
as the hardware RSS in order to verify that they both line up.

I plan on adding the latter at some point soon so we can ensure that
things end up on the right queue.

For higher connection rate workloads, we want to ensure that the
initial hash selection for outbound matches the inbound, as we don't
want to be grabbing non-local-CPU locks to migrate connections to
other pcbgroups after they've been established. We absolutely want to
keep things on the local CPU as much as we can.

Thanks!


-a