Adding Flow Director sysctls to ixgbe(4)

[George Neville-Neil]

On Sep 8, 2011, at 14:49 , Navdeep Parhar wrote:

> On Thu, Sep 08, 2011 at 08:34:11AM -0400, John Baldwin wrote:
>> On Monday, September 05, 2011 7:21:12 am Ben Hutchings wrote:
>>> On Mon, 2011-09-05 at 15:51 +0900, Takuya ASADA wrote:
>>>> Hi,
>>>> 
>>>> I implemented Ethernet Flow Director sysctls to ixgbe(4), here's a detail:
>>>> 
>>>> - Adding removing signature filter
>>>> On linux version of ixgbe driver, it has ability to set/remove perfect
>>>> filter from userland using ethtool command.
>>>> I implemented similar feature, but on sysctl, and not perfect filter
>>>> but signature filter(which means hash collision may occurs).
>>> [...]
>>> 
>>> Linux also has a generic interface to RX filtering and hashing
>>> (ethtool_rxnfc) which ixgbe supports; wouldn't it be better for FreeBSD
>>> to support something like that?
>> 
>> Some sort of shared interface might be nice.  The cxgb(4) and cxgbe(4) drivers
>> both provide their own tools to manipulate filters, though they do not
>> provide explicit steering IIRC.
> 
> Both of them can filter as well as steer (and the tools let you do that).
> cxgbe(4) can do a lot more (rewrite + switch, replicate, etc.) but those
> features are perhaps too specialized to be configurable via a general
> purpose tool.
> 
>> 
>> We would need to come up with some sort of standard interface (ioctls?) for 
>> adding filters however.
> 
> +1 for a standard interface.
> 
> imho the kernel needs to be aware of the rx and tx queues of a NIC, and
> not just for steering.  But that's a separate discussion.
> 

Well I do think this is actually all of a part.  Most of us realize by now that
high speed (e.g. 10G and higher) NICs only make sense if you can steer traffic and
pin queues to cores etc.  Without those pieces in place it's impossible to get
the full utilization out of the NIC because things like cache misses just kill
your performance.  Having looked at a few 10G NICs in the last couple of years the
one thing I notice is that everyone wants to do things like offload and steering
but that the specifics are quite different between different cards.  Some of that
has to do with the libraries that expose these things to the kernel and user programs,
but some has to do with how the device is modeled.  What this means is that we have
a failure of abstraction.  Abstraction has a cost, and some of the people who want
access to low level queues are not interested in paying an extra abstraction cost.

I think that some of the abstractions we need are tied up in the work that Takuya did
for SoC and some of it is in the work done by Luigi on netmap.  I'd go so far as to say
that what we should do is try to combine those two pieces of code into a set of
low level APIs for programs to interact with high speed NICs.  The one thing most
people do not talk about is extending our socket API to do two things that I think would
be a win for 80% of our users.  If a socket, and also a kqueue, could be pinned
to a CPU as well as a NIC queue that should improve overall bandwidth for a large
number of our users.  The API there is definitely an ioctl() and the hard part is
doing the tying together.  To do this we need to also work out our low level story.

Best,
George