igb(4) at peak in big purple

[Jack Vogel]

I suspect to do it right would involve having the stack/kernel have more
interaction with the driver/interface data, and this IS the way RSS was
envisioned to work. Its been talked about but hasn't happened so far.

Jack


On Fri, Apr 27, 2012 at 1:00 PM, Juli Mallett <jmallett at freebsd.org> wrote:

> On Fri, Apr 27, 2012 at 12:29, Sean Bruno <seanbru at yahoo-inc.com> wrote:
> > On Thu, 2012-04-26 at 11:13 -0700, Juli Mallett wrote:
> >> Queue splitting in Intel cards is done using a hash of protocol
> >> headers, so this is expected behavior.  This also helps with TCP and
> >> UDP performance, in terms of keeping packets for the same protocol
> >> control block on the same core, but for other applications it's not
> >> ideal.  If your application does not require that kind of locality,
> >> there are things that can be done in the driver to make it easier to
> >> balance packets between all queues about-evenly.
> >
> > Oh? :-)
> >
> > What should I be looking at to balance more evenly?
>
> Dirty hacks are involved :)  I've sent some code to Luigi that I think
> would make sense in netmap (since for many tasks one's going to do
> with netmap, you want to use as many cores as possible, and maybe
> don't care about locality so much) but it could be useful in
> conjunction with the network stack, too, for tasks that don't need a
> lot of locality.
>
> Basically this is the deal: the Intel NICs hash of various header
> fields.  Then, some bits from that hash are used to index a table.
> That table indicates what queue the received packet should go to.
> Ideally you'd want to use some sort of counter to index that table and
> get round-robin queue usage if you wanted to evenly saturate all
> cores.  Unfortunately there doesn't seem to be a way to do that.
>
> What you can do, though, is regularly update the table that is indexed
> by hash.  Very frequently, in fact, it's a pretty fast operation.  So
> what I've done, for example, is to go through an rotate all of the
> entries every N packets, where N is something like the number of
> receive descriptors per queue divided by the number of queues.  So
> bucket 0 goes to queue 0 and bucket 1 goes to queue 1 at first.  Then
> a few hundred packets are received, and the table is reprogrammed, so
> now bucket 0 goes to queue 1 and bucket 1 goes to queue 0.
>
> I can provide code to do this, but I don't want to post it publicly
> (unless it is actually going to become an option for netmap) for fear
> that people will use it in scenarios where it's harmful and then
> complain.  It's potentially one more painful variation for the Intel
> drivers that Intel can't support, and that just makes everyone
> miserable.
>
> Thanks,
> Juli.
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>