FreeBSD I/OAT (QuickData now?) driver
kmacy at freebsd.org
Tue Jun 7 12:15:34 UTC 2011
All 10GigE NICs and some newer 10 GigE NICs have multiple hardware
queues with a separate MSI-x vector per queue, where each vector is
directed to a different CPU. The current operating model is to have a
separate interrupt thread per vector. This obviously gets bogged down
if one has multiple cards as the interrupt threads end up requiring
the scheduler to distribute work fairly between cards as multiple
threads will end up running on the same CPUs. Nokia had a reasonable
interface for coping with this that was reminiscent of NAPI whereby
cooperative sharing between interfaces was provided by having a single
taskqueue thread per-core and the cards would queue tasks (which would
be re-queued if more than a certain amount of work were required) as
interrupts were delivered. There has been talk off and on of porting
this "net_task" interface to freebsd.
None of this addresses PF_RING's facility for pushing packets in to
userland - but presumably Rizzo's netmap work addresses those in need
of that sufficiently.
On Tue, Jun 7, 2011 at 4:13 AM, grarpamp <grarpamp at gmail.com> wrote:
> Is this work part of what's needed to enable the FreeBSD
> equivalent of TNAPI?
> I know we've got polling. And probably MSI-X in a couple drivers.
> Pretty sure there is still one CPU doing the interrupt work?
> And none of the multiple queue thread spreading tech exists?
> TNAPI attempts to solve the following problems:
> * Distribute the traffic across cores (i.e. the more core the more
> scalable is your networking application) for improving scalability.
> * Poll packets simultaneously from each RX queue (contraty to
> sequential NAPI polling) for fetching packets as fast as possible
> hence improve performance.
> * Through PF_RING, expose the RX queues to the userland so that
> the application can spawn one thread per queue hence avoid using
> semaphores at all.
> TNAPI achieves all this by starting one thread per RX queue. Received
> packets are then pushed to PF_RING (if available) or through the
> standard Linux stack. However in order to fully exploit this
> technology it is necessary to use PF_RING as it provides a straight
> packet path from kernel to userland. Furthermore it allows to create a
> virtual ethernet card per RX queue.
> freebsd-net at freebsd.org mailing list
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
More information about the freebsd-hackers