netisr observations

[hiren panchasara]

On Fri, Apr 11, 2014 at 11:30 AM, Patrick Kelsey <kelsey at ieee.org> wrote:
>
> The output of netstat -Q shows IP dispatch is set to default, which is
> direct (NETISR_DISPATCH_DIRECT).  That means each IP packet will be
> processed on the same CPU that the Ethernet processing for that packet was
> performed on, so CPU selection for IP packets will not be based on flowid.
> The output of netstat -Q shows Ethernet dispatch is set to direct
> (NETISR_DISPATCH_DIRECT if you wind up reading the code), so the Ethernet
> processing for each packet will take place on the same CPU that the driver
> receives that packet on.
>
> For the igb driver with queues autoconfigured and msix enabled, as the
> sysctl output shows you have, the driver will create a number of queues
> subject to device limitations, msix message limitations, and the number of
> CPUs in the system, establish a separate interrupt handler for each one, and
> bind each of those interrupt handlers to a separate CPU.  It also creates a
> separate single-threaded taskqueue for each queue.  Each queue interrupt
> handler sends work to its associated taskqueue when the interrupt fires.
> Those taskqueues are where the Ethernet packets are received and processed
> by the driver.  The question is where those taskqueue threads will be run.
> I don't see anything in the driver that makes an attempt to bind those
> taskqueue threads to specific CPUs, so really the location of all of the
> packet processing is up to the scheduler (i.e., arbitrary).
>
> The summary is:
>
> 1. the hardware schedules each received packet to one of its queues and
> raises the interrupt for that queue
> 2. that queue interrupt is serviced on the same CPU all the time, which is
> different from the CPUs for all other queues on that interface
> 3. the interrupt handler notifies the corresponding task queue, which runs
> its task in a thread on whatever CPU the scheduler chooses
> 4. that task dispatches the packet for Ethernet processing via netisr, which
> processes it on whatever the current CPU is
> 5. Ethernet processing dispatches that packet for IP processing via netisr,
> which processes it on whatever the current CPU is

I really appreciate you taking time and explaining this. Thank you.

I am specially confused with ip "Queued" column from netstat -Q
showing 203888563 only for cpu3. Does this mean that cpu3 queues
everything and then distributes among other cpus? Where does this
queuing on cpu3 happens out of 5 stages you mentioned above?

This value gets populated in snwp->snw_queued field for each cpu
inside sysctl_netisr_work().

>
> You might want to try changing the default netisr dispatch policy to
> 'deferred' (sysctl net.isr.dispatch=deferred).  If you do that, the Ethernet
> processing will still happen on an arbitrary CPU chosen by the scheduler,
> but the IP processing should then get mapped to a CPU based on the flowid
> assigned by the driver.  Since igb assigns flowids based on received queue
> number, all IP (and above) processing for that packet should then be
> performed on the same CPU the queue interrupt was bound to.

I will give this a try and see how things behave.

I was also thinking about net.isr.bindthreads. netisr_start_swi() does
intr_event_bind() if we have it bindthreads set to 1. What would that
gain me, if anything?

Would it stop moving intr{swi1: netisr 3} on to different cpus (as I
am seeing in 'top' o/p) and bind it to a single cpu?

I've came across a thread discussing some side-effects of this though:
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-January/037597.html

Thanks a ton, again.

cheers,
Hiren