ixgbe and fast interrupts
John Baldwin
jhb at freebsd.org
Fri Nov 18 17:20:06 UTC 2011
On Friday, November 18, 2011 12:06:15 pm Luigi Rizzo wrote:
> On Fri, Nov 18, 2011 at 08:00:06AM -0500, John Baldwin wrote:
> > On Friday, November 18, 2011 3:46:02 am Matteo Landi wrote:
> > > > you probably want to be using MSI-X for a 10G NIC instead of INTx anyway.
> > >
> > > Why do you say that? Is MSI-X faster than INTx in terms of interrupt
> > > latency? When should I use MSI-X, instead of fast filters interrupts
> > > (fast interrupt?), instead of ithread interrupts? Thanks in advace.
> >
> > With MSI-X you can have more than one interrupt and those interrupts can be
> > distributed across CPUs. This means you can (somewhat) tie each queue on your
> > NIC to a different CPU.
> >
> > MSI-X vs INTx is orthogonal to fast vs filter, but in general MSI and MSI-X
> > interrupts are not shared, and require no interrupt masking in hardware (they
> > are effectively edge-triggered), so using a filter for MSI is rather pointless
> > and only adds needless complexity. For MSI I would just use a theraded
> > interrupt handler. For INTx, I would only use a fast interrupt handler if
> > there is a really good reason to do so (e.g. em(4) does so to work around
> > broken Intel Host-PCI bridges).
>
> A bit more context: Matteo is looking at the latency of RPCs across
> the network involving userspace processes, and possibly using the
> netmap API. As we understand it:
>
> if you are not using a filter, the interrupt calls a "predefined"
> filter (kern_intr.c::intr_event_schedule_thread() ? ) which wakes
> up the handler thread which in turn wakes up the user process. This
> means two scheduler invocations on each side.
Yes, but if you use a filter you still have to do that as your filter would
just be queueing a task to on a taskqueue which would then do the actual
selwakeup() from a taskqueue thread. Filters are typically used to avoid
masking the interrupt in the PIC, or to limit the handlers executed on a
shared interrupt.
> In the case of netmap, all the handler needs to do is a selwakeup()
> of the user thread blocked on the file descriptor, so if this
> can be done in the filter we can save an extra step through the
> scheduler.
You can't call selwakeup() from a filter, it is not safe since it uses
mutexes, etc. There are only a few things you can do from a filter.
You could do a plain wakeup() if you let userland use a custom ioctl to
block on the filter, but not selwakeup().
--
John Baldwin
More information about the freebsd-current
mailing list