Chelsio NETMAP performance

[Navdeep Parhar]

On Wed, Feb 05, 2020 at 02:38:32PM +0300, Slawa Olhovchenkov wrote:
> On Tue, Feb 04, 2020 at 12:37:08PM -0800, Navdeep Parhar wrote:
> 
> > >> nm_holdoff_tmr_idx is a 0-based index into the list above.  So if the
> > >> tmr idx is 0 you are using the 0th (first) value from the list of
> > >> timers.  Try increasing nm_holdoff_tmr_idx and see if that brings down
> > >> the interrupt rate under control.
> > >>
> > >> # sysctl hw.cxgbe.nm_holdoff_tmr_idx=3/4/5
> > > 
> > > OK, interrupt rate go down, but interrupt time about same.
> > > (interrupt rate for intel card about 0, compared to 25% chelsio).
> > 
> > I think iflib runs a lot of stuff in taskqueues rather than the driver
> > ithread so the CPU accounting may vary.  Use dtrace to see if
> 
> Don't think this is impact: worker's CPU core w/o any syscalls and
> only w/ bunding workker thread and NIC irq handler show about 100%
> user CPU time.
> 
> May be some cache-miss work performed later, at poll(2) time in case
> of intel driver compared to chelsio (do at interrupt time)?

Could be.  While we are here, is it possible for you to try the patches
in these two?

https://reviews.freebsd.org/D17868
https://reviews.freebsd.org/D17869

> 
> > netmap_rx_irq is being called by an ithread or a taskqueue to figure out
> > what driver does what.
> 
> Can you explain some more?
> I am not sure about dtrace probe to use and later evaluation

# dtrace -n 'fbt::netmap_rx_irq:entry {stack()}'

Take a look at the stack and see if it's an ithread or one of iflib's
taskqueues that called netmap_rx_irq.

Regards,
Navdeep