Advice on a multithreaded netisr patch?

[Barney Cordoba]

--- On Tue, 4/7/09, Sepherosa Ziehau <sepherosa at gmail.com> wrote:

> From: Sepherosa Ziehau <sepherosa at gmail.com>
> Subject: Re: Advice on a multithreaded netisr patch?
> To: "Robert Watson" <rwatson at freebsd.org>
> Cc: freebsd-net at freebsd.org, "Ivan Voras" <ivoras at freebsd.org>
> Date: Tuesday, April 7, 2009, 9:57 AM
> On Tue, Apr 7, 2009 at 8:54 PM, Robert Watson
> <rwatson at freebsd.org> wrote:
> >
> > On Tue, 7 Apr 2009, Sepherosa Ziehau wrote:
> >
> >> On Sun, Apr 5, 2009 at 9:34 PM, Ivan Voras
> <ivoras at freebsd.org> wrote:
> >>>
> >>> Robert Watson wrote:
> >>>>
> >>>> On Sun, 5 Apr 2009, Ivan Voras wrote:
> >>>>
> >>>>> I thought this has something to deal
> with NIC moderation (em) but
> >>>>> can't really explain it. The bad
> performance part (not the jump) is
> >>>>> also visible over the loopback
> interface.
> >>>>
> >>>> FYI, if you want high performance, you
> really want a card supporting
> >>>> multiple input queues -- igb, cxgb, mxge,
> etc.  if_em-only cards are
> >>
> >> PCI-E em(4) supports 2 RX queues.  82571/82572
> support 2 TX queues. I have
> >> not tested multi-TX queues, but em(4) multi-RX
> queues work well in dfly
> >> (tested with 82573 and 82571)
> >
> > You may not have seen, but in FreeBSD 7.x and higher,
> we have a new if_igb
> > driver to support more recent Intel gigabit devices,
> which now probes a few
> > of the devices historically associated with if_em. 
> For example, on one of
> > the boxes I use:
> 
> If I understand the code correctly, it only takes 82575 and
> 82576; I
> don't have the hardware, else I would have already
> added dfly support
> (with multi rx queues at least, it seems 82576 supports 16
> RX queues
> :)

Regarding if_igb:

1) Multiple TX queues are not supported. There's some hokey code to
test, but it doesn't properly separate flows to the queues.
2) 2 Rx queues don't work, so only 1 and 4 work
3) With 4 queues, it just sucks up CPU under heavy load on 4 cpus. It will
blow 4 cpus at a lower load than em will with 1
4) You'll need to fix DMA setup, as it sets the alignment requirement
to PAGE_SIZE. I haven't been able to convince Jack that its wrong, not
that I've tried very hard since its easy to just fix myself.

Barney