igb and ALTQ in 9.1-rc3

[Barney Cordoba]

--- On Fri, 3/29/13, Pieper, Jeffrey E <jeffrey.e.pieper at intel.com> wrote:

> From: Pieper, Jeffrey E <jeffrey.e.pieper at intel.com>
> Subject: RE: igb and ALTQ in 9.1-rc3
> To: "Barney Cordoba" <barney_cordoba at yahoo.com>, "Jack Vogel" <jfvogel at gmail.com>, "Nick Rogers" <ncrogers at gmail.com>
> Cc: "freebsd-net at freebsd.org" <freebsd-net at freebsd.org>, "Clement Hermann (nodens)" <nodens2099 at gmail.com>
> Date: Friday, March 29, 2013, 11:45 AM
> 
> 
> -----Original Message-----
> From: owner-freebsd-net at freebsd.org
> [mailto:owner-freebsd-net at freebsd.org]
> On Behalf Of Barney Cordoba
> Sent: Friday, March 29, 2013 5:51 AM
> To: Jack Vogel; Nick Rogers
> Cc: freebsd-net at freebsd.org;
> Clement Hermann (nodens)
> Subject: Re: igb and ALTQ in 9.1-rc3
> 
> 
> 
> --- On Thu, 3/28/13, Nick Rogers <ncrogers at gmail.com>
> wrote:
> 
> > From: Nick Rogers <ncrogers at gmail.com>
> > Subject: Re: igb and ALTQ in 9.1-rc3
> > To: "Jack Vogel" <jfvogel at gmail.com>
> > Cc: "Barney Cordoba" <barney_cordoba at yahoo.com>,
> "Clement Hermann (nodens)" <nodens2099 at gmail.com>,
> "freebsd-net at freebsd.org"
> <freebsd-net at freebsd.org>
> > Date: Thursday, March 28, 2013, 9:29 PM
> > On Thu, Mar 28, 2013 at 4:16 PM, Jack
> > Vogel <jfvogel at gmail.com>
> > wrote:
> > > Have been kept fairly busy with other matters,
> one
> > thing I could do short
> > > term is
> > > change the defines in igb the way I did in the em
> > driver so you could still
> > > define
> > > the older if_start entry. Right now those are
> based on
> > OS version and so you
> > > will
> > > automatically get if_transmit, but I could change
> it to
> > be IGB_LEGACY_TX or
> > > so,
> > > and that could be defined in the Makefile.
> > >
> > > Would this help?
> > 
> > I'm currently using ALTQ successfully with the em
> driver, so
> > if igb
> > behaved the same with respect to using if_start instead
> of
> > if_transmit
> > when ALTQ is in play, that would be great. I do not
> > completely
> > understand the change you propose as I am not very
> familiar
> > with the
> > driver internals. Any kind of patch or extra
> > Makefile/make.conf
> > definition that would allow me to build a 9-STABLE
> kernel
> > with an igb
> > driver that works again with ALTQ, ASAP, would be much
> > appreciated.
> > 
> > >
> > > Jack
> > >
> > >
> > >
> > > On Thu, Mar 28, 2013 at 2:31 PM, Nick Rogers
> <ncrogers at gmail.com>
> > wrote:
> > >>
> > >> On Tue, Dec 11, 2012 at 1:09 AM, Jack Vogel
> <jfvogel at gmail.com>
> > wrote:
> > >> > On Mon, Dec 10, 2012 at 11:58 PM, Gleb
> > Smirnoff <glebius at freebsd.org>
> > >> > wrote:
> > >> >
> > >> >> On Mon, Dec 10, 2012 at 03:31:19PM
> -0800,
> > Jack Vogel wrote:
> > >> >> J> UH, maybe asking the owner of
> the
> > driver would help :)
> > >> >> J>
> > >> >> J> ... and no, I've never been
> aware of
> > doing anything to stop
> > >> >> supporting
> > >> >> altq
> > >> >> J> so you wouldn't see any
> commits. If
> > there's something in the altq
> > >> >> code
> > >> >> or
> > >> >> J> support (which I have nothing
> to do
> > with) that caused this no-one
> > >> >> informed
> > >> >> J> me.
> > >> >>
> > >> >> Switching from if_start to
> if_transmit
> > effectively disables ALTQ
> > >> >> support.
> > >> >>
> > >> >> AFAIR, there is some magic
> implemented in
> > other drivers that makes them
> > >> >> modern (that means using
> if_transmit), but
> > still capable to switch to
> > >> >> queueing
> > >> >> mode if SIOCADDALTQ was casted upon
> them.
> > >> >>
> > >> >>
> > >> > Oh, hmmm, I'll look into the matter after
> my
> > vacation.
> > >> >
> > >> > Jack
> > >>
> > >> Has there been any progress on resolving this
> > issue? I recently ran
> > >> into this problem upgrading my servers from
> 8.3 to
> > 9.1-RELEASE and am
> > >> wondering what the latest recommendation is.
> I've
> > used ALTQ and igb
> > >> successfully for years and it is unfortunate
> it no
> > longer works.
> > >> Appreciate any advice.
> > >>
> >
> >Do yourself a favor and either get a cheap dual port
> 82571 card or
> >2 cards and disable the IGB ports. The igb driver is
> defective, and until
> >they back out the new, untested multi-queue stuff you're
> just neutering 
> >your system trying to use it.
> >
> >Frankly this project made a huge mistake by moving
> forward with multi
> >queue just for the sake of saying that you support it;
> without having
> >any credible plan for implementing it. That nonsense
> that Bill Macy did
> >should have been tarballed up and deposited in the trash
> folder. The
> >biggest mess in programming history.
> >
> >That being said, the solution is not to hack the igb
> driver; its to make
> >ALTQ if_transmit compatible, which shouldn't be all that
> difficult. 
> >
> >BC
> 
> I may be misunderstanding what you are saying, but if the
> solution is, as you say "not to hack the igb driver", then
> how is it defective in this case? Or are you just directing
> vitriol toward Intel? Multi-queue is working fine in igb. 
> 
> Jeff

It works like crap. Your definition of "works" is that it doesnt crash.
Mine is that it works better with multiple queues than with 1, which
it doesn't. And if you load a system up, it will blow up with  multiqueue
before it will with 1 queue. The point of using Multiqueue isn't to 
exhaust all of the cpus instead of just 2. It's to get past the wall of
using only 2 cpus when they're exhausted. The goal is to increase the 
capacity of the system; not to make it look like you're using more cpus
without any analysis of overall system efficiency. 

That's my definition of defective. The default tuning is wrong also, 
because is ASSumes that you only have 1 card in the box. So if you 
have 2 cards, it has 2 rx queues contending for the same cpus, which is
exactly the opposite of what you want.

There's no vitriol towards Intel. That's what Jack said; that Intel 
drivers are just samples and they're not optimized. Obviously some
drivers are better than others because he's spent more time on them. 
What I don't like is that nobody ever tells anyone which drivers are good
and which suck. If you think that a driver is a driver then you should
be doing something else with your life other than networking.

The truth is that there is no plan for multiqueue; the idea that spreading
things across cpus randomly is better is just plain wrong; in fact
there's a strong case to be made to keep things on the same cpu (and in 
the same cache). Igb is a college project. Using it will make your
system slower than using good single queue cards.

BC