kern/135222: [igb] low speed routing between two igb interfaces
Michael
freebsdusb at bindone.de
Fri Jun 19 15:39:15 UTC 2009
Barney Cordoba wrote:
>
>
> --- On Wed, 6/17/09, Michael <freebsdusb at bindone.de> wrote:
>
>> From: Michael <freebsdusb at bindone.de>
>> Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces
>> To: freebsd-net at FreeBSD.org
>> Date: Wednesday, June 17, 2009, 9:40 PM
>> The following reply was made to PR
>> kern/135222; it has been noted by GNATS.
>>
>> From: Michael <freebsdusb at bindone.de>
>> To: Barney Cordoba <barney_cordoba at yahoo.com>
>> Cc: freebsd-gnats-submit at FreeBSD.org
>> Subject: Re: kern/135222: [igb] low speed routing between
>> two igb interfaces
>> Date: Thu, 18 Jun 2009 03:32:15 +0200
>>
>> Barney Cordoba wrote:
>> >
>> >
>> > --- On Wed, 6/17/09, Michael <freebsdusb at bindone.de>
>> wrote:
>> >
>> >> From: Michael <freebsdusb at bindone.de>
>> >> Subject: Re: kern/135222: [igb] low speed routing
>> between two igb interfaces
>> >> To: "Barney Cordoba" <barney_cordoba at yahoo.com>
>> >> Cc: freebsd-net at FreeBSD.org
>> >> Date: Wednesday, June 17, 2009, 5:28 PM
>> >> Barney Cordoba wrote:
>> >>>
>> >>> --- On Fri, 6/12/09, Michael <freebsdusb at bindone.de>
>> >> wrote:
>> >>>> From: Michael <freebsdusb at bindone.de>
>> >>>> Subject: Re: kern/135222: [igb] low speed
>> routing
>> >> between two igb interfaces
>> >>>> To: freebsd-net at FreeBSD.org
>> >>>> Date: Friday, June 12, 2009, 5:50 AM
>> >>>> The following reply was made to PR
>> >>>> kern/135222; it has been noted by GNATS.
>> >>>>
>> >>>> From: Michael <freebsdusb at bindone.de>
>> >>>> To: Cc: freebsd-gnats-submit at FreeBSD.org
>> >>>> Subject: Re: kern/135222: [igb] low speed
>> routing
>> >> between
>> >>>> two igb interfaces
>> >>>> Date: Fri, 12 Jun 2009 11:45:47 +0200
>> >>>>
>> >>>> The original poster
>> reported that the
>> >> suggested fix works
>> >>>> for him:
>> >>>> ---
>> >>>> Hello Michael,
>> >>>>
>> >>>> Thank you. It's
>> working.
>> >>>>
>> >>>> I consider it necessary
>> to put this into the
>> >> release
>> >>>> errata.
>> >>>>
>> >>>>
>> >>>> Mishustin Andrew wrote:
>> >>>> >> Number:
>>
>> >>>> 135222
>> >>>> >>
>> Category:
>> >> kern
>> >>>> >>
>> Synopsis:
>> >> [igb]
>> >>>> low speed routing between two igb
>> interfaces
>> >>>> >>
>> Confidential: no
>> >>>> >>
>> Severity:
>> >> serious
>> >>>> >>
>> Priority:
>> >> medium
>> >>>> >>
>> Responsible:
>> >> freebsd-bugs
>> >>>> >> State:
>>
>> >> open
>> >>>> >> Quarter:
>>
>> >>>> >>
>> Keywords:
>> >>
>> >>>> >> Date-Required:
>> >>>> >> Class:
>>
>> >> sw-bug
>> >>>> >>
>> >> Submitter-Id: current-users
>> >>>> >>
>> Arrival-Date: Wed
>> >> Jun 03
>> >>>> 18:30:01 UTC 2009
>> >>>> >> Closed-Date:
>> >>>> >> Last-Modified:
>> >>>> >> Originator:
>> >> Mishustin
>> >>>> Andrew
>> >>>> >> Release:
>>
>> >> FreeBSD
>> >>>> 7.1-RELEASE amd64, FreeBSD 7.2-RELEASE
>> amd64
>> >>>> >> Organization:
>> >>>> > HNT
>> >>>> >> Environment:
>> >>>> > FreeBSD test.hnt
>> 7.2-RELEASE FreeBSD
>> >> 7.2-RELEASE #12:
>> >>>> Thu Apr 30 18:28:15 MSD 20
>> >>>> > 09
>> admin at test.hnt:/usr/src/sys/amd64/compile/GENERIC
>> >>>> amd64
>> >>>> >> Description:
>> >>>> > I made a FreeBSD
>> multiprocesor server
>> >> to act as
>> >>>> simple gateway.
>> >>>> > It use onboard
>> Intel 82575EB Dual-Port
>> >> Gigabit
>> >>>> Ethernet Controller.
>> >>>> > I observe traffic
>> speed near 400
>> >> Kbit/s.
>> >>>> > I test both
>> interfaces separately -
>> >>>> > ftp client work at
>> speed near 1 Gbit/s
>> >> in both
>> >>>> directions.
>> >>>> > Then I change NIC
>> to old Intel "em" NIC
>> >> - gateway
>> >>>> work at speed near 1 Gbit/s.
>> >>>> >
>> >>>> > Looks like a bug in
>> igb driver have an
>> >> effect upon
>> >>>> forwarded traffic.
>> >>>> >
>> >>>> > If you try
>> >>>> >
>> hw.igb.enable_aim=0
>> >>>> > The speed is near 1
>> Mbit/s
>> >>>> >
>> >>>> > hw.igb.rxd,
>> hw.igb.txd, "ifconfig -tso"
>> >> has no
>> >>>> effect.
>> >>>> >
>> >>>> > Nothing in
>> messages.log
>> >>>> >
>> >>>> > netstat -m
>> >>>> > 516/1674/2190 mbufs
>> in use
>> >> (current/cache/total)
>> >>>> > 515/927/1442/66560
>> mbuf clusters in
>> >> use
>> >>>> (current/cache/total/max)
>> >>>> > 515/893
>> mbuf+clusters out of packet
>> >> secondary zone in
>> >>>> use (current/cache)
>> >>>> > 0/44/44/33280 4k
>> (page size) jumbo
>> >> clusters in use
>> >>>> (current/cache/total/max)
>> >>>> > 0/0/0/16640 9k
>> jumbo clusters in use
>> >>>> (current/cache/total/max)
>> >>>> > 0/0/0/8320 16k
>> jumbo clusters in use
>> >>>> (current/cache/total/max)
>> >>>> > 1159K/2448K/3607K
>> bytes allocated to
>> >> network
>> >>>> (current/cache/total)
>> >>>> > 0/0/0 requests for
>> mbufs denied
>> >>>> (mbufs/clusters/mbuf+clusters)
>> >>>> > 0/0/0 requests for
>> jumbo clusters
>> >> denied (4k/9k/16k)
>> >>>> > 0/0/0 sfbufs in use
>> (current/peak/max)
>> >>>> > 0 requests for
>> sfbufs denied
>> >>>> > 0 requests for
>> sfbufs delayed
>> >>>> > 0 requests for I/O
>> initiated by
>> >> sendfile
>> >>>> > 0 calls to protocol
>> drain routines
>> >>>> >
>> >>>> > I use only IPv4
>> traffic.
>> >>>> >
>> >>>> >> How-To-Repeat:
>> >>>> > On machine with two
>> igb interfaces
>> >>>> > use rc.conf like
>> this:
>> >>>> >
>> >>>> >
>> hostname="test.test"
>> >>>> >
>> gateway_enable="YES"
>> >>>> > ifconfig_igb0="inet
>> 10.10.10.1/24"
>> >>>> > ifconfig_igb1="inet
>> 10.10.11.1/24"
>> >>>> >
>> >>>> > And try create
>> heavy traffic between
>> >> two networks.
>> >>>> >> Fix:
>> >>>> >
>> >>>> >
>> >>>> >> Release-Note:
>> >>>> >> Audit-Trail:
>> >>>> >> Unformatted:
>> >>>> >
>> >> _______________________________________________
>> >>>> > freebsd-bugs at freebsd.org
>> >>>
>> >>> This is not a bug. Unless you consider poorly
>> written
>> >> drivers to be bugs. You need to provide your
>> tuning
>> >> parameters for the card as well otherwise there's
>> nothing to
>> >> learn.
>> >>> The issue is that the driver doesn't address
>> the
>> >> purpose of the controller; which is to utilize
>> >> multiprocessor systems more effectively. The
>> effect is that
>> >> lock contention actually makes things worse than
>> if you just
>> >> use a single task as em does. Until the
>> multiqueue drivers
>> >> are re-written to manage locks properly you are
>> best advised
>> >> to save your money and stick with em.
>> >>> You should get similar performance using 1
>> queue as
>> >> with em. You could also force legacy
>> configuration by
>> >> forcing igb_setup_msix to return 0. Sadly, this
>> is the best
>> >> performance you will get from the stock driver.
>> >>> Barney
>> >>>
>> >>> Barney
>> >>>
>> >>>
>> >>>
>> >> I tried using 1 queue and it didn't make things
>> any better
>> >> (actually I'm
>> >> not sure if that worked at all). If it is
>> considered a bug
>> >> or not
>> >> doesn't really matter, what actually matters for
>> users (who
>> >> cannot
>> >> always chose which network controller will be
>> on-board) is
>> >> that they get
>> >> a least decent performance when doing IP
>> forwarding (and
>> >> not the
>> >> 5-50kb/s I've seen). You can get this out of the
>> >> controller, when
>> >> disabling lro through the sysctl. That's why I've
>> been
>> >> asking to put
>> >> this into the release errata section and/or at
>> least the
>> >> igb man page,
>> >> because the sysctl isn't documented anywhere.
>> Also the
>> >> fact, that tuning
>> >> the sysctl only affects the behaviour when it's
>> set on boot
>> >> might be
>> >> considered problematic.
>> >>
>> >> So at the very least, I think the following
>> should be
>> >> done:
>> >> 1. Document the sysctl in man igb(4)
>> >> 2. Put a known issues paragraph to man igb(4)
>> which
>> >> explains the issue
>> >> and what to put in sysctl.conf to stop this from
>> happening
>> >> 3. Add an entry to the release errata page about
>> this issue
>> >> (like I
>> >> suggested in one of my earlier emails) and
>> stating
>> >> something like "see
>> >> man igb(4) for details)
>> >>
>> >> This is not about using the controller to its
>> full
>> >> potential, but to
>> >> safe Joe Admin from spending days on figuring out
>> why the
>> >> machine is
>> >> forwarding packages slower than his BSD 2.x
>> machine did in
>> >> the 90s.
>> >>
>> >> cheers
>> >> Michael
>> >
>> > None of the offload crap should be enabled by
>> default.
>> >
>> > The real point is that "Joe Admin" shouldn't be using
>> controllers that have bad drivers at all. If you have to use
>> whatever hardware you have laying around, and don't have
>> enough flexibility to lay out $100 for a 2 port controller
>> that works to use with your $2000 server, than you need to
>> get your priorities in order. People go out and buy
>> redundant power supplies, high GHZ quad core processors and
>> gobs of memory and then they use whatever crappy onboard
>> controller they get no matter how poorly its suppo rted. Its
>> mindless.
>> >
>> > Barney
>> >
>> >
>> >
>>
>> How should anybody know that the controller is poorly
>> supported if there
>> is nothing in the documentation, release notes, man pages
>> or anywhere
>> else about this?
>>
>> The fact of the matter is that "the offload crap" _is_
>> enabled by
>> default. The release is out, it claims to support the
>> controller. There
>> _is_ a workaround and I'm asked if somebody could document
>> this so users
>> will have a chance. I'm also not convinced that it is a
>> crappy
>> controller per se, but just poorly supported. We used
>> those a lot before
>> without any issues, unfortunately now we had touse IP
>> forwarding in a
>> machine that has that controller (it has 6 interfaces in
>> total, four em
>> ports and two igb ports, all of them are in use and I
>> don't feel like
>> hooking up the sodering iron).
>>
>> So bottomline:
>> I said, there is a problem with the driver, there is a
>> workaround and it
>> should be documented.
>>
>> You say, the driver is bad and nobody should use it and if
>> they do it's
>> their own damn fault. We won't do anything about it and
>> refuse to tell
>> anybody, because we are the only ones who should know. We
>> don't care if
>> people can actually use our software and still claim the
>> hardware is
>> actually supported.
>>
>> Your attitude is really contra productive (actually
>> googling around I
>> see you made similar statements in the past about
>> stupid people not
>> willing to spend xxx$ on whatever piece of hardware, so
>> maybe you're
>> just trolling).
>>
>> Michael
>
> Tuning the card to be brain-dead isn't really a workaround. I'm sorry that you're not able to understand, but you can't educate the woodchucks, so carry on and feel free to do whatever you wish.
>
> BC
>
>
>
Without tuning the card: 5kb/s, with tuning the card: 50mb/s
That's the definition of a workaround, the fix would be making lro work
correctly - in general I prefer a brain-dead card to a brain-dead
mailing list subscriber. Welcome to the real world :)
Anyway, I'll stop feeding you now, this is getting boring and leads nowhere.
I still think that this should be noted somewhere in the docs, whoever
has permissions to commit might proceed in doing so...
More information about the freebsd-net
mailing list