Request for errate notice on igb(4)

Mon Jun 8 10:53:08 UTC 2009

Hello Doc-Team,

I would like to request an errata notice on igb(4). Without disabling
lro, ip forwarding results in horrible performance (at least with
onboard Intel 82575EB Dual-Port Gigabit Ethernet Controllers). As a
workaround lro can be disabled, but this needs to be done on reboot
(sysctl.conf), tuning the sysctls it after interfaces have been
activated has no effect. See also my email "Bug in igb driver" and the
mail "[igb] low speed routing between two igb interfaces" a few days
later (kern/135222).

This problem happens on 7.1-RELEASE, 7.2-RELEASE, 7-STABLE and 8-CURRENT.

I'm writing you directly, because nobody at the bugs mailing list thinks
 this problem needs to be adressed - I totally disagree, ecause these
adapters are very common and a small errata noticed would've saved us
days of frustration (and potentially will others as well).

cheers
Michael

--- Suggested errata notice (something similar):

An issue was found in the igb(4) driver that causes desastrous
performance when FreeBSD acts as a gateway (ip forwarding enabled) and
LRO (large receive offloading) is enabled (which is the default
setting). As a workaround, disable LRO by adding the following lines to
/etc/sysctl.conf:
dev.igb.0.enable_lro=0
dev.igb.1.enable_lro=0
and reboot.

------- My Mail on freebsd-bugs Jun 2nd 2009
Hello,

we encountered problems using igb driver on a Supermicro dual quad-core
system (X7DWU).
It features an Intel Gigabit dual port controller (82575).

The IP forwarding performance was extremely bad (5 kb/s up to 50 kb/s)
until we disabled
Large Receive Offloading in /etc/sysctl.conf:
dev.igb.0.enable_lro=0
dev.igb.1.enable_lro=0

(changing it using ifconfig -lro and down/up the interfaces didn't help).

With the systctl values set as shown above, we get the expected
performance ob 50-60 MB/s.
We tested this on 7.1-RELEASE, 7.2-RELEASE and 7-STABLE. Right now we're
installing
8-CURRENT on the machine to see if it makes a difference.

It think it would be worthwhile to list this issue in the 7.2 release
errata section (that would've
saved us a significant amount of time  :) .

Cheers
Michael

------ Mail thread on kern/135222
The following reply was made to PR kern/135222; it has been noted by GNATS.

From: Michael <freebsdusb at bindone.de>
To: Mishustin Andrew <1 at hnt.ru>
Cc: freebsd-gnats-submit at FreeBSD.org
Subject: Re: kern/135222: [igb] low speed routing between two igb interfaces
Date: Thu, 04 Jun 2009 09:41:22 +0200

 Hello Andrew,

 look at my post "Bug in igb driver" from two days ago.
 We had exactly the same issues using this adapter (and presumambly the
 same board, your bug description matches 100% our setup  :)

 The following should fix those issues:

 echo "dev.igb.0.enable_lro=0" >>/etc/sysctl.conf
 echo "dev.igb.1.enable_lro=0" >>/etc/sysctl.conf
 reboot

 (this disabled large receive offloading - tuning this while the machine
 is running has no effect).

 I suggested putting this into the release errata, but got no feedback on
 this request so far.

 cheers
 Michael

 Mishustin Andrew wrote:
 >> Number:         135222
 >> Category:       kern
 >> Synopsis:       [igb] low speed routing between two igb interfaces
 >> Confidential:   no
 >> Severity:       serious
 >> Priority:       medium
 >> Responsible:    freebsd-bugs
 >> State:          open
 >> Quarter:
 >> Keywords:
 >> Date-Required:
 >> Class:          sw-bug
 >> Submitter-Id:   current-users
 >> Arrival-Date:   Wed Jun 03 18:30:01 UTC 2009
 >> Closed-Date:
 >> Last-Modified:
 >> Originator:     Mishustin Andrew
 >> Release:        FreeBSD 7.1-RELEASE amd64, FreeBSD 7.2-RELEASE amd64
 >> Organization:
 > HNT
 >> Environment:
 > FreeBSD test.hnt 7.2-RELEASE FreeBSD 7.2-RELEASE #12: Thu Apr 30
18:28:15 MSD 20
 > 09     admin at test.hnt:/usr/src/sys/amd64/compile/GENERIC  amd64
 >> Description:
 > I made a FreeBSD multiprocesor server to act as simple gateway.
 > It use onboard Intel 82575EB Dual-Port Gigabit Ethernet Controller.
 > I observe traffic speed near 400 Kbit/s.
 > I test both interfaces separately -
 > ftp client work at speed near 1 Gbit/s in both directions.
 > Then I change NIC to old Intel "em" NIC - gateway work at speed near
1 Gbit/s.
 >
 > Looks like a bug in igb driver have an effect upon forwarded traffic.
 >
 > If you try
 > hw.igb.enable_aim=0
 > The speed is near 1 Mbit/s
 >
 > hw.igb.rxd, hw.igb.txd, "ifconfig -tso" has no effect.
 >
 > Nothing in messages.log
 >
 > netstat -m
 > 516/1674/2190 mbufs in use (current/cache/total)
 > 515/927/1442/66560 mbuf clusters in use (current/cache/total/max)
 > 515/893 mbuf+clusters out of packet secondary zone in use (current/cache)
 > 0/44/44/33280 4k (page size) jumbo clusters in use
(current/cache/total/max)
 > 0/0/0/16640 9k jumbo clusters in use (current/cache/total/max)
 > 0/0/0/8320 16k jumbo clusters in use (current/cache/total/max)
 > 1159K/2448K/3607K bytes allocated to network (current/cache/total)
 > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
 > 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
 > 0/0/0 sfbufs in use (current/peak/max)
 > 0 requests for sfbufs denied
 > 0 requests for sfbufs delayed
 > 0 requests for I/O initiated by sendfile
 > 0 calls to protocol drain routines
 >
 > I use only IPv4 traffic.
 >
 >> How-To-Repeat:
 > On machine with two igb interfaces
 > use rc.conf like this:
 >
 > hostname="test.test"
 > gateway_enable="YES"
 > ifconfig_igb0="inet 10.10.10.1/24"
 > ifconfig_igb1="inet 10.10.11.1/24"
 >
 > And try create heavy traffic between two networks.
 >> Fix:
 >
 >
 >> Release-Note:
 >> Audit-Trail:
 >> Unformatted:
 > _______________________________________________
 > freebsd-bugs at freebsd.org mailing list
 > http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 > To unsubscribe, send any mail to "freebsd-bugs-unsubscribe at freebsd.org"

_______________________________________________
freebsd-bugs at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscribe at freebsd.org"

--- Feedback received by M. Andrew (outside of the mailing list)
Hello Michael,

Thank you. It's working.

I consider it necessary to put this into the release errata.