Sudden mbuf demand increase and shortage under the load (igb issue?)

[Pyun YongHyeon]

On Thu, Feb 18, 2010 at 05:05:16PM -0800, Maxim Sobolev wrote:
> Folks,
> 
> Indeed, it looks like igb(4) issue. Replacing the card with the 
> desktop-grade em(4)-supported card has fixed the problem for us. The 
> system has been happily pushing 110mbps worth of RTP traffic and 2000 
> concurrent calls without any problems for two days now.
> 
> em0 at pci0:7:0:0: class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00 
> hdr=0x00
>     vendor     = 'Intel Corporation'
>     class      = network
>     subclass   = ethernet
> 
> em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xec00-0xec1f mem 
> 0xfbee0000-0xfbefffff,0xfbe00000-0xfbe7ffff,0xfbedc000-0xfbedffff irq 24 
> at device 0.0 on pci7
> em0: Using MSIX interrupts
> em0: [ITHREAD]
> em0: [ITHREAD]
> em0: [ITHREAD]
> em0: Ethernet address: 00:1b:21:50:02:49
> 
> I really think that this has to be addressed before 7.3 release is out. 
> FreeBSD used to be famous for its excellent network performance and it's 
> shame to see that deteriorating due to sub-standard quality drivers. 
> Especially when there is a multi-billion vendor supporting the driver in 
> question. No finger pointing, but it really looks like either somebody 
> is not doing his job or the said vendor doesn't care so much about 
> supporting FreeBSD. I am pretty sure the vendor in question has access 
> to numerous load-testing tools, that should have catched this issue.
> 
> This is the second time during the past 6 months I have issue with the 
> quality of the Intel-based drivers - the first one is filed as 
> kern/140326, which has stalled apparently despite me providing all 
> necessary debug information.
> 

I can reproduce this bug on my box and I guess the root cause comes
from PBA(Packet Buffer Allocation) configuration. Some controllers
seems to require more TX buffer size to use 9000 MTU. The datasheet
is not clear which controller has how much amount of Packet Buffer
storage. This parameter seems to affect performance a lot because
increasing TX buffer size results in decreasing RX buffer size. The
attached patch seems to fix the issue for me but Jack may know
better the hardware details as publicly available datasheet seems
to be useless here. 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: em.ich10.jumbo.diff
Type: text/x-diff
Size: 425 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20100219/89601028/em.ich10.jumbo.bin