Sudden mbuf demand increase and shortage under the load (igb issue?)

[Pyun YongHyeon]

On Thu, Feb 18, 2010 at 11:43:20PM -0800, Jack Vogel wrote:
> This thread is confusing, first he says its an igb problem, then you offer
> an em patch :)
> 
> I have an important rev of igb that I am about ready to release, anyone that
> wishes to
> test against a problem they have would be welcome to have early access, just
> let me
> know.
> 
> I am not sure about this ich10 change, there are client NICs that
> specifically do NOT
> support jumbo frames, I'll need to look into it tomorrow at work.

Hmm, then how could I see advertised MSS 8960 on this controller?
I guess em(4) already checks MAC type to limit jumbo frame support
on these controllers.

09:44:18.701271 IP (tos 0x0, ttl 64, id 32888, offset 0, flags [DF], proto TCP (6), length 60) 192.168.30.2.55754 > 192.168.30.1.22: S, cksum 0xbd82 (incorrect (-> 0x3f62), 529388419:529388419(0) win 65535 <mss 8960,nop,wscale 3,sackOK,timestamp 263541083 0>
09:44:18.701538 IP (tos 0x0, ttl 64, id 40268, offset 0, flags [DF], proto TCP (6), length 60) 192.168.30.1.22 > 192.168.30.2.55754: S, cksum 0xa6ce (correct), 3828990973:3828990973(0) ack 529388420 win 65535 <mss 8960,nop,wscale 3,sackOK,timestamp 2084729864 263541083>
09:44:18.701549 IP (tos 0x0, ttl 64, id 32889, offset 0, flags [DF], proto TCP (6), length 52) 192.168.30.2.55754 > 192.168.30.1.22: ., cksum 0xbd7a (incorrect (-> 0xd2e1), 1:1(0) ack 1 win 8192 <nop,nop,timestamp 263541084 2084729864> 

em0 at pci0:0:25:0:        class=0x020000 card=0x027f1028 chip=0x10de8086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Intel Gigabit network connection (83567LM-3 )'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xfdae0000, size 131072, enabled
    bar   [14] = type Memory, range 32, base 0xfdad9000, size 4096, enabled
    bar   [18] = type I/O Port, range 32, base 0xecc0, size 32, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 13[e0] = PCI Advanced Features: FLR TP

> 
> Jack
> 
> 
> On Thu, Feb 18, 2010 at 7:42 PM, Pyun YongHyeon <pyunyh at gmail.com> wrote:
> 
> > On Thu, Feb 18, 2010 at 05:05:16PM -0800, Maxim Sobolev wrote:
> > > Folks,
> > >
> > > Indeed, it looks like igb(4) issue. Replacing the card with the
> > > desktop-grade em(4)-supported card has fixed the problem for us. The
> > > system has been happily pushing 110mbps worth of RTP traffic and 2000
> > > concurrent calls without any problems for two days now.
> > >
> > > em0 at pci0:7:0:0: class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00
> > > hdr=0x00
> > >     vendor     = 'Intel Corporation'
> > >     class      = network
> > >     subclass   = ethernet
> > >
> > > em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xec00-0xec1f mem
> > > 0xfbee0000-0xfbefffff,0xfbe00000-0xfbe7ffff,0xfbedc000-0xfbedffff irq 24
> > > at device 0.0 on pci7
> > > em0: Using MSIX interrupts
> > > em0: [ITHREAD]
> > > em0: [ITHREAD]
> > > em0: [ITHREAD]
> > > em0: Ethernet address: 00:1b:21:50:02:49
> > >
> > > I really think that this has to be addressed before 7.3 release is out.
> > > FreeBSD used to be famous for its excellent network performance and it's
> > > shame to see that deteriorating due to sub-standard quality drivers.
> > > Especially when there is a multi-billion vendor supporting the driver in
> > > question. No finger pointing, but it really looks like either somebody
> > > is not doing his job or the said vendor doesn't care so much about
> > > supporting FreeBSD. I am pretty sure the vendor in question has access
> > > to numerous load-testing tools, that should have catched this issue.
> > >
> > > This is the second time during the past 6 months I have issue with the
> > > quality of the Intel-based drivers - the first one is filed as
> > > kern/140326, which has stalled apparently despite me providing all
> > > necessary debug information.
> > >
> >
> > I can reproduce this bug on my box and I guess the root cause comes
> > from PBA(Packet Buffer Allocation) configuration. Some controllers
> > seems to require more TX buffer size to use 9000 MTU. The datasheet
> > is not clear which controller has how much amount of Packet Buffer
> > storage. This parameter seems to affect performance a lot because
> > increasing TX buffer size results in decreasing RX buffer size. The
> > attached patch seems to fix the issue for me but Jack may know
> > better the hardware details as publicly available datasheet seems
> > to be useless here.
> >