Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX
nitroboost at gmail.com
Fri Oct 7 21:06:47 UTC 2011
On Fri, Oct 7, 2011 at 12:55 PM, Arnaud Lacombe <lacombar at gmail.com> wrote:
> On Fri, Oct 7, 2011 at 3:24 PM, Mike Tancsa <mike at sentex.net> wrote:
> When you disable MSI-X, you mean via hw.pci.enable_msix=0 across the
> > board, or you disable multi-queue for the NIC, so it uses just one
> > interrupt, rather than separate ones for xmit and recv ?
As noticed it was the prior, all were disabled via hw.em.enable_msix=0
em(4)'s multiqueue is misleading. By default, with MSI-X enabled,
> before AFAIK, April 2010 it used 2 (RX+TX) queue + 1, ie. 5 MSI-X
> vectors. After April 2010, it uses 1 * (RX+TX) queue + 1, ie. 3
> MSI-X vectors. There is no logic for the driver to use 1 vector with
> MSI-X enabled.
> As a side note, the only gain of EM_MULTIQUEUE, now, is to allow the
> driver to use the buf_ring(9) lockless queue API, compared to the
> locked ifq. Today, em(4) should waste about 16k of memory for when
> !EM_MULTIQUEUE. This is the memory, 4096 * sizeof(void *), allocated
> for the buf_ring(9) structure which is not used in the !EM_MULTIQUEUE
> > Also, what is the purpose of
> > hw.pci.do_power_nodriver=3 vs 0 (3 means put absolutely everything
> > in D3 state.)
0 is the default state and means no 'power management' is done. My stripped
down kernels don't have usb support so '3' just reduces the consumption by
not powering those devices. If you load up the modules power will be
supplied and the devices function.
> > net.link.ifqmaxlen 1024 vs 50 (does anything else need to be adjusted of
> > this value is increased?)
I haven't seen any evidence that there are other supporting adjustments for
this, this was actually something I stumbled across while researching the
> He might as well try to enable EM_MULTIQUEUE.
> > hw.em.rxd="2048"
> > hw.em.txd="2048"
> As it starts to be well known here, I am not a fan of bumping a limit
> to hide a bug. So I'd rather lower this to 512 or 256, and hope it
> triggers the issue more often, so that it could be diagnosticed and
> fixed for good.
> - Arnaud
> : actually it depends on a field in the chip NVM, which can be up
> to 4 (0 based accounting, this would translate in 5 vectors), but
> happened to be 2 (3 vector) in 82574 I've got access to. Last time I
> checked, this setting could not be seen with the standard NVM dump
> sysctl, which limit the output's size. On those chip, the
> pre-April-2010 code would falls back on MSI even if 3 were available.
> > Have you tried leaving these two at the default on 7.2.3 ?
> > if_em.h implies 1024 for each.
> > ---Mike
Bumping rx/tx descriptors to 2048 was actually for performance reasons and
not to try to get around the issue. I did some fairly in depth testing and
found under heavy load it performed the best with those settings.
As mentioned on the other thread I'll re enable MSI-X on a few servers here
and collect uptime and the kernel msgbuf in addition. I'll bump the
descriptors down to 512 to try and increase our chances and compile the
driver with EM_MULTIQUEUE also.
More information about the freebsd-net