individual queue blocking entire rx unit on ixgbe (Re: How do I balance bandwidth over several virtual NICs?)

[Luigi Rizzo]

On Wed, Oct 1, 2014 at 11:12 PM, Kim Shrier <kim at westryn.net> wrote:

> Not sure if this is related.  I was testing on 10.1 beta3 and I was copying
> approximately 250 GB of data to the test machine, I noticed that the
> network would periodically slow down to about 4 mbits/sec or it would
> pause for about 20 seconds and then continue at full speed.
>
> The ethernet interface on the test machine is:
>
> bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev.
> 0x57766001> mem 0xa0400000-0xa040ffff,0xa0410000-0xa041ffff irq 16 at
> device 0.0 on pci1
> bge0: CHIP ID 0x57766001; ASIC REV 0x57766; CHIP REV 0x577660; PCI-E
>

​no that seems totally unrelated to this thread.
We are talking specifically about the ixgbe / 82599

cheers
luigi
​


>
>
> The machine I am transferring from is using the em0 interface:
>
> em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0xdc00-0xdc1f mem
> 0xfb5e0000-0xfb5fffff,0xfb5dc000-0xfb5dffff irq 16 at device 0.0 on pci2
>
>
> I haven’t noticed any slow down when transferring between machines where
> both sides are using the em driver.
>
> During the slow downs and pauses, when I had a top command running,
> the process state would show up as “dp->dp” or “rl->l_” instead of
> something
> normal like “select” or CPUn.
>
> If this is related, then maybe the problem is somewhere other than the
> device
> driver.
>
> Kim
>
> On Oct 1, 2014, at 3:48 AM, Luigi Rizzo <rizzo at iet.unipi.it> wrote:
>
> > reviving this thread:
> >
> > i am just running experiments on 10.1 beta3 and even
> > setting dev.ix.*.fc=0 and flipping the interface up and down
> > does not seem to help: if i read only from a subset of the
> > queues, the entire rx unit stalls eventually.
> >
> > I need to drain all queues to keep moving.
> >
> > Just tested this with 8 instances of netmap-ipfw running
> > on an 8-core machine (8 queues enabled).
> >
> >   netmap-ipfw netmap:ix0-0 netmap:ix1-0
> >   netmap-ipfw netmap:ix0-1 netmap:ix1-1
> >   ...
> >
> > and the source on another box is blasting on multiple queues with
> >
> >   pkt-gen -f tx -i ix0 -d 10.0.10.0-10.0.10.255
> >
> >
> > I going to look at the driver's code now to see if/how
> > this issue can be addressed.
> >
> > cheers
> > luigi
> >
> >
> > On Tue, Sep 23, 2014 at 6:00 PM, Adrian Chadd <adrian at freebsd.org>
> wrote:
> >
> >> Ah, this behaviour.
> >>
> >> It's called DROP_EN on the intel igb / ixgbe hardware. Grep the
> >> drivers for that particular register bit/setting.
> >>
> >> Set that bit for an RX queue and it'll instruct the MAC to drop frames
> >> destined if that RX ring is full to it and keep receiving on the other
> >> rings. Otherwise yes, receiving on that ring with the ring full cuases
> >> the MAC to stop receiving on all rings until that ring has free space.
> >>
> >> You flip this on with ixgbe and igb by disabling tx/rx flowcontrol
> >> (sysctl dev.ix|igb.X.fc=0) before configuring the interface.
> >>
> >>
> >>
> >> -a
> >>
> >
> >
> >
> > --
> > -----------------------------------------+-------------------------------
> > Prof. Luigi RIZZO, rizzo at iet.unipi.it  . Dip. di Ing. dell'Informazione
> > http://www.iet.unipi.it/~luigi/        . Universita` di Pisa
> > TEL      +39-050-2211611               . via Diotisalvi 2
> > Mobile   +39-338-6809875               . 56122 PISA (Italy)
> > -----------------------------------------+-------------------------------
> > _______________________________________________
> > freebsd-net at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
>


-- 
-----------------------------------------+-------------------------------
 Prof. Luigi RIZZO, rizzo at iet.unipi.it  . Dip. di Ing. dell'Informazione
 http://www.iet.unipi.it/~luigi/        . Universita` di Pisa
 TEL      +39-050-2211611               . via Diotisalvi 2
 Mobile   +39-338-6809875               . 56122 PISA (Italy)
-----------------------------------------+-------------------------------