[Bug 194311] New: [ixgbe] DROP_EN needs to be set early to avoid RX queue hanging

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194311

            Bug ID: 194311
           Summary: [ixgbe] DROP_EN needs to be set early to avoid RX
                    queue hanging
           Product: Base System
           Version: 11.0-CURRENT
          Hardware: Any
                OS: Any
            Status: Needs Triage
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: freebsd-bugs at FreeBSD.org
          Reporter: adrian at freebsd.org

>From Luigi Rizzo:

reviving this thread:

i am just running experiments on 10.1 beta3 and even
setting dev.ix.*.fc=0 and flipping the interface up and down
does not seem to help: if i read only from a subset of the
queues, the entire rx unit stalls eventually.

I need to drain all queues to keep moving.

Just tested this with 8 instances of netmap-ipfw running
on an 8-core machine (8 queues enabled).

   netmap-ipfw netmap:ix0-0 netmap:ix1-0
   netmap-ipfw netmap:ix0-1 netmap:ix1-1
   ...

and the source on another box is blasting on multiple queues with

   pkt-gen -f tx -i ix0 -d 10.0.10.0-10.0.10.255


I going to look at the driver's code now to see if/how
this issue can be addressed.


...

 don't have a way to test this on HEAD.

Do you know if there is any change that could be related ?

On 10.1 beta 3 I noticed is that when I open a single queue on an ixgbe
(and with incoming traffic), the rx unit stalls no matter what the
previous state of dev.ix.*.fc (i.e. the DROP_EN bits) or QDE are.

In this state, toggling DROP_EN has no effect, whereas something that seems
effective is setting the QDE bit(s) in the PFQDE register for all queues,
_after_ i open the device.

>From the above my take is the following:

- on NIC reset, the SRRCTL register starts at 0 including DROP_EN;
  same goes for PFQDE.QDE

- setting SRRCTL.DROP_EN must happen before the receive unit is started;

- conversely, toggling PFQDE.QDE has effect even when the receive
  unit has started

- sysctl dev.ix.*.fc sets/clear DROP_EN but at the wrong time
  (the right time seems to be the window between reset and start)

I am going to run more tests to figure out.

cheers
...


Index: sys/dev/ixgbe/ixgbe.c
===================================================================
--- sys/dev/ixgbe/ixgbe.c    (revision 272974)
+++ sys/dev/ixgbe/ixgbe.c    (working copy)
@@ -4377,6 +4377,20 @@
         srrctl &= ~IXGBE_SRRCTL_BSIZEPKT_MASK;
         srrctl |= bufsz;
         srrctl |= IXGBE_SRRCTL_DESCTYPE_ADV_ONEBUF;
+
+        /*
+         * Set DROP_EN iff we have no flow control and >1 queue.
+         * Note that srrctl was cleared shortly before during reset,
+         * so we do not need to clear the bit, but do it just in case
+         * this code is moved elsewhere.
+         */
+        if (adapter->num_queues > 1 &&
+            adapter->hw.fc.requested_mode == ixgbe_fc_none) {
+            srrctl |= IXGBE_SRRCTL_DROP_EN;
+        } else {
+            srrctl &= ~IXGBE_SRRCTL_DROP_EN;
+        }
+
         IXGBE_WRITE_REG(hw, IXGBE_SRRCTL(i), srrctl);

         /* Setup the HW Rx Head and Tail Descriptor Pointers */

-- 
You are receiving this mail because:
You are the assignee for the bug.