9.2 ixgbe tx queue hang

[Jack Vogel]

I strongly discourage anyone from disabling TSO on 10G, its necessary to
get the
performance one wants to see on the hardware.

Here is a patch to do what i'm talking about:

*** ixgbe.c    Fri Jan 10 18:12:20 2014
--- ixgbe.jfv.c    Thu Mar 20 23:04:15 2014
*************** ixgbe_init_locked(struct adapter *adapte
*** 1140,1151 ****
          */
          if (adapter->max_frame_size <= 2048)
                  adapter->rx_mbuf_sz = MCLBYTES;
-         else if (adapter->max_frame_size <= 4096)
-                 adapter->rx_mbuf_sz = MJUMPAGESIZE;
-         else if (adapter->max_frame_size <= 9216)
-                 adapter->rx_mbuf_sz = MJUM9BYTES;
          else
!                 adapter->rx_mbuf_sz = MJUM16BYTES;

          /* Prepare receive descriptors and buffers */
          if (ixgbe_setup_receive_structures(adapter)) {
--- 1140,1147 ----
          */
          if (adapter->max_frame_size <= 2048)
                  adapter->rx_mbuf_sz = MCLBYTES;
          else
!                 adapter->rx_mbuf_sz = MJUMPAGESIZE;

          /* Prepare receive descriptors and buffers */
          if (ixgbe_setup_receive_structures(adapter)) {






On Thu, Mar 20, 2014 at 3:12 PM, Christopher Forgeron
<csforgeron at gmail.com>wrote:

> Hi Jack,
>
>  I'm on ixgbe 2.5.15
>
>  I see a few other threads about using MJUMPAGESIZE instead of MJUM9BYTES.
>
>  If you have a patch you'd like me to test, I'll compile it in and let you
> know. I was just looking at Garrett's if_em.c patch and thinking about
> applying it to ixgbe..
>
>  As it stands I seem to not be having the problem now that I have disabled
> TSO on ix0, but I still need more test runs to confirm - Which is also in
> line (i think) with what you are all saying.
>
>
>
>
> On Thu, Mar 20, 2014 at 7:00 PM, Jack Vogel <jfvogel at gmail.com> wrote:
>
>> What he's saying is that the driver should not be using 9K mbuf clusters,
>> I thought
>> this had been changed but I see the code in HEAD is still using the
>> larger clusters
>> when you up the mtu.  I will put it on my list to change with the next
>> update to HEAD.
>>
>>
>> What version of ixgbe are you using?
>>
>> Jack
>>
>>
>>
>> On Thu, Mar 20, 2014 at 2:34 PM, Christopher Forgeron <
>> csforgeron at gmail.com> wrote:
>>
>>> I have found this:
>>>
>>> http://lists.freebsd.org/pipermail/freebsd-net/2013-October/036955.html
>>>
>>> I think what you're saying is that;
>>> - a MTU of 9000 doesn't need to equal a 9k mbuf / jumbo cluster
>>> - modern NIC drivers can gather 9000 bytes of data from various memory
>>> locations
>>> - The fact that I'm seeing 9k jumbo clusters is showing me that my driver
>>> is trying to allocate 9k of contiguous space, and it's failing.
>>>
>>> Please correct me if I'm off here, I'd love to understand more.
>>>
>>>
>>> On Thu, Mar 20, 2014 at 6:13 PM, Garrett Wollman <
>>> wollman at hergotha.csail.mit.edu> wrote:
>>>
>>> > In article
>>> > <CAB2_NwAOmPtZjB03pdDiTK2OvQgqk-tYf83Jq4Ukt9jnZA8CNA at mail.gmail.com>,
>>> > csforgeron at gmail.com writes:
>>> >
>>> > >50/27433/0 requests for jumbo clusters denied (4k/9k/16k)
>>> >
>>> > This is going to screw you.  You need to make sure that no NIC driver
>>> > ever allocates 9k jumbo pages -- unless you are using one of those
>>> > mythical drivers that can't do scatter/gather DMA on receive, which
>>> > you don't appear to be.
>>> >
>>> > These failures occur when the driver is trying to replenish its
>>> > receive queue, but is unable to allocate three *physically* contiguous
>>> > pages of RAM to construct the 9k jumbo cluster (of which the remaining
>>> > 3k is simply wasted).  This happens on any moderately active server,
>>> > once physical memory gets checkerboarded with active single pages,
>>> > particularly with ZFS where those pages are wired in kernel memory and
>>> > so can't be evicted.
>>> >
>>> > -GAWollman
>>> >
>>> _______________________________________________
>>> freebsd-net at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>
>>
>>
>