9.2 ixgbe tx queue hang

[Jack Vogel]

Your 4K mbuf pool is not being used, make sure you increase the size once
you are
using that or you'll just be having the same issue with a different pool.

Oh, and that patch was against the code in HEAD, it might need some manual
hacking
if you're using anything older.

Not sure what you mean about memory allocation in 10, this change is not 10
specific, its
something I intended on doing and it just slipped between the cracks.

Jack



On Thu, Mar 20, 2014 at 3:32 PM, Christopher Forgeron
<csforgeron at gmail.com>wrote:

> I agree, performance is noticeably worse with TSO off, but I thought it
> would be a good step in troubleshooting. I'm glad you're a regular reader
> of the list, so I don't have to settle for slow performance. :-)
>
> I'm applying your patch now, I think it will fix it - but I'll report in
> after it's run iometer for the night regardless.
>
> On another note: What's so different about memory allocation in 10 that is
> making this an issue?
>
>
>
>
> On Thu, Mar 20, 2014 at 7:24 PM, Jack Vogel <jfvogel at gmail.com> wrote:
>
>> I strongly discourage anyone from disabling TSO on 10G, its necessary to
>> get the
>> performance one wants to see on the hardware.
>>
>> Here is a patch to do what i'm talking about:
>>
>> *** ixgbe.c    Fri Jan 10 18:12:20 2014
>> --- ixgbe.jfv.c    Thu Mar 20 23:04:15 2014
>> *************** ixgbe_init_locked(struct adapter *adapte
>> *** 1140,1151 ****
>>           */
>>           if (adapter->max_frame_size <= 2048)
>>                   adapter->rx_mbuf_sz = MCLBYTES;
>> -         else if (adapter->max_frame_size <= 4096)
>> -                 adapter->rx_mbuf_sz = MJUMPAGESIZE;
>> -         else if (adapter->max_frame_size <= 9216)
>> -                 adapter->rx_mbuf_sz = MJUM9BYTES;
>>           else
>> !                 adapter->rx_mbuf_sz = MJUM16BYTES;
>>
>>           /* Prepare receive descriptors and buffers */
>>           if (ixgbe_setup_receive_structures(adapter)) {
>> --- 1140,1147 ----
>>           */
>>           if (adapter->max_frame_size <= 2048)
>>                   adapter->rx_mbuf_sz = MCLBYTES;
>>           else
>> !                 adapter->rx_mbuf_sz = MJUMPAGESIZE;
>>
>>           /* Prepare receive descriptors and buffers */
>>           if (ixgbe_setup_receive_structures(adapter)) {
>>
>>
>>
>>
>>
>>
>> On Thu, Mar 20, 2014 at 3:12 PM, Christopher Forgeron <
>> csforgeron at gmail.com> wrote:
>>
>>> Hi Jack,
>>>
>>>  I'm on ixgbe 2.5.15
>>>
>>>  I see a few other threads about using MJUMPAGESIZE instead of
>>> MJUM9BYTES.
>>>
>>>  If you have a patch you'd like me to test, I'll compile it in and let
>>> you know. I was just looking at Garrett's if_em.c patch and thinking about
>>> applying it to ixgbe..
>>>
>>>  As it stands I seem to not be having the problem now that I have
>>> disabled TSO on ix0, but I still need more test runs to confirm - Which is
>>> also in line (i think) with what you are all saying.
>>>
>>>
>>>
>>>
>>> On Thu, Mar 20, 2014 at 7:00 PM, Jack Vogel <jfvogel at gmail.com> wrote:
>>>
>>>> What he's saying is that the driver should not be using 9K mbuf
>>>> clusters, I thought
>>>> this had been changed but I see the code in HEAD is still using the
>>>> larger clusters
>>>> when you up the mtu.  I will put it on my list to change with the next
>>>> update to HEAD.
>>>>
>>>>
>>>> What version of ixgbe are you using?
>>>>
>>>> Jack
>>>>
>>>>
>>>>
>>>> On Thu, Mar 20, 2014 at 2:34 PM, Christopher Forgeron <
>>>> csforgeron at gmail.com> wrote:
>>>>
>>>>> I have found this:
>>>>>
>>>>> http://lists.freebsd.org/pipermail/freebsd-net/2013-October/036955.html
>>>>>
>>>>> I think what you're saying is that;
>>>>> - a MTU of 9000 doesn't need to equal a 9k mbuf / jumbo cluster
>>>>> - modern NIC drivers can gather 9000 bytes of data from various memory
>>>>> locations
>>>>> - The fact that I'm seeing 9k jumbo clusters is showing me that my
>>>>> driver
>>>>> is trying to allocate 9k of contiguous space, and it's failing.
>>>>>
>>>>> Please correct me if I'm off here, I'd love to understand more.
>>>>>
>>>>>
>>>>> On Thu, Mar 20, 2014 at 6:13 PM, Garrett Wollman <
>>>>> wollman at hergotha.csail.mit.edu> wrote:
>>>>>
>>>>> > In article
>>>>> > <CAB2_NwAOmPtZjB03pdDiTK2OvQgqk-tYf83Jq4Ukt9jnZA8CNA at mail.gmail.com
>>>>> >,
>>>>> > csforgeron at gmail.com writes:
>>>>> >
>>>>> > >50/27433/0 requests for jumbo clusters denied (4k/9k/16k)
>>>>> >
>>>>> > This is going to screw you.  You need to make sure that no NIC driver
>>>>> > ever allocates 9k jumbo pages -- unless you are using one of those
>>>>> > mythical drivers that can't do scatter/gather DMA on receive, which
>>>>> > you don't appear to be.
>>>>> >
>>>>> > These failures occur when the driver is trying to replenish its
>>>>> > receive queue, but is unable to allocate three *physically*
>>>>> contiguous
>>>>> > pages of RAM to construct the 9k jumbo cluster (of which the
>>>>> remaining
>>>>> > 3k is simply wasted).  This happens on any moderately active server,
>>>>> > once physical memory gets checkerboarded with active single pages,
>>>>> > particularly with ZFS where those pages are wired in kernel memory
>>>>> and
>>>>> > so can't be evicted.
>>>>> >
>>>>> > -GAWollman
>>>>> >
>>>>> _______________________________________________
>>>>> freebsd-net at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>>>
>>>>
>>>>
>>>
>>
>