9.2 ixgbe tx queue hang

[Rick Macklem]

Christopher Forgeron wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> Update:
> 
> For giggles, I set IP_MAXPACKET = 32768.
> 
Well, I'm pretty sure you don't want to do that, except for an experiment.
You can just set if_hw_tsomax to whatever you want to try, at the place
my ixgbe.patch put it (just before the call to ether_ifattach()).

> Over a hour of runtime, and no issues. This is better than with the
> TSO patch and the 9.2 ixgbe, as that was just a drastic reduction in
> errors.
> 
So now the question becomes "how much does if_hw_tsomax need to be
reduced from 65535 to get this?". If reducing it by the additional
4bytes for a vlan header is sufficient, then I understand what is
going on. If it needs to be reduced by more than that, then there
is something going on that I still don't understand.

> Still have an 'angry' netstat -m on boot, and I'm still incrementing
> denied netbuf calls, so something else is wrong.
> 
> I'm going to modify Rick's prinft in ixgbe to also output when we're
> over 32768. I'm sure it's still happening, but with an extra 32k of
> space, we're not busting like we did before.
> 
> 
> I notice a few interesting ip->ip_len changes since 9.2 - Like here,
> at line 720
> 
> http://fxr.watson.org/fxr/diff/netinet/ip_output.c?v=FREEBSD10;im=kwqeqdhhvovqn;diffval=FREEBSD92;diffvar=v
> 
> Looks like older code didn't byteswap with ntohs - I see that often
> in tcp_output.c, and in tcp_options.c.
> 
> 
> I'm also curious about this:Line 524
> http://fxr.watson.org/fxr/diff/netinet/ip_options.c?v=FREEBSD10;diffval=FREEBSD92;diffvar=v
> 
> 
> New 10 code:
> 
> ip ->ip_len = htons ( ntohs ( ip ->ip_len) + optlen); Old 9.2 Code:
> ip ->ip_len += optlen;
> 
Well, TSO segments aren't generated when optlen > 0, so I doubt this
matters for our issue (and I would find it hard to believe that this
would have been broken?). You can always look at the svn commit logs
to see why/how something was changed.

> 
> 
> I wonder if there are any unexpected consequences of these changes,
> or perhaps a line someplace that doesn't make the change.
> 
> Is there a dtrace command I could use to watch these functions and
> compare the new ip_len with ip->ip_len or other variables?
> 
> 
> 
> 
> 
> 
> 
> On Sun, Mar 23, 2014 at 12:25 PM, Christopher Forgeron <
> csforgeron at gmail.com > wrote:
> 
> 
> 
> 
> 
> 
> 
> On Sat, Mar 22, 2014 at 11:58 PM, Rick Macklem < rmacklem at uoguelph.ca
> > wrote:
> 
> 
> 
> 
> Christopher Forgeron wrote:
> > 
> 
> > Also should we not also subtract ETHER_VLAN_ENCAP_LEN from tsomax
> > to
> > make sure VLANs fit?
> > 
> I took a look and, yes, this does seem to be needed. It will only be
> needed for the case where a vlan is in use and hwtagging is disabled,
> if I read the code correctly.
> 
> 
> 
> Yes, or in the rare care where you configure your switch to pass the
> v_lan header through to the NIC.
> 
> 
> 
> Do you use vlans?
> 
> 
> (Answered in above email)
> 
> 
> 
> 
> 
> I've attached an updated patch.
> 
> It might be nice to have the printf() patch in the driver too, so
> we can see how big the ones that are too big are?
> 
> 
> 
> Yes, I'm going to leave those in until I know we have this fixed..
> will probably leave it in a while longer as it should only have a
> minor performance impact to iter-loop like that, and I'd like to see
> what the story is a few months down the road.
> 
> 
> Thanks for the patches, will have to start giving them code-names so
> we can keep them straight. :-) I guess we have printf, tsomax, and
> this one.
> 
>