Is there any way to limit the amount of data in an mbuf chain submitted to a driver?

Sat May 18 12:11:25 UTC 2013

Can you outline the changes? Anything with the general processing? I have to makea case to hold off a deployment.
and what happened to 9, 10, and 11?
Laurie

--- On Fri, 5/17/13, Jack Vogel <jfvogel at gmail.com> wrote:

From: Jack Vogel <jfvogel at gmail.com>
Subject: Re: Is there any way to limit the amount of data in an mbuf chain submitted to a driver?
To: "Laurie Jennings" <laurie_jennings_1977 at yahoo.com>
Cc: "FreeBSD Net" <freebsd-net at freebsd.org>, "Adrian Chadd" <adrian at freebsd.org>, "Richard Sharpe" <realrichardsharpe at gmail.com>
Date: Friday, May 17, 2013, 3:32 PM

No Laurie, I ran into an issue that I needed to resolve, and then my
validation engineer
went out of town a couple days. Should be early next week.

Jack

On Fri, May 17, 2013 at 3:25 PM, Laurie Jennings <
laurie_jennings_1977 at yahoo.com> wrote:

> Did you get to commit that new code yet?
>
> Laurie
>
> --- On *Sat, 5/4/13, Jack Vogel <jfvogel at gmail.com>* wrote:
>
>
> From: Jack Vogel <jfvogel at gmail.com>
> Subject: Re: Is there any way to limit the amount of data in an mbuf chain
> submitted to a driver?
> To: "Richard Sharpe" <realrichardsharpe at gmail.com>
> Cc: "FreeBSD Net" <freebsd-net at freebsd.org>, "Adrian Chadd" <
> adrian at freebsd.org>
> Date: Saturday, May 4, 2013, 2:18 PM
>
>
> Ahh, Twinville, new hardware :)  The version at the tip is 2.5.8 and I am
> working on version 2.5.12 internally that I hope to commit next week...
> so your version is "a bit old" :) I would do some testing on newer code.
>
> Jack
>
>
>
> On Sat, May 4, 2013 at 1:54 PM, Richard Sharpe
> <realrichardsharpe at gmail.com<http://mc/compose?to=realrichardsharpe@gmail.com>
> >wrote:
>
> > On Sat, May 4, 2013 at 1:41 PM, Jack Vogel <jfvogel at gmail.com<http://mc/compose?to=jfvogel@gmail.com>>
> wrote:
> > > If you don't use TSO you will hurt your TX performance significantly
> from
> > > the tests that I've run. What exactly is the device you are using, I
> > don't
> > > have the source in front of me now, but I'm almost sure that the limit
> is
> > > not 64K but 256K, or are you using some ancient version of the driver?
> >
> >             ix0 pnpinfo vendor=0x8086 device=0x1528 subvendor=0x8086
> > subdevice=0x0001 class=0x020000 at slot=0 function=0
> >             ix1 pnpinfo vendor=0x8086 device=0x1528 subvendor=0x8086
> > subdevice=0x0001 class=0x020000 at slot=0 function=1
> >
> > The version calls itself ixgbe-2.4.4 ...
> >
> > Hmmm, copyright is 2001-2010 ... so perhaps a bit old.
> >
> > > Jack
> > >
> > >
> > >
> > > On Sat, May 4, 2013 at 1:30 PM, Richard Sharpe <
> > realrichardsharpe at gmail.com<http://mc/compose?to=realrichardsharpe@gmail.com>
> >
> > > wrote:
> > >>
> > >> On Sat, May 4, 2013 at 10:39 AM, Adrian Chadd <adrian at freebsd.org<http://mc/compose?to=adrian@freebsd.org>
> >
> > wrote:
> > >> > On 4 May 2013 06:52, Richard Sharpe <realrichardsharpe at gmail.com<http://mc/compose?to=realrichardsharpe@gmail.com>
> >
> > wrote:
> > >> >> Hi folks,
> > >> >>
> > >> >> I understand better why I am seeing EINVAL intermittently when
> > sending
> > >> >> data from Samba via SMB2.
> > >> >>
> > >> >> The ixgbe driver, for TSO reasons, limits the amount of data that
> can
> > >> >> be DMA'd to 65535 bytes. It returns EINVAL for any mbuf chain
> larger
> > >> >> than that.
> > >> >>
> > >> >> The SO_SNDBUF for that socket is set to 131972. Mostly there is
> less
> > >> >> than 64kiB of space available, so that is all TCP etc can put into
> > the
> > >> >> socket in one chain of mbufs. However, every now and then there is
> > >> >> more than 65535 bytes available in the socket buffers, and we have
> an
> > >> >> SMB packet that is larger than 65535 bytes, and we get hit.
> > >> >>
> > >> >> To confirm this I am going to set SO_SNDBUF back to the default of
> > >> >> 65536 and test again. My repros are very reliable.
> > >> >>
> > >> >> However, I wondered if my only way around this if I want to
> continue
> > >> >> to use SO_SNDBUF sizes larger than 65536 is to fragment large mbuf
> > >> >> chains in the driver?
> > >> >
> > >> > Hm, is this is a problem without TSO?
> > >>
> > >> We are using the card without TSO, so I am thinking of changing that
> > >> limit to 131072 and retesting.
> > >>
> > >> I am currently testing with SO_SNDBUF=32768 and have not hit the
> > problem.
> > >>
> > >> > Is the problem that the NIC can't handle a frame that big, or a
> buffer
> > >> > that big?
> > >> > Ie - if you handed the hardware two descriptors of 64k each, for the
> > >> > same IP datagram, will it complain?
> > >>
> > >> I can't find any documentation, but it seems that with TSO it cannot
> > >> handle a frame that big. Actually, since we are not using TSO, there
> > >> really should not be a problem with larger frames.
> > >>
> > >> > Or do you need to break it up into two separate IP datagrams, facing
> > >> > the driver, with a maximum size of 64k each?
> > >>
> > >> Not sure, but it looks like we need to do that.
> > >>
> > >>
> > >> --
> > >> Regards,
> > >> Richard Sharpe
> > >> (何以解憂？唯有杜康。--曹操)
> > >> _______________________________________________
> > >> freebsd-net at freebsd.org<http://mc/compose?to=freebsd-net@freebsd.org>mailing list
> > >> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org<http://mc/compose?to=freebsd-net-unsubscribe@freebsd.org>
> "
> > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Richard Sharpe
> > (何以解憂？唯有杜康。--曹操)
> >
> _______________________________________________
> freebsd-net at freebsd.org <http://mc/compose?to=freebsd-net@freebsd.org>mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org<http://mc/compose?to=freebsd-net-unsubscribe@freebsd.org>
> "
>
>
_______________________________________________
freebsd-net at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"