bizarre em + TSO + MSS issue in RELENG_7

[Kip Macy]

On Nov 17, 2007 5:28 PM, Mike Andrews <mandrews at bit0.com> wrote:
>
> Kip Macy wrote:
> > On Nov 17, 2007 3:23 PM, Mike Andrews <mandrews at bit0.com> wrote:
> >> On Sat, 17 Nov 2007, Kip Macy wrote:
> >>
> >>> On Nov 17, 2007 2:33 PM, Mike Andrews <mandrews at bit0.com> wrote:
> >>>> On Sat, 17 Nov 2007, Kip Macy wrote:
> >>>>
> >>>>> On Nov 17, 2007 10:33 AM, Denis Shaposhnikov <dsh at vlink.ru> wrote:
> >>>>>> On Sat, 17 Nov 2007 00:42:54 -0500 (EST)
> >>>>>> Mike Andrews <mandrews at bit0.com> wrote:
> >>>>>>
> >>>>>>> Has anyone run into problems with MSS not being respected when using
> >>>>>>> TSO, specifically on em cards?
> >>>>>> Yes, I wrote about this problem on the beginning of 2007, see
> >>>>>>
> >>>>>>     http://tinyurl.com/3e5ak5
> >>>>>>
> >>>>> if_em.c:3502
> >>>>>        /*
> >>>>>         * Payload size per packet w/o any headers.
> >>>>>         * Length of all headers up to payload.
> >>>>>         */
> >>>>>        TXD->tcp_seg_setup.fields.mss = htole16(mp->m_pkthdr.tso_segsz);
> >>>>>        TXD->tcp_seg_setup.fields.hdr_len = hdr_len;
> >>>>>
> >>>>>
> >>>>> Please print out the value of tso_segsz here. It appears to be being
> >>>>> set correctly. The only thing I can think of is that t_maxopd is not
> >>>>> correct. As tso_segsz is correct here:
> >>>>
> >>>> It repeatedly prints 1368 during a 1 meg file transfer over a connection
> >>>> with a 1380 MSS.  Any other printf's I can add?  I'm working on a web page
> >>>> with tcpdump / firewall log output illustrating the issue...
> >>> Mike -
> >>> Denis' tcpdump output doesn't show oversized segments, something else
> >>> appears to be happening there. Can you post your tcpdump output
> >>> somewhere?
> >>
> >> URL sent off-list.
> >
> >        if (tso) {
> >                m->m_pkthdr.csum_flags = CSUM_TSO;
> >                m->m_pkthdr.tso_segsz = tp->t_maxopd - optlen;
> >        }
> >
> >
> > Please print the value of maxopd and optlen under "if (tso)" in
> > tcp_output. I think the calculated optlen may be too small.
>
>
> maxopt=1380 - optlen=12 = tso_segsz=1368
>
> Weird though, after this reboot, I had to re-copy a 4 meg file 5 times
> to start getting the firewall to log any drops.  Transfer rate was
> around 240KB/sec before the firewall started to drop, then it went down
> to about 64KB/sec during the 5th copy, and stayed there for subsequent
> copies.  The actual packet size the firewall said it was dropping was
> varying all over the place still, yet the maxopt/optlen/tso_segsz values
> stayed constant.  But it's interesting that it didn't start dropping
> immediately after the reboot -- though the transfer rate was still
> sub-optimal.

Ok, next theory :D. You shouldn't be seeing "bad len" packets from
tcpdump. I'm wondering if that means you're sending down more than
64k. Can you please print out the value of mp->m_pkthdr.len around the
same place that you printed out tso_segsz? 64k is the generally
accepted limit for TSO, I'm wondering if the card firmware does
something weird if you give it more.

Thanks.

 -Kip