IF_HANDOFF vs. IFQ_HANDOFF

[Bruce Evans]

On Tue, 20 Jun 2006, Pyun YongHyeon wrote:

> On Tue, Jun 20, 2006 at 05:11:18PM +1000, Bruce Evans wrote:
> > On Mon, 19 Jun 2006, Pyun YongHyeon wrote:

> > My sk_start() tries to fill the tx ring (to length 512) and then put
> > an interrupt mark only on the last fragment in a packet nearest to 32
> > from the end, so in the best case N is about 480, but it us less if
> > tx is not streaming.  Cases where there is not much choice are harder
> > to program.  I had some success with removing interrupt marks and with
> > dummy packets of length 0 whose purpose is just to hold an interrupt
> > mark, but I don't trust those methods.  I didn't try putting an
> > interrupt mark on fragments in the middle of a packet.  That would be
> > simpler if it works.
>
> I think it would take a long time to generate an Tx completion
> interrupt for committed frames(every frame vs. the last frame) The
> hardware may have some free Tx descriptors before generating an
> Tx completion interrupt. I guess it would be more efficient if we
> know there are some free Tx descriptors and use it before waiting for
> an Tx completion interrupt. Just waiting for a completion interrupt
> would add additional latency. Anyway, I have to experiment it.

Having hundreds or thousands if packets in flight already gives large
latency for new packets.  Then it makes little difference to latency
when tx interrupts occur.  New packets don't go out faster if tx
interrupts occur earlier -- they just get put into the h/w queue earlier
and wait there longer if that queue is long,  They might go out slower
if they get reordered in higher queues.

> > >We may also need to add a routine to reclaim pending Tx descriptors
> > >before sending frames in sk_start if number of available Tx descriptors
> > >are less then a threshold.
> >
> > I'm not sure what you mean here.  If there are < 32 tx descriptors
> > available, AND there is an (active) descriptor with an interrupt mark,
> > then my sk_start() just sets IFF_OACTIVE and returns.  The case where
> > there are < 32 tx descriptors but no descriptor with an interrupt mark
> > is trickier: a mark must be added, and I don't trust adding it to an
> > active packet, so it must be added to a new packet, but it might be
> > impossible to add one for the following reasons:
> > - no space.  The magic 32 is hopefully enough.
> > - no packets in the ifq.  My sk_start() tries to leave a spare one when
> >   one might be needed, but I think upper layers can eat it.
> > A dummy packet of length 0 can be used to handle both cases but may be
> > bad for the network -- does the hardware send a frame with no data?
>
> I can't sure.
> Since you know when you have to insert interrupt mark in sk_encap
> I think you can use m_defrag and set SK_TXCTL_EOF_INTR.

Yes, that might work well.

I don't actually know where to insert the interrupt mark in sk_encap().
My sk encap() doesn't set either SK_TXCTL_EOF_INTR or SK_TXCTL_OWN.
Then after filling the tx ring as far as possible, sk_start() checks
the ring indexes to determine where to put SK_TXCTL_EOF_INTR and puts
SK_TXCTL_EOF_INTR on one fragment and SK_TXCTL_OWN on all the fragments
that it just queued.  It is necessary to determine the final ring state
in some way before deciding where to put SK_TXCTL_EOF_INTR, and this
way seems best.

> > >However I don't know how the driver should handle transmit errors
> > >occurred between interrupt-less Tx operations. Just flushing all
> > >committed frames would result in poor TCP performance.
> >
> > Doesn't the hardware just proceed to the next packet without interrupting
> > (except possibly for a special error interrupt), and anyway act the same
> > as if the interrupt were delayed by interrupt moderation?  Errors for
> > individual packets don't seem to be detected or reported in either case.
>
> Yes that is the problem. It seems that there is no way to know which
> packet caused Tx errors and I think we have no choice but flushing
> entire FIFOs. SK just flushes all frames in FIFO if it detect Tx
> FIFO underrun or Rx FIFO overflow. But I can't sure how Yukon should
> handle this case. The flushing routine in sk is guess work from
> Linux skge implementation and I don't know internal details of Yukon
> hardware. Since Yukon uses defferent registers to flush FIFOs and the
> existence of unique registers related with interrupt and FIFOs I guess
> it uses completely different approach.

The old version that I use doesn't even reference if_oerrors :-).  Do
you mean flushing the whole ring?  Ideally there would be an error mark
on old unOWNed packets but there would be no point in flushing those,
and if the interrupt is delayed there would just be more old packets and
fewer OWNed packets that would be lost by flushing the whole ring.

Bruce