two problems in dev/e1000/if_lem.c::lem_handle_rxtx()

[Luigi Rizzo]

On Thu, Jan 17, 2013 at 08:48:25AM -0800, Adrian Chadd wrote:
> There's also the subtle race condition in TX and RX handling that
> re-queuing the taskqueue gets around.
> 
> Which is:
> 
> * The hardware is constantly receiving frames , right until you blow
> the FIFO away by filling it up;
> * The RX thread receives a bunch of frames;
> * .. and processes them;
> * .. once it's done processing, the hardware may have read some more
> frames in the meantime;

(in the "lem" driver this cannot happen until you release some
rx slots, which only happens once at the end of the lem_rxeof() routine,
not long before re-enabling interrupts)

> * .. and the hardware may have generated a mitigated interrupt which
> you're ignoring, since you're processing frames;
> * So if your architecture isn't 100% paranoid, you may end up having
> to wait for the next interrupt to handle what's currently in the
> queue.
> 
> Now if things are done correct:
> 
> * The hardware generates a mitigated interrupt
> * The mask register has that bit disabled, so you don't end up receiving it;
> * You finish your RX queue processing, and there's more stuff that's
> appeared in the FIFO (hence why the hardware has generated another
> mitigated interrupt);
> * You unmask the interrupt;
> * .. and the hardware immediately sends you the MSI or signals an interrupt;
> * .. thus you re-enter the RX processing thread almost(!) immediately.

the problem i was actually seeing are slightly different, namely:
- once the driver lags behind, it does not have a chance to recover
  even if there are CPU cycles available, because both interrupt 
  rate and packets per interrupt are capped.
- much worse, once the input stream stops, you have a huge backlog that
  is not drained. And if, say, you try to ping the machine,
  the incoming packet is behind another 3900 packets, so the first
  interrupt drains 100 (but not the ping request, so no response),
  you keep going for a while, eventually the external world sees the
  machine as not responding and stops even trying to talk to it.

cheers
luigi