Latency issues with buf_ring

[Barney Cordoba]

--- On Thu, 12/6/12, Robert Watson <rwatson at FreeBSD.org> wrote:

> From: Robert Watson <rwatson at FreeBSD.org>
> Subject: Re: Latency issues with buf_ring
> To: "Andre Oppermann" <oppermann at networx.ch>
> Cc: "Barney Cordoba" <barney_cordoba at yahoo.com>, "Adrian Chadd" <adrian at freebsd.org>, "John Baldwin" <jhb at freebsd.org>, freebsd-net at freebsd.org
> Date: Thursday, December 6, 2012, 4:39 AM
> On Tue, 4 Dec 2012, Andre Oppermann
> wrote:
> 
> > For most if not all ethernet drivers from 100Mbit/s the
> TX DMA rings are so large that buffering at the IFQ level
> doesn't make sense anymore and only adds latency.  So
> it could simply directly put everything into the TX DMA and
> not even try to soft-queue.  If the TX DMA ring is full
> ENOBUFS is returned instead of filling yet another
> queue.  However there are ALTQ interactions and other
> mechanisms which have to be considered too making it a bit
> more involved.
> 
> I asserted for many years that software-side queueing would
> be subsumed by increasingly large DMA descriptor rings for
> the majority of devices and configurations.  However,
> this turns out not to have happened in a number of
> scenarios, and so I've revised my conclusions there.  I
> think we will continue to need to support transmit-side
> buffering, ideally in the form of a set of "libraries" that
> device drivers can use to avoid code replication and
> integrate queue management features fairly transparently.
> 
> I'm a bit worried by the level of copy-and-paste between
> 10gbps device drivers right now -- for 10/100/1000 drivers,
> the network stack contains the majority of the code, and the
> responsibility of the device driver is to advertise hardware
> features and manage interactions with rings, interrupts,
> etc.  On the 10gbps side, we see lots of code
> replication, especially in queue management, and it suggests
> to me (as discussed for several years in a row at BSDCan and
> elsehwere) that it's time to do a bit of revisiting of
> ifnet, pull more code back into the central stack and out of
> device drivers, etc.  That doesn't necessarily mean
> changing notions of ownership of event models, rather,
> centralising code in libraries rather than all over the
> place.  This is something to do with some care, of
> course.
> 
> Robert


More troubling than that is the notion that the same code that's suitable
for 10/100Gb/s should be used in a 10Gb/s environment. 10Gb/s requires a
completely different way of thinking.

BC