MQ Patch.

[Adrian Chadd]

[snip everything]

ok, I've reviewed the work.

TL;DR - it's a clearly correct step in the right direction, but I
think we need to just think it through a tad bit more first.

There have been queue discipline and queue management discussions in
the past. Randall's work is a good step in that direction.

I think though that we can take a step back up a little further.

* In terms of queuing frames into multiple queues - yes, we absolutely
should have an if_transmit() path to the driver that obeys "a" QoS
field in the mbuf and pushes it into the relevant queue - with
randalls work, it's in the driver, but it doesn't _have_ to be;
* In terms of queue servicing and management - we likely need to have
a variety of queue plugins that determine which frame from which queue
gets chosen next to hand to the hardware. The hardware may have
multiple queues! The hardware may have one queue! The application
developer may only want to use one queue! That should be flexible and
easy to plug into things.
* Then we need to support dropping frames during queue and dropping
frames during dequeue (ie, on its way to the hardware). That way we
can implement the currently interesting kinds of queue disciplines (eg
CODEL, etc.)
* Should this be done at the driver layer (ie it's a library that each
driver creates and owns), or as a layer above it, controlling the
network device (ie, the linux queue discipline method.)

So, my comments:

* I don't like how it's hard-coding drbr's into the drivers. Yes, the
underlying state should be a drbr for now. But I'd rather we have a
queue discipline plugin API that drivers create an instance of.
* It'll have methods to init/flush the rings, queue a frame into a
ring, dequeue a frame from a ring, be notified of transmit completions
so more work can be done, etc.
* For people who do latency-sensitive things, they can just bypass
this entirely and go straight to the hardware queues without going
through this kind of intermediary queue layer.

Randall - I think we can take your work and turn it into a net library
that implements your queue management routines. That way we can start
enabling people to tinker with it and replace it if they need to.

What do you think?