[RFC] serialising net80211 TX

[Monthadar Al Jaberi]

On Thu, Feb 14, 2013 at 6:14 AM, Adrian Chadd <adrian at freebsd.org> wrote:
> Hi,
>
> I'd like to work on the net80211 TX serialisation now. I'll worry
> about driver serialisation and if_transmit methods later.
>
> The 30 second version - it happens in parallel, which means preemption
> and multi-core devices can and will hit lots of subtle and
> hard-to-debug races in the TX path.
>
> We actually need an end-to-end serialisation method - not only for the
> 802.11 state (sequence number, correct aggregation handling, etc) but
> to line up 802.11 sequence number allocation with the encryption IV/PN
> values. Otherwise you end up with lots of crazy subtle out of order
> packets occuring. The other is the seqno/CCMP IV race between the raw
> transmit path and the normal transmit path. There are other nagging
> issues that I'm trying to resolve - but, one thing at a time.
>
> So there are three current contenders:
>
> * wrap everything in net80211 TX in a per-vap TX lock; grab it at the
> beginning of ieee80211_output() and ieee80211_start(), and don't
> release it until the frame is queued to something (a power save queue,
> an age queue, the driver.) That guarantees that the driver is called
> in lock-step with each frame being processed.
> * do deferred transmit- ie, the net80211 entry points simply queue
> mbufs to a queue, and a taskqueue runs over the ifnet queue and runs
> those frames in-order. There's no need for a lock here as there's only
> one sending context (either per-VAP or per-IC).

Seems like the best architectural wise, first-in first out. I am just
thinking of one can extend this too have like more than one queue,
more like the QoS concept, and each packet have a time-stamp assigned
to it.Would that help?

What I am thinking is more of an optimization maybe. the Wifi can run
in different modes BSS (AP/STA), IBSS and MBSS. And for the MBSS mode
there is not just data comming from up the stack, or sending beacons.
The MBSS also needs to handle data forwarding and management frame
propagation. Would it help to have different queues?

A mesh path discovery would be delayed too much if the complete system
is highly active? Should the path request frames have higher priority
to lets say data frames? Would that help for when sending beacons from
the stack too.

Maybe I am way off :)

> * A hybrid setup - use a per-vap TX lock; do a try-acquire on it and
> direct dispatch from the queue head if we have it; otherwise defer
> frames into a queue and have a taskqueue handle those.

That maybe gives the most speed up?

>
> 1) is what drivers like iwn(4) do internally.
> 2) is what I've tinkered with - but we become a slave to the
> scheduler. Task switching is expensive and unpredictable; doubly so
> for a non-preemption kernel. We'd have to run the TX taskqueue at some
> very high priority to get something resembling direct-dispatch
> behaviour.
> 3) is what the gige/10ge drivers do. They hold a big TX lock for each
> TX (from xxx_transmit() to hardware dispatch) and if they can't
> acquire the TX lock, they defer it to a drbd lockless ring buffer and
> service that via a taskqueue.
>
> I can implement any of the above. architecturally I'd prefer 2) - it
> massively simplifies and streamlines things, but the scheduling
> latency is just plain stab-worthy.I'm tempted to just do 1) for now
> and turn it into 3) if we need to.
>
> The main reason against doing 1) (and why 2) is nicer) is recursion -
> if the TX path wants to call the net80211 TX code for some odd reason,
> we'll hit lock recursion. I'd rather have the system crash at this
> point (and then fix the misbehaving driver) but that's just me.
>
> So - what do people think?
>
> Once this is done I'd like to make sure that the wifi chipset drivers
> do the same - ie, ensure that the frame order is preserved both
> between the normal and the raw xmit paths.
> That should fix all of the odd CCMP out of order crap that I see under
> heavy, heavy test conditions.
>
> Thanks,
>
>
> Adrian
> _______________________________________________
> freebsd-wireless at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-wireless
> To unsubscribe, send any mail to "freebsd-wireless-unsubscribe at freebsd.org"



-- 
Monthadar Al Jaberi