Fwd: Multiple TX queue support in nic driver

[K. Macy]

forwarding response to the list

> ---------- Forwarded message ----------
> From: Naresh <gbal.naresh at gmail.com>
> To: "freebsd-drivers at freebsd.org" <freebsd-drivers at freebsd.org>
> Date: Thu, 15 Sep 2011 20:26:16 +0530
> Subject: Multiple TX queue support in nic driver
> Hi All,
>
> I am working on a 10G Ethernet driver.  I have a requirement to support multiple TX queues. I looked into IXGBE and CXGB drivers for reference, I have some questions regarding the implementation.
>
> 1) For supporting multiple TXQs, I see the above mentioned drivers create a ring buffer in the driver for each hardware TX queue, and queue the packets from stack to the ring buffer before queueing on the hardware queue. I don't understand why to create a ring buffer for supporting multiple TXQS. There might be a very good reason for them to do that way, but I am unable to get that.

The software queues are only used if the txq lock is already held by
another thread. What alternative are you thinking of? Waiting for the
txq lock would be much slower than a simple ring buffer enqueue. A
global queue where threads randomly pulled off and put on an available
txq would cause packet reording within individual flows.

> Is it ok to queue the packets to hardware queue directly based on the flowid supplied in the mbuf  packet header in if_transmit routine ?

Yes. This is what those drivers already do if the txq lock is not held.



> 2) Regarding flowid, for the first TX packet the flowid is not set and the packet goes out on the default TX queue and when the corresponding RX response comes to hardware it queues   the packet to appropriate RXQ based on the rss hash. The driver then before posting RX packet to stack it should set the flowid in packet to RXQ index.
> After that the TX packets from stack for that TCP connection will come with flowid supplied by us. Is this understanding correct ?

Yes. For using multiple queues with UDP unidirectionally one needs to
either use the flowtable or change the inpcb creation routine.

Cheers