Network TX/RX fairness is not honored by USB stack

[Pyun YongHyeon]

On Sat, Oct 02, 2010 at 08:41:57AM +0200, Hans Petter Selasky wrote:
> On Saturday 02 October 2010 02:11:00 Pyun YongHyeon wrote:
> > Hi,
> > 
> > I don't know how long it had been there but it seems current USB
> > stack does not honor fairness of TX/RX on USB ethernet controller.
> > Unidirectional performance test(UDP) or most-unidirectional
> > performance(TCP) test works well without problems. However if heavy
> > TX/RX traffic hits controller at the same time either TX or RX is
> > not served at all. I'm under the impression that whenever TX work
> > is done it seems USB reschedules next pending TX again instead of
> > processing RX such that RX is starved to death. This can be easily
> > reproduced on two hosts with the netperf performance test.
> > Whenever both hosts send tiny UDP datagrams to the other host
> > either TX or RX packet counters are not increasing until the end
> > of the UDP torture test. The number of EHCI interrupt is about 8K/s
> > while test is in progress so I think it reached its maximum
> > processing limit. After netperf testing, it can still process TX/RX
> > packets even though it dropped too many RX packets. But these
> > dropped packets are not counted so netstat(1) shows 0 dropped
> > frames even though it lost millions of packets.
> > 
> > Hans, do you have any idea what's going on here?
> > You can use the following netperf command on both hosts after
> > running netserver.
> > %netperf -c -H ip_addr_of_other_host -tUDP_STREAM -l 300 -- -m 1
> > 
> > Another odd thing I noticed is number of interrupts does not go
> > down to 0 after the testing. It constantly generates 1k/s
> > interrupts after that.
> 
> Maybe we are triggering a bug. Can you enable USB debugging to figure out what 
> data lengths are transmitted or received.
> 

In the middle of testing? If yes, that would be meaningless as it
would generate bunch of messages. The test case generates payload
size 1 UDP datagrams with full speed so enabling debug messages
will change timing. Note, I'm exercising number of packets per
second, not number of bytes per second.

> USB EHCI uses round robin, so this is either USB device problem or a test-
> program software failure.
> 

I'm pretty sure the benchmark program is not broken, so either
axe(4) or USB stack could be wrong here. I see three issues from
the UDP torture test.
 - Either TX or RX could be starved to death. If you start TX test
   first, RX would be stuck. If you start RX test first, TX would
   be stuck.
 - The number of packets sent or received are much lower than
   expected.
   For TX case, the number of packets sent per second is exactly 8k
   which is much less than that of non-USB controllers. For gigabit
   controllers number of TX packets could be several hundred 
   thousands per second. For RX packets it shows 14K/s packets with
   8K/s interrupts. I thought USB ethernet controllers can send
   more than 8k packets per second. Because the number of
   interrupts per second and 8k packets per second is the same,
   this also make me wonder there could be some relations there.
 - Number of interrupts does not go back to 0 after the testing.

I'll let you know if I find some clue but it may take long time as
I'm not familiar with USB stack. :-(

> Check the CPU usage of the host computer during the test. Do you see anything?
> 

I didn't notice odd thing except 8k/s interrupts.

> > The only way I stop that interrupts was to
> > down the ue0 interface with "ifconfig ue0 down" command.
> 
> --HPS