Major performance hit with ToS setting

[Kevin Oberman]

On Thu, May 24, 2012 at 4:43 PM, Bjoern A. Zeeb
<bzeeb-lists at lists.zabbadoz.net> wrote:
> On 24. May 2012, at 22:55 , Kevin Oberman wrote:
>
>> When we set the ToS bits for less than best effort (also called
>> scavenger) on packets (ToS=32), performance on FreeBSD 8.2 is
>> terrible. It was as good as best effort on FreeBSD 7.3 (assuming no
>> congestion). I will look into what 9 does, but does anyone have an
>> idea of why 8.2 behaves so badly when ToS is set to 32?
>> Here is an example of what happens to performance:
>> nuttcp with ToS=0:
>> 622.5000 MB /   1.00 sec = 5221.7570 Mbps
>> 623.3125 MB /   1.00 sec = 5228.5883 Mbps
>> 624.0000 MB /   1.00 sec = 5234.4495 Mbps
>>
>> With ToS=32 (0x20):
>> 0.3750 MB /   1.00 sec =    3.1457 Mbps
>> 0.5000 MB /   1.00 sec =    4.1942 Mbps
>> 0.5000 MB /   1.00 sec =    4.1942 Mbps
>>
>> This is,of course, on a 10G interface. On 7.3 there is little
>> difference between the two. We are using cubic CC on the 8.2 system.
>
> This looks like a different problem than just TOS.  I assume however
> that not setting the TOS you get the same as expected performance?
>
> Which NIC?

The two nuttcp runs were identical and we got similar results with iperf.

All tests were using Myricom (mxge) cards.
-- 
R. Kevin Oberman, Network Engineer
E-mail: kob6558 at gmail.com