Major performance hit with ToS setting

[Kevin Oberman]

When we set the ToS bits for less than best effort (also called
scavenger) on packets (ToS=32), performance on FreeBSD 8.2 is
terrible. It was as good as best effort on FreeBSD 7.3 (assuming no
congestion). I will look into what 9 does, but does anyone have an
idea of why 8.2 behaves so badly when ToS is set to 32?
Here is an example of what happens to performance:
nuttcp with ToS=0:
622.5000 MB /   1.00 sec = 5221.7570 Mbps
623.3125 MB /   1.00 sec = 5228.5883 Mbps
624.0000 MB /   1.00 sec = 5234.4495 Mbps

With ToS=32 (0x20):
0.3750 MB /   1.00 sec =    3.1457 Mbps
0.5000 MB /   1.00 sec =    4.1942 Mbps
0.5000 MB /   1.00 sec =    4.1942 Mbps

This is,of course, on a 10G interface. On 7.3 there is little
difference between the two. We are using cubic CC on the 8.2 system.
-- 
R. Kevin Oberman, Network Engineer
E-mail: kob6558 at gmail.com