4.7 vs 5.2.1 SMP/UP bridging performance

[Gerrit Nagelhout]

>>>I would like to move to CURRENT for new hardware support, and the 
>>>ability to properly use multi-threading in user-space, but can't do
>>>this until the performance bottlenecks are solved.  I realize that 
>>>5.x is still a work in progress and hasn't been tuned as well as 4.7 
>>>yet, but are there any plans for optimizations in this area?  Does 
>>>anyone have any suggestions on what else I can try?
>> 
>> 
>> Try rwatson's netperf patches:
>> 
>>   http://www.watson.org/~robert/freebsd/netperf/
>> 
>> There is at least one outstanding panic condition known, but more
>> testing will be a great help.
>> 
>> Kris
>> 
>> P.S. You didn't mention the status of WITNESS, but I'm assuming you
>> read the docs and disabled it since it's a huge performance killer.

>WITNESS and INVARIANTS are turned off for the 5.2.1 release bits.
>However, the debug.mpsafenet sysctl is also turned off.  Turning this
>on might give a significant performance boost for bridging.

>Scott

Thanks for all the responses so far.  WITNESS is definitely disabled, 
as are the other INVARIANTS.  I had a look through the netperf patches, 
but I don't think they will affect bridging very much.  They seem be 
directed more towards the socket layer and above.

I still think that one of the bigger bottlenecks is the cost of all
the mutexes in SMP mode, and some of the new bus_dma and mbuf code that
was introduced.  

With previous platforms I have worked on (vxWorks), we had similar 
issues, and ended up pushing buckets of packets through the data path, 
so each mutex was only taken once for every 10-100 packets.

Also, polling is currently done by only one CPU at a time.  If this 
were changed to have multiple threads poll multiple devices at the
same time, the performance should become much better.

Thanks,

Gerrit