network performance

[Stefan Lambrev]

Greetings,

Kris Kennaway wrote:
> Stefan Lambrev wrote:
>
>>> I'll use again hwpmc and LOCK_PROFILING to see what's going on.
>>> And will try the same benchmark on quad core processor as now 
>>> numbers of cores/cpus matter :)
>>>
>> Here are promised results - http://89.186.204.158/lock_profiling-8.txt
Finally I got pmcstat working - http://89.186.204.158/hwpmc-p4.txt
The stats are gathered during 600kpp incoming.
I think that syncache or what calls MD5Transform is not SMP able, and 
that's why outgoing 250kpps is the limit that I can't beat.
>
> Thanks.  There is further work needed on the route locking, and also 
> you are hitting limitations of the em driver (or possibly hardware; if 
> you only have a single transmit queue then outbound packets from 
> multiple CPUs have to be serialized in the driver no matter what).  
> Hopefully there will be further improvements in the coming months, and 
> these changes will also migrate into CVS.
>
> If you want to start hacking things to see how much further progress 
> is feasible, you can apply the attached hack that nulls out all route 
> locking :)  This should be OK as long as your routes are not changing, 
> although you might get some spam on the console (if this is excessive, 
> comment out the printfs also ;-).  It may not help much though, all 
> the contention will probably just fall through onto the ethernet driver.
I'll do tomorrow. I still did not tested how if_lagg will perform in 
current situation.
I thing that I didn't reached the maximum of the network card today.
In this configuration best values was 1,122,674 incoming packets + 
210kpps outgoing with both cpu cores 100% busy.
systat -ifstat showed:
            em0  in     63.985 MB/s         63.985 MB/s           47.465 GB
                 out    11.674 MB/s         11.723 MB/s           12.975 GB

It is good, that now I hit a bottleneck that can be upgraded, if needed :)
>
>> Btw I got kernel panic first time when I run sysctl 
>> debug.lock.prof.stats
>
> Yeah, it is a bit broken in 8.0 even in CVS.  Also make sure not to 
> reset it while the CPUs are loaded :)
>
>> I'm still trying to get hwpmc working with my cpu's and new kernel.
>> Do you have any patches Kris?
>> Is it supposed to work with your sources on my CPU?
>> I can fetch your latest src/lib/libpmc from from p4 if this will help :)
>
> It works on my systems...try with libpmc from my branch, make sure to 
> install the new includes first and then rebuild and reinstall libpmc 
> and pmcstat.  I have attached a patch against the CVS libpmc which 
> might be easier than checking it out from p4...it relies on kernel 
> changes also though, which are in the kernel you already have but not 
> in CVS.
>
> Kris
Yes there was a problem with the installed include file, I edit it by 
hand, compiled new libpmc and it works again :)