Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

Hi,



So basically this is my conclusion, if I cpuset iperf on at least the receiving end i get great performance. Anything outside of that is random. I've tried just about every network tuning knob in FreeBSD as well as what Mellanox recommends in their driver manual, none of these make any significant impact, sometimes they even reduce performance. So I m hoping when this goes into production the scheduler will be sane enough to do the same, but we shall see.



Thanks.






---- On Fri, 17 Jun 2022 11:03:04 -0400 Dave Cottlehuber <dch@skunkwerks.at> wrote ---



On Fri, 17 Jun 2022, at 02:38, Mike Jakubik wrote: 
> Hi, 
> 
> I believe you hit the nail on the head! I am now getting consistent 
> high speeds, even higher than on Linux! Is this a problem with the 
> scheduler? Should someone in that area of expertise be made aware of 
> this? More importantly i guess, would this affect real world 
> performance, these servers will be running RabbitMQ (it uses quite a 
> bit of bandwidth) and PostgreSQL w/ replication. 
 
pinning cores for unimpeded access is very common for high performance systems. Do this both for the nics and also your apps. Be mindful of the NUMA topooogy. 
 
You should look into both the  erlang scheduler flags for core pinning, and also ensuring that your erlang processes have unimpeded access to their own cores too. A reasonable approach is to make a simple cowboy or Phoenix app and hammer it with wrk or similar load tool to get a feel for things, and then profile and tune your own app based on those specific results. 
 
For rabbit there is an excellent load testing tool from the pivotal team if you don’t have suitable load generators yourselves. 
 
Tsung is an excellent tool if you put in the work to craft something specific for your use case. 
 
Please post back to the list with your specific findings and nic/ tcp tunables, these are very helpful for the next person! 
 
Dave 
 





Mike Jakubik

https://www.swiftsmsgateway.com/



Disclaimer: This e-mail and any attachments are intended only for the use of the addressee(s) and may contain information that is privileged or confidential. If you are not the intended recipient, or responsible for delivering the information to the intended recipient, you are hereby notified that any dissemination, distribution, printing or copying of this e-mail and any attachments is strictly prohibited. If this e-mail and any attachments were received in error, please notify the sender by reply e-mail and delete the original message.