From nobody Tue Jun 28 21:43:39 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DDCAE8710E2 for ; Tue, 28 Jun 2022 21:43:54 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) Received: from sender4-op-o13.zoho.com (sender4-op-o13.zoho.com [136.143.188.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LXdQK3yvRz4Zfh for ; Tue, 28 Jun 2022 21:43:53 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) ARC-Seal: i=1; a=rsa-sha256; t=1656452621; cv=none; d=zohomail.com; s=zohoarc; b=c1v6eUwahofHTy8ICsY+xUS7nmOLL1bj0y3lpNx4GS5bp8b82vOAKskz2TxumNY+4w0+ChlhyjOgR72edHXzhIZ7Qv4jCY2mRlxt5ZAk1E/+OF8HEUI+MXiPg5CknG0jOh5Dl9L+8mBNqmHPxvarC9alt5NzPhOPwsGvfBFQWUQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1656452621; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:Reply-To:References:Subject:To; bh=eYsh6ZDG4Qiw/1POUiz24NsqtXKUt4nHYc/85rqiomw=; b=NCFXtI2Kgx5WHP3caHeK5RnvtQMauH1TC4H9HVFccwa9oPCWV/nbuzRdpAUECjF+72FhTk2QhRRmf0mrK21V4AvEFpgvQq2wazjAgL6I9uGMmfZcoGE08MseOVuPty4luN7u2F0tWrCnJ+c/cU1/iUuneToK4WrO6mYbxvYQS1c= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=swiftsmsgateway.com; spf=pass smtp.mailfrom=michal.jakubik@zoho.com; dmarc=pass header.from= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:reply-to:to:cc:message-id:in-reply-to:references:subject:mime-version:content-type:user-agent; b=vTWG2uzeBMoK97XF0eZDF85+1MPUrOanD8HD69F4LzNCVo+qNvmG/6oasVKsEqOogYmngi18aqAO TiZTg1Sh7Jz4UEQZmNjLSo2Od/VUJz3CYENOGSK0Bop1wi6yHWpc DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1656452621; s=zoho; d=swiftsmsgateway.com; i=mike.jakubik@swiftsmsgateway.com; h=Date:Date:From:From:Reply-To:Reply-To:To:To:Cc:Cc:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type; bh=eYsh6ZDG4Qiw/1POUiz24NsqtXKUt4nHYc/85rqiomw=; b=vo1AUf+LBbdq0q6dC27zNIlzPeqm6UytjnprJWWbP1A9to7XtVbj5lsBumvwhOQv IyN5B+LfW0+jE1nxpSPMqGNIG2lRIoQpfXppY1NutbK1ij4Madp2oDknVpQVraX4KAs QQ/hTUn01Mn/wsaCoh1nzu+tedyDg4xb20dH+mOI= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1656452619992546.9815905948282; Tue, 28 Jun 2022 14:43:39 -0700 (PDT) Date: Tue, 28 Jun 2022 17:43:39 -0400 From: Mike Jakubik Reply-To: mike.jakubik@swiftsmsgateway.com To: "freebsd-net" Cc: "Alexander V. Chernikov" , "Dave Cottlehuber" Message-Id: <181ac451eba.10ec692122616769.3280827109477949645@swiftsmsgateway.com> In-Reply-To: <9892d8f8-df5d-4e73-ab48-cb5909b4f9c6@www.fastmail.com> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> <1816f19416a.b852ce5b2189187.4131912798685804323@swiftsmsgateway.com> <9892d8f8-df5d-4e73-ab48-cb5909b4f9c6@www.fastmail.com> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_8280592_2002495593.1656452619962" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Rspamd-Queue-Id: 4LXdQK3yvRz4Zfh X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=swiftsmsgateway.com header.s=zoho header.b=vo1AUf+L; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (mx1.freebsd.org: domain of michal.jakubik@zoho.com designates 136.143.188.13 as permitted sender) smtp.mailfrom=michal.jakubik@zoho.com X-Spamd-Result: default: False [-2.51 / 15.00]; HAS_REPLYTO(0.00)[mike.jakubik@swiftsmsgateway.com]; XM_UA_NO_VERSION(0.01)[]; R_SPF_ALLOW(-0.20)[+ip4:136.143.188.0/24]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[swiftsmsgateway.com:+]; FORGED_SENDER(0.30)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[zoho.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; ASN(0.00)[asn:2639, ipnet:136.143.188.0/23, country:US]; NEURAL_HAM_MEDIUM(-0.96)[-0.955]; R_DKIM_ALLOW(-0.20)[swiftsmsgateway.com:s=zoho]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; ARC_ALLOW(-1.00)[zohomail.com:s=zohoarc:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[swiftsmsgateway.com]; NEURAL_SPAM_SHORT(0.64)[0.637]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[136.143.188.13:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_8280592_2002495593.1656452619962 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, So basically this is my conclusion, if I cpuset iperf on at least the recei= ving end i get great performance. Anything outside of that is random. I've = tried just about every network tuning knob in FreeBSD as well as what Mella= nox recommends in their driver manual, none of these make any significant i= mpact, sometimes they even reduce performance. So I m hoping when this goes= into production the scheduler will be sane enough to do the same, but we s= hall see. Thanks. ---- On Fri, 17 Jun 2022 11:03:04 -0400 Dave Cottlehuber wrote --- On Fri, 17 Jun 2022, at 02:38, Mike Jakubik wrote:=20 > Hi,=20 >=20 > I believe you hit the nail on the head! I am now getting consistent=20 > high speeds, even higher than on Linux! Is this a problem with the=20 > scheduler? Should someone in that area of expertise be made aware of=20 > this? More importantly i guess, would this affect real world=20 > performance, these servers will be running RabbitMQ (it uses quite a=20 > bit of bandwidth) and PostgreSQL w/ replication.=20 =20 pinning cores for unimpeded access is very common for high performance syst= ems. Do this both for the nics and also your apps. Be mindful of the NUMA t= opooogy.=20 =20 You should look into both the erlang scheduler flags for core pinning, and= also ensuring that your erlang processes have unimpeded access to their ow= n cores too. A reasonable approach is to make a simple cowboy or Phoenix ap= p and hammer it with wrk or similar load tool to get a feel for things, and= then profile and tune your own app based on those specific results.=20 =20 For rabbit there is an excellent load testing tool from the pivotal team if= you don=E2=80=99t have suitable load generators yourselves.=20 =20 Tsung is an excellent tool if you put in the work to craft something specif= ic for your use case.=20 =20 Please post back to the list with your specific findings and nic/ tcp tunab= les, these are very helpful for the next person!=20 =20 Dave=20 =20 Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. ------=_Part_8280592_2002495593.1656452619962 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
Hi,

So basically this= is my conclusion, if I cpuset iperf on at least the receiving end i get gr= eat performance. Anything outside of that is random. I've tried just about = every network tuning knob in FreeBSD as well as what Mellanox recommends in= their driver manual, none of these make any significant impact, sometimes = they even reduce performance. So I m hoping when this goes into production = the scheduler will be sane enough to do the same, but we shall see.

Thanks.



---- On Fri, 17 Jun 2022 11:03:04 -0400 Dave Cottlehuber <= ;dch@skunkwerks.at> wrote ---

On Fri, 17 Jun 2022, at 02:38, Mike Jakubik wrote= :
> Hi,
>
> I believe you hit the nail on the head! I = am now getting consistent
> high speeds, even higher than on Linux! = Is this a problem with the
> scheduler? Should someone in that area = of expertise be made aware of
> this? More importantly i guess, woul= d this affect real world
> performance, these servers will be runnin= g RabbitMQ (it uses quite a
> bit of bandwidth) and PostgreSQL w/ re= plication.

pinning cores for unimpeded access is very common for h= igh performance systems. Do this both for the nics and also your apps. Be m= indful of the NUMA topooogy.

You should look into both the erlang= scheduler flags for core pinning, and also ensuring that your erlang proce= sses have unimpeded access to their own cores too. A reasonable approach is= to make a simple cowboy or Phoenix app and hammer it with wrk or similar l= oad tool to get a feel for things, and then profile and tune your own app b= ased on those specific results.

For rabbit there is an excellent l= oad testing tool from the pivotal team if you don=E2=80=99t have suitable l= oad generators yourselves.

Tsung is an excellent tool if you put i= n the work to craft something specific for your use case.

Please p= ost back to the list with your specific findings and nic/ tcp tunables, the= se are very helpful for the next person!

Dave


Mike Jakubik
https://www.swiftsmsgateway.com/

Disclaim= er: This e-mail and any attachments are intended only for the use of the ad= dressee(s) and may contain information that is privileged or confidential. = If you are not the intended recipient, or responsible for delivering the in= formation to the intended recipient, you are hereby notified that any disse= mination, distribution, printing or copying of this e-mail and any attachme= nts is strictly prohibited. If this e-mail and any attachments were receive= d in error, please notify the sender by reply e-mail and delete the origina= l message.


------=_Part_8280592_2002495593.1656452619962--