From nobody Fri Jun 17 15:03:04 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2D9B8836BFE for ; Fri, 17 Jun 2022 15:03:36 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LPj3W164rz3pcq for ; Fri, 17 Jun 2022 15:03:35 +0000 (UTC) (envelope-from dch@skunkwerks.at) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id BCE0E32009D7; Fri, 17 Jun 2022 11:03:26 -0400 (EDT) Received: from imap44 ([10.202.2.94]) by compute2.internal (MEProxy); Fri, 17 Jun 2022 11:03:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skunkwerks.at; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm3; t=1655478206; x= 1655564606; bh=BdX2iXv8p1meh9BD5KUXNryh15ju4k6hY9eiRH6blNg=; b=j lW4IUtCCGjSm2gcNJqleoT8Ojrz7QfUEEhmO1G+6lJ1kMsJ5/gc6dtT5m8lMuyVy 7K8SSKnqxy7+gPaoxq37Q3j6mJQy3jESpgS/+3/1HRBQJcGs1zVvE6mPSK7DUpSh LmUzRLTlMVqKr4uvF3f3epWHHHV4bQv4WUA8lYqgsRJ9QrEPpUp0G1wuC1e28vH+ YI4vkDLPneWCiKb5HvGLLzoz1CJwU80UzMYhzIBKFVrdw2MK7Nh54OqorwGWRwi5 Pm1kejKaA5BpViDX6qBCqtx6NXZQjhIQuqyXqSaMLKLqXjCksnFcyFXRdFgFDDp4 yB88tGqRxdUg9ApJLkXCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1655478206; x= 1655564606; bh=BdX2iXv8p1meh9BD5KUXNryh15ju4k6hY9eiRH6blNg=; b=u r6Vcrnt2yuBQ3INhnxmLrYtLOcUEXAXbVGdxS4/PFwYJHuMxT4/bNC4p5+6f3oC6 rBGaeghHL52sBd+IV+5LdyNBeer7DIMXIwa/nb77lljj4wfVnUVB+ypCnHndlsSL vmAVp/7nAsVmewlDiZFn8LRMCG7/K35qtXPl4Jb7aYDKBjWloyIjhBGgOmgXX0RD 1+w7hCFuyIxVYGuC30/ZA2pVl400tgYHSsEGdvV6Q3vsGU6wTyvY/ih7vDTrZo0K eEvCIyn5mf2w3VqBEBONxpQPBUAJqWmiIFXJ3tGHaphXGg5zipNhNWaDX6hWCGB5 KkfIG1VvCjoBVWcy4TtOQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddvhedgkeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtgfesthhqredtreerjeenucfhrhhomhepfdff rghvvgcuvehothhtlhgvhhhusggvrhdfuceouggthhesshhkuhhnkhifvghrkhhsrdgrth eqnecuggftrfgrthhtvghrnhepkefgudehgfeukeeiteeiledvieekgffgudetueeggfet uedtffehieevjeekhfejnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrg hilhhfrhhomhepuggthhesshhkuhhnkhifvghrkhhsrdgrth X-ME-Proxy: Feedback-ID: ic0e84090:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id BFED936A0071; Fri, 17 Jun 2022 11:03:24 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-712-gb9e94258b0-fm-20220610.001-gb9e94258 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 Message-Id: <9892d8f8-df5d-4e73-ab48-cb5909b4f9c6@www.fastmail.com> In-Reply-To: <1816f19416a.b852ce5b2189187.4131912798685804323@swiftsmsgateway.com> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> <1816f19416a.b852ce5b2189187.4131912798685804323@swiftsmsgateway.com> Date: Fri, 17 Jun 2022 17:03:04 +0200 From: "Dave Cottlehuber" To: "Mike Jakubik" , freebsd-net Cc: "Alexander V. Chernikov" Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4LPj3W164rz3pcq X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=skunkwerks.at header.s=fm3 header.b="j lW4IUt"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="u r6Vcrn"; dmarc=none; spf=pass (mx1.freebsd.org: domain of dch@skunkwerks.at designates 64.147.123.19 as permitted sender) smtp.mailfrom=dch@skunkwerks.at X-Spamd-Result: default: False [-3.59 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[skunkwerks.at:s=fm3,messagingengine.com:s=fm2]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.19]; MIME_GOOD(-0.10)[text/plain]; FREEFALL_USER(0.00)[dch]; DMARC_NA(0.00)[skunkwerks.at]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[skunkwerks.at:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MLMMJ_DEST(0.00)[freebsd-net]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; MID_RHS_WWW(0.50)[]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.19:from] X-ThisMailContainsUnwantedMimeParts: N On Fri, 17 Jun 2022, at 02:38, Mike Jakubik wrote: > Hi, > > I believe you hit the nail on the head! I am now getting consistent=20 > high speeds, even higher than on Linux! Is this a problem with the=20 > scheduler? Should someone in that area of expertise be made aware of=20 > this? More importantly i guess, would this affect real world=20 > performance, these servers will be running RabbitMQ (it uses quite a=20 > bit of bandwidth) and PostgreSQL w/ replication. pinning cores for unimpeded access is very common for high performance s= ystems. Do this both for the nics and also your apps. Be mindful of the = NUMA topooogy. You should look into both the erlang scheduler flags for core pinning, = and also ensuring that your erlang processes have unimpeded access to th= eir own cores too. A reasonable approach is to make a simple cowboy or P= hoenix app and hammer it with wrk or similar load tool to get a feel for= things, and then profile and tune your own app based on those specific = results. For rabbit there is an excellent load testing tool from the pivotal team= if you don=E2=80=99t have suitable load generators yourselves. Tsung is an excellent tool if you put in the work to craft something spe= cific for your use case. Please post back to the list with your specific findings and nic/ tcp tu= nables, these are very helpful for the next person! Dave=20