From nobody Mon Aug 15 10:52:38 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M5rhw6y8Fz4YWGt for ; Mon, 15 Aug 2022 10:52:48 +0000 (UTC) (envelope-from SRS0=TY6jks=YT=codenetworks.net=sm@eigbox.net) Received: from bosmailout05.eigbox.net (bosmailout05.eigbox.net [66.96.189.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4M5rhv6Sdvz3Cqt for ; Mon, 15 Aug 2022 10:52:47 +0000 (UTC) (envelope-from SRS0=TY6jks=YT=codenetworks.net=sm@eigbox.net) Received: from bosmailscan01.eigbox.net ([10.20.15.1]) by bosmailout05.eigbox.net with esmtp (Exim) id 1oNXiB-0005Oh-Gm for freebsd-net@freebsd.org; Mon, 15 Aug 2022 06:52:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=codenetworks.net; s=dkim; h=Sender:In-Reply-To:Subject:From:References:To: MIME-Version:Date:Message-ID:Content-Type:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=vLQiM5YkFAzuiYe2ys7RT13eM7l/8evDuJaoUkZw2d8=; b=RSuR1zyUmg4+iBB2WWKDMFa2W k/zkIXhcCqlSh0R4QsPEkGNq+GUkybril3qL9HgnHdLxk8Zmftyh05VFgKJLcu+IFqAKeSWow1s/W 1wHMPpumb2gMBIW7A1RN1e32HbkKl+Hb/wtxD4gKXPakGE0FFp2648Gz37+Oll8jTNULQXGM3mLR3 1u9oaMs0CInqgveGnaEAczuj+IOXoTZghiSGLA9CB9BBxdIdPQYX38TAce1a19lETc0WOex4ENxaJ alCoNZB+sCRXIjj0QXg15dA3bB+zbiQVYto9bKD27cXpu3txwGgHCumv8SGlZlkSGlDiAZeB94K/z 1iR06wTZA==; Received: from [10.115.3.32] (helo=bosimpout12) by bosmailscan01.eigbox.net with esmtp (Exim) id 1oNXiA-0002fy-VV for freebsd-net@freebsd.org; Mon, 15 Aug 2022 06:52:46 -0400 Received: from bosauthsmtp05.yourhostingaccount.com ([10.20.18.5]) by bosimpout12 with id 7msj2800706Zqne01msmQb; Mon, 15 Aug 2022 06:52:46 -0400 X-Authority-Analysis: v=2.3 cv=d4VuNSrE c=1 sm=1 tr=0 a=eBvjjtMVdWwtQGedh7GyLg==:117 a=Ek/qOh1uPkKSHvd30yk7rg==:17 a=biHskzXt2R4A:10 a=-Yl_685HdVUA:10 a=r77TgQKjGQsHNAKrUKIA:9 a=jlvWEfeLAAAA:8 a=TBnFSiwzkSzUpOUq7w0A:9 a=QEXdDO2ut3YA:10 a=Y2PtTiLEAAAA:8 a=_1d3SJlyhpwWRqlN1ukA:9 a=TGAxWDhB9J0FpHmL:21 a=_W_S_7VecoQA:10 a=BUduvz6nQKmfCEOu4uBS:22 a=7OkdhSbzDfCo1JLT1PZq:22 Received: from cm-81-9-194-73.telecable.es ([81.9.194.73]:27020 helo=[192.168.3.100]) by bosauthsmtp05.eigbox.net with esmtpa (Exim) id 1oNXi7-0002eT-49; Mon, 15 Aug 2022 06:52:43 -0400 Content-Type: multipart/alternative; boundary="------------YpV97TbqJ2YsOr80kZyWwDqe" Message-ID: Date: Mon, 15 Aug 2022 12:52:38 +0200 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Thunderbird/91.12.0 Content-Language: en-US To: Benoit Chesneau , "freebsd-net@FreeBSD.org" , Michael Dexter References: <2f362689-0feb-bd41-93b2-afb46b4a4a08@codenetworks.net> From: Santiago Martinez Subject: Re: 25/100 G performance on freebsd In-Reply-To: X-EN-UserInfo: d3bdfab0736480cedf04ed92aaea2ef5:931c98230c6409dcc37fa7e93b490c27 X-EN-AuthUser: sm@codenetworks.net X-EN-OrigIP: 81.9.194.73 X-EN-OrigHost: cm-81-9-194-73.telecable.es X-Rspamd-Queue-Id: 4M5rhv6Sdvz3Cqt X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none ("invalid DKIM record") header.d=codenetworks.net header.s=dkim header.b=RSuR1zyU; dmarc=none; spf=pass (mx1.freebsd.org: domain of "SRS0=TY6jks=YT=codenetworks.net=sm@eigbox.net" designates 66.96.189.5 as permitted sender) smtp.mailfrom="SRS0=TY6jks=YT=codenetworks.net=sm@eigbox.net" X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FORGED_SENDER(0.30)[sm@codenetworks.net,SRS0=TY6jks=YT=codenetworks.net=sm@eigbox.net]; R_SPF_ALLOW(-0.20)[+ip4:66.96.128.0/18:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:29873, ipnet:66.96.128.0/18, country:US]; R_DKIM_PERMFAIL(0.00)[codenetworks.net:s=dkim]; RWL_MAILSPIKE_POSSIBLE(0.00)[66.96.189.5:from]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[66.96.189.5:from]; RCVD_TLS_LAST(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[sm@codenetworks.net,SRS0=TY6jks=YT=codenetworks.net=sm@eigbox.net]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[codenetworks.net:~]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[codenetworks.net: no valid DMARC record]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------YpV97TbqJ2YsOr80kZyWwDqe Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi Benoit, Not sure what the environment, is this to host VNF? those 2x25 will be both forwardings or are active/standby). In my case I use:    *    Vale for Inter-VM inside the same host.    *    Vale to connect to the external network ( hence a phy interface). In my case Intel 40G NICS.    *    SR-IOV for some specific use cases (for example, BNG stress test tools running on Linux). For JAILS:     *    I tend to use just VNET. Can't get more than 7.2Gbps ( >1400b) from an epair without a bridge in the middle.     *    Right now I'm doing some tests with RSS enabled, but is not looking good, actually no passing traffic... If your NICs start to play nice with SR-IOV you can pass a VF to the Jail, some NICs allow creating L2 "high speed" switches in the card ( never used one). Regarding L3 (in-kernel), the overhead will be bigger than using vale, but then you can leverage multi-path, VXLAN termination, IPFW, PF, dummynet,  etc. Hope it makes sense. Santi On 8/13/22 11:20, Benoit Chesneau wrote: > > Santiago thanks for the help. > > I am curious about your vale setup. Do you have only internal bridges? > Do you bridge the NIC interface or are doing L3? > > Afaik i am trying to dind what would be the most efficient way to use > the 25GB interfaces whle isolating the services on them.  I very > hesitant of the approach and unsure if freebsd these days can fit the > bill: > > * run isolated services over the 2x25G . would jails limit the bandwith? > * possibly run bhyve services when linux or else  is needed . > > Would using only L3 routing solve some performances issues? > > > benoit > > > On Wed, Aug 10, 2022 at 23:31, Santiago Martinez > wrote: >> Hi Benoit, sorry to hear that the SR-IOV still not working on your HW. >> >> Have you tested the last patch from Intel? >> >> Regarding Bhyve, you can use Vale switches (based on netmap). >> On my machines, i get around ~33Gbps between VM (same local machine), >> sometimes going towards 40Gbps... ( These are basic tests with iperf3 >> and TSO/LRO enabled). >> >> @Michael Dexter is working on a document that contains configuration >> examples and test results for the different network backend available >> in bhyve. >> >> If you need help, let me know and we can set up a call. >> Take care. >> Santi >> >> On 8/8/22 08:57, Benoit Chesneau wrote: >>> For some reasons. I can’t use SR-IOV on my freebsd machines (HPE >>> DL160 gen10) with latest 25G HPE branded cards. I opened tickets for >>> that but since then no move happened. >>> >>> So I wonder id there is a good setup to use these cards with the >>> virtualization. Which kind of performance should I expect using >>> if_bridge? What if i am doing L3 routing instead using epair or tap >>> (for bhyve). Would it work better? >>> >>> Any hint is welcome, >>> >>> Benoît >>> --------------YpV97TbqJ2YsOr80kZyWwDqe Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Hi Benoit,

Not sure what the environment, is this to host VNF? those 2x25 will be both forwardings or are active/standby).

In my case I use:

   *    Vale for Inter-VM inside the same host.

   *    Vale to connect to the external network ( hence a phy interface). In my case Intel 40G NICS.

   *    SR-IOV for some specific use cases (for example, BNG stress test tools running on Linux).


For JAILS:

    *    I tend to use just VNET. Can't get more than 7.2Gbps ( >1400b) from an epair without a bridge in the middle.

    *    Right now I'm doing some tests with RSS enabled, but is not looking good, actually no passing traffic...


If your NICs start to play nice with SR-IOV you can pass a VF to the Jail, some NICs allow creating L2 "high speed" switches in the card ( never used one).

Regarding L3 (in-kernel), the overhead will be bigger than using vale, but then you can leverage multi-path, VXLAN termination, IPFW, PF, dummynet,  etc.

Hope it makes sense.

Santi


On 8/13/22 11:20, Benoit Chesneau wrote:

Santiago thanks for the help.

I am curious about your vale setup. Do you have only internal bridges? Do you bridge the NIC interface or are doing L3?  

Afaik i am trying to dind what would be the most efficient way to use the 25GB interfaces whle isolating the services on them.  I very hesitant of the approach and unsure if freebsd these days can fit the bill:

* run isolated services over the 2x25G . would jails limit the bandwith? 
* possibly run bhyve services when linux or else  is needed . 

Would using only L3 routing solve some performances issues? 


benoit


On Wed, Aug 10, 2022 at 23:31, Santiago Martinez <sm@codenetworks.net> wrote:
Hi Benoit, sorry to hear that the SR-IOV still not working on your HW.

Have you tested the last patch from Intel?

Regarding Bhyve, you can use Vale switches (based on netmap).
On my machines, i get around ~33Gbps between VM (same local machine), sometimes going towards 40Gbps... ( These are basic tests with iperf3 and TSO/LRO enabled).

@Michael Dexter is working on a document that contains configuration examples and test results for the different network backend available in bhyve.

If you need help, let me know and we can set up a call.
Take care.
Santi

On 8/8/22 08:57, Benoit Chesneau wrote:
For some reasons. I can’t use SR-IOV on my freebsd machines (HPE DL160 gen10) with latest 25G HPE branded cards. I opened tickets for that but since then no move happened.

So I wonder id there is a good setup to use these cards with the virtualization. Which kind of performance should I expect using if_bridge? What if i am doing L3 routing instead using epair or tap (for bhyve). Would it work better?

Any hint is welcome,

Benoît

--------------YpV97TbqJ2YsOr80kZyWwDqe--