From nobody Tue May 25 06:16:48 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AFB999EAF1B for ; Tue, 25 May 2021 06:17:01 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-yb1-xb2c.google.com (mail-yb1-xb2c.google.com [IPv6:2607:f8b0:4864:20::b2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fq3l06W24z3hYj for ; Tue, 25 May 2021 06:17:00 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by mail-yb1-xb2c.google.com with SMTP id o18so3767826ybc.8 for ; Mon, 24 May 2021 23:17:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=VP2JHCEKB9G1BaPxFVZsWryjrZubP7cisBmLwZxPb7Q=; b=iNNY0M4MnNiopQSwBanvZEctY0mZcnqPDlPGQJwBi/D+ebpxFa9NMqj87W9HSQsjN7 i9PGws7hlPOz3iAkC4BWaf2h5EPFLVZ9dc2mkhnVjJIQ1DXSa9d4KGPN+nkpm95DbSeU kdbtmLal6O1fF9yZ5+K/PBJ3d763cyYFkTRB0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=VP2JHCEKB9G1BaPxFVZsWryjrZubP7cisBmLwZxPb7Q=; b=ZPIACxviMSADU7cbfUNRQibZGnmVIC4Gbb+Ms2HseHiNCro7qxFr1MGdHMojfqw0He FnooWKPF7oaoZx647qUQ29e1XZ34pSAXIeW+LLRDgv+ClKlxtYMKHyf6cnHy5666CSdX SoTGt3Mz49aR9W7hMMxSls1VfxJ1WVHPUIQ30kQyz8K3nEJ1RmkS0xJsWMuU6HqrApWD HB3MIh4eLsqLgzPxZ2YpVA39UjMv1+msYBid3txnpJgcuJLtB46msj4S8TEko3z+3BfB xpIBp4xeXzjAQiKQ05gZEAScYVC3IPYph7LLAr2BENokWfELjAIf//9ykRcMtLXdezxQ uoVA== X-Gm-Message-State: AOAM530tn8zIxaT1AQC1ce2GjZd/7BKnE0WA2JZ2WtO8WdeX5um9Pr56 96gaoPkTT1WHYL9rSS6WCV50EiUlmzqk+dENzfp0YQ== X-Google-Smtp-Source: ABdhPJwqdox8es5QKPUmdpJvfq2LSCbRIP3FmVWEcz3WHPNb5X2RNJJmd8fXNCBshmhLYXoHveU0BggyT8R9wRjau3M= X-Received: by 2002:a25:44d6:: with SMTP id r205mr39564485yba.455.1621923419821; Mon, 24 May 2021 23:16:59 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 References: <91e21d18a4214af4898dd09f11144493@EX16-05.ad.unipi.it> <20210517192054.0907beea@x23> In-Reply-To: From: Kevin Bowling Date: Mon, 24 May 2021 23:16:48 -0700 Message-ID: Subject: Re: Vector Packet Processing (VPP) portability on FreeBSD To: Vincenzo Maffione Cc: Marko Zec , Francois ten Krooden , Jacques Fourie , "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4Fq3l06W24z3hYj X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none (invalid DKIM record) header.d=kev009.com header.s=google header.b=iNNY0M4M; dmarc=none; spf=pass (mx1.freebsd.org: domain of kevin.bowling@kev009.com designates 2607:f8b0:4864:20::b2c as permitted sender) smtp.mailfrom=kevin.bowling@kev009.com X-Spamd-Result: default: False [2.05 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; FREEMAIL_CC(0.00)[fer.hr,nanoteq.com,gmail.com,freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCPT_COUNT_FIVE(0.00)[5]; DKIM_TRACE(0.00)[kev009.com:~]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::b2c:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.85)[0.847]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; DMARC_NA(0.00)[kev009.com]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::b2c:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b2c:from]; R_DKIM_PERMFAIL(0.00)[kev009.com:s=google]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net]; SUSPICIOUS_RECIPS(1.50)[] I don't fully understand the issue, but in iflib_fast_intr_rxtx https://cgit.freebsd.org/src/tree/sys/net/iflib.c#n1581 it seems like we end up re-enabling interrupts per course instead of only handling spurious cases or some low water threshold (which seems like it would be tricky to do here). The idea is we want to pump interrupts by disabling them in the msix_que handler, and then wait to re-enable only when we have more work to do in the ift_task grouptask. It was a lot easier to reason about this with separate TX and RX interrupts. Doing the combined TXRX is definitely a win in terms of reducing msi-x vector usage (which is important in a lot of FreeBSD use cases), but it's tricky to understand. My time has been sucked away due to work, so I haven't been looking at this problem to the depth I want to. I'd be interested in discussing it further with anyone that is interested in it. Regards, Kevin On Tue, May 18, 2021 at 2:11 PM Vincenzo Maffione w= rote: > > > > Il giorno mar 18 mag 2021 alle ore 09:32 Kevin Bowling ha scritto: >> >> >> >> On Mon, May 17, 2021 at 10:20 AM Marko Zec wrote: >>> >>> On Mon, 17 May 2021 09:53:25 +0000 >>> Francois ten Krooden wrote: >>> >>> > On 2021/05/16 09:22, Vincenzo Maffione wrote: >>> > >>> > > >>> > > Hi, >>> > > Yes, you are not using emulated netmap mode. >>> > > >>> > > In the test setup depicted here >>> > > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap- >>> > > interfaces#test-setup >>> > > I think you should really try to replace VPP with the netmap >>> > > "bridge" application (tools/tools/netmap/bridge.c), and see what >>> > > numbers you get. >>> > > >>> > > You would run the application this way >>> > > # bridge -i ix0 -i ix1 >>> > > and this will forward any traffic between ix0 and ix1 (in both >>> > > directions). >>> > > >>> > > These numbers would give you a better idea of where to look next >>> > > (e.g. VPP code improvements or system tuning such as NIC >>> > > interrupts, CPU binding, etc.). >>> > >>> > Thank you for the suggestion. >>> > I did run a test with the bridge this morning, and updated the >>> > results as well. +-------------+------------------+ >>> > | Packet Size | Throughput (pps) | >>> > +-------------+------------------+ >>> > | 64 bytes | 7.197 Mpps | >>> > | 128 bytes | 7.638 Mpps | >>> > | 512 bytes | 2.358 Mpps | >>> > | 1280 bytes | 964.915 kpps | >>> > | 1518 bytes | 815.239 kpps | >>> > +-------------+------------------+ >>> >>> I assume you're on 13.0 where netmap throughput is lower compared to >>> 11.x due to migration of most drivers to iflib (apparently increased >>> overhead) and different driver defaults. On 11.x I could move 10G line >>> rate from one ix to another at low CPU freqs, where on 13.x the CPU >>> must be set to max speed, and still can't do 14.88 Mpps. >> >> >> I believe this issue is in the combined txrx interrupt filter. It is ca= using a bunch of unnecessary tx re-arms. > > > Could you please elaborate on that? > > TX completion is indeed the one thing that changed considerably with the = porting to iflib. And this could be a major contributor to the performance = drop. > My understanding is that TX interrupts are not really used anymore on mul= ti-gigabit NICs such as ix or ixl. Instead, "softirqs" are used, meaning th= at a timer is used to perform TX completion. I don't know what the motivati= ons were for this design decision. > I had to decrease the timer period to 90us to ensure timely completion (s= ee https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D248652). However, th= e timer period is currently not adaptive. > > >> >> >>> >>> #1 thing which changed: default # of packets per ring dropped down from >>> 2048 (11.x) to 1024 (13.x). Try changing this in /boot/loader.conf: >>> >>> dev.ixl.0.iflib.override_nrxds=3D2048 >>> dev.ixl.0.iflib.override_ntxds=3D2048 >>> dev.ixl.1.iflib.override_nrxds=3D2048 >>> dev.ixl.1.iflib.override_ntxds=3D2048 >>> etc. >>> >>> For me this increases the throughput of >>> bridge -i netmap:ixl0 -i netmap:ixl1 >>> from 9.3 Mpps to 11.4 Mpps >>> >>> #2: default interrupt moderation delays seem to be too long. Combined >>> with increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62 >>> (default) to 40 increases the throughput further from 11.4 to 14.5 Mpps >>> >>> Hope this helps, >>> >>> Marko >>> >>> >>> > Besides for the 64-byte and 128-byte packets the other sizes where >>> > matching the maximum rates possible on 10Gbps. This was when the >>> > bridge application was running on a single core, and the cpu core was >>> > maxing out at a 100%. >>> > >>> > I think there might be a bit of system tuning needed, but I suspect >>> > most of the improvement would be needed in VPP. >>> > >>> > Regards >>> > Francois >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"