Vector Packet Processing (VPP) portability on FreeBSD

[Kevin Bowling]

On Mon, May 17, 2021 at 10:20 AM Marko Zec <zec at fer.hr> wrote:

> On Mon, 17 May 2021 09:53:25 +0000
> Francois ten Krooden <ftk at Nanoteq.com> wrote:
>
> > On 2021/05/16 09:22, Vincenzo Maffione wrote:
> >
> > >
> > > Hi,
> > >   Yes, you are not using emulated netmap mode.
> > >
> > >   In the test setup depicted here
> > > https://github.com/ftk-ntq/vpp/wiki/VPP-throughput-using-netmap-
> > > interfaces#test-setup
> > > I think you should really try to replace VPP with the netmap
> > > "bridge" application (tools/tools/netmap/bridge.c), and see what
> > > numbers you get.
> > >
> > > You would run the application this way
> > > # bridge -i ix0 -i ix1
> > > and this will forward any traffic between ix0 and ix1 (in both
> > > directions).
> > >
> > > These numbers would give you a better idea of where to look next
> > > (e.g. VPP code improvements or system tuning such as NIC
> > > interrupts, CPU binding, etc.).
> >
> > Thank you for the suggestion.
> > I did run a test with the bridge this morning, and updated the
> > results as well. +-------------+------------------+
> > | Packet Size | Throughput (pps) |
> > +-------------+------------------+
> > |   64 bytes  |    7.197 Mpps    |
> > |  128 bytes  |    7.638 Mpps    |
> > |  512 bytes  |    2.358 Mpps    |
> > | 1280 bytes  |  964.915 kpps    |
> > | 1518 bytes  |  815.239 kpps    |
> > +-------------+------------------+
>
> I assume you're on 13.0 where netmap throughput is lower compared to
> 11.x due to migration of most drivers to iflib (apparently increased
> overhead) and different driver defaults.  On 11.x I could move 10G line
> rate from one ix to another at low CPU freqs, where on 13.x the CPU
> must be set to max speed, and still can't do 14.88 Mpps.
>

I believe this issue is in the combined txrx interrupt filter.  It is
causing a bunch of unnecessary tx re-arms.


> #1 thing which changed: default # of packets per ring dropped down from
> 2048 (11.x) to 1024 (13.x).  Try changing this in /boot/loader.conf:
>
> dev.ixl.0.iflib.override_nrxds=2048
> dev.ixl.0.iflib.override_ntxds=2048
> dev.ixl.1.iflib.override_nrxds=2048
> dev.ixl.1.iflib.override_ntxds=2048
> etc.
>
> For me this increases the throughput of
> bridge -i netmap:ixl0 -i netmap:ixl1
> from 9.3 Mpps to 11.4 Mpps
>
> #2: default interrupt moderation delays seem to be too long.  Combined
> with increasing the ring sizes, reducing dev.ixl.0.rx_itr from 62
> (default) to 40 increases the throughput further from 11.4 to 14.5 Mpps
>
> Hope this helps,
>
> Marko
>
>
> > Besides for the 64-byte and 128-byte packets the other sizes where
> > matching the maximum rates possible on 10Gbps. This was when the
> > bridge application was running on a single core, and the cpu core was
> > maxing out at a 100%.
> >
> > I think there might be a bit of system tuning needed, but I suspect
> > most of the improvement would be needed in VPP.
> >
> > Regards
> > Francois
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>