PPPoE RX traffic is limited to one queue

[Richard Pasztor]

Dear all,

continuing the discussion from here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856

summary:
(technically affects any NIC) Multi-queue NIC can use only RX queue "0", if
PPPoE session is established (due to point-to-point connection, the RSS
cannot load balance among multiple queues, at least that is what I was told
and at the moment I have no idea if that is true or not)

Result: 1 Gigabit traffic cannot be reached on some embedded SoC (in my
case PC Engines APU2, Quad-core AMD 1Ghz CPU) due to less-than-server-grade
single-core performance to handle the flow.
Note: I am not planning to achieve wire-speed 1Gbit using IMIX (smallest
packet size possible), on the contrary, I was testing using iperf3 with
max. MTU + MSS
Note2: I am not at the level of building a proper PPPoE simlator network to
properly validate the final performance, all my tests were performed using
pure IP routing. So expect PPPoE can be by definition only worse than what
I can possibly reach using pure IP.

I was suggested to set the following:
net.isr.numthreads=4
net.isr.maxthreads=4
net.isr.bindthreads=1
net.isr.dispatch=deferred

It didnt improve the situation, max. throughput was about the same, but CPU
load in interrupt handling effectively became doubled.

On the other hand, installing a small Linux firewall distrib (IPfire), I
could easily manage to get 900+ MBit of traffic with less interrupt load on
the system. So I dont know how could Linux overcome the above said
limitation.

As the original bugreport was against a possibly "igb" Intel driver issues,
and that was said not the case, the PR was closed. So no advancement is
expected in this topic, hence I opened it here, hoping there is some
solution for this issue.

Regards,
Richard