Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1

[Maxim Sobolev]

OK, so following Luigi's suggestion we've re-enabled AIM, set
max_interrupt_rate to 8000 (matching igb), and reduced number of queues to
6. We'll have next peak in about 14 hours, I'll try to capture and record
history of the per-queue interrupt rate. It still remains somewhat puzzling
why something that worked with old-gen hardware/driver almost out of the
box now needs some extra tuning. Thanks everyone for useful suggestions in
any case!

On Tue, Aug 11, 2015 at 4:04 PM, Luigi Rizzo <rizzo at iet.unipi.it> wrote:

> On Wed, Aug 12, 2015 at 12:46 AM, Maxim Sobolev <sobomax at sippysoft.com>
> wrote:
> > Hi Luigi, thanks for the input. Are you sure about the number of queues?
> > Keep in mind we are not using netmap or anything as advanced, so we need
> lot
> > of parallel pipes just to push packet through the whole UDP/IP kernel
> stack.
> > To make things worse, our app cannot use connected sockets at the
> moment, so
> > we are having worst case scenario wrt kernel overhead. Single thread
> sender
> > on that hardware is limited to only 225K PPS until it hits CPU limit. We
> use
> > 8 queues on other older servers with I350 NICs. My send-only testing
> shows
> > that the hardware peaks at about 5 concurrent threads for connected
> sockets
> > and 7 concurrent threads for unconnected. This is with default 8 hardware
> > queues. The test is somewhat crippled since it only tests outgoing
> traffic.
> >
>
> thing is, you are using 8 queues per NIC so the total number of different
> interrupts is huge and will likely eat all of your cores.
> This is why i'd try to start low and ramp up later.
>
> > http://bit.ly/1MiLFHg
> >
> > With regards to the moderation, what effect it has on the traffic? Is NIC
> > going to drop packets once it starts moderating interrupts?
>
> moderation only means you get larger batches on each interrupt,
> which overall makes the system more efficient. The NIC will only drop
> packets when the CPU is not fast enough to drain them,.
>
> (this said, it might be a good idea to try netmap, passing packets
> that you do not know how to handle to the host stack, and instead
> intercepting those that you can deal with in netmap. those
> unconnected udp sockets are the ideal case for netmap.)
>
> cheers
> luigi
>