pf performance?

[Ryan Stone]

On Fri, Apr 26, 2013 at 10:49 AM, Erich Weiler <weiler at soe.ucsc.edu> wrote:

> The pf isn't a process, so you can't see it in top. pf has some helper
>> threads however, but packet processing isn't performed by any of them.
>>
>
> But the work pf does would show up in 'system' on top right?  So if I see
> all my CPUs tied up 100% in 'interrupts' and very little 'system', would it
> be a reasonable assumption to think that if I got more CPU cores to handle
> the interrupts that eventually I would see 'system' load increase as the
> interrupt load became faster to be handled?  And thus increase my bandwidth?
>
> In other words, until I see like 100% system usage in one core, I would
> have room to grow?
>

Probably not.  Mutexes in FreeBSD use "adaptive spinning".  This means that
when a thread is unable to acquire a mutex, if the owner of the mutex is
still running on another CPU core then the thread will spin and wait for
the mutex to become free rather than block.  This improves the latency of
acquiring the mutex and saves us many expensive calls through the
scheduler, but the downside is that you have one heavily contended mutex
you can have many cores wasted spinning on the lock.  In your case it is
likely that your interrupt threads are spinning on the pf lock.  You can
partially confirm this by profiling the system.  The quick way is:

kldload hwpmc
pmcstat -S unhalted-cycles -T
(if unhalted-cycles does not work, try instructions).

If _mtx_lock_sleep is the top function then you are hitting mutex
contention and more cores are unlikely to help.  In this case you might
actually be able to improve performance by *reducing* the number of threads
that are contending in pf, for example by using fewer receive queues in
your NIC.

If this machine is dedicated for pf then setting sysctl net.isr.direct=0
might also improve performance, by forcing all packets to go through a
single netisr thread (assuming that net.isr.maxthreads is 1).  Note that
this will apply to traffic that does not go through pf, so if this machine
were doing other network things (e.g. serving NFS) then it would bottleneck
your other traffic behind your pf traffic.