pf performance?

[Erich Weiler]

>     In other words, until I see like 100% system usage in one core, I
>     would have room to grow?
>
>
> Probably not.  Mutexes in FreeBSD use "adaptive spinning".  This means
> that when a thread is unable to acquire a mutex, if the owner of the
> mutex is still running on another CPU core then the thread will spin and
> wait for the mutex to become free rather than block.  This improves the
> latency of acquiring the mutex and saves us many expensive calls through
> the scheduler, but the downside is that you have one heavily contended
> mutex you can have many cores wasted spinning on the lock.  In your case
> it is likely that your interrupt threads are spinning on the pf lock.
> You can partially confirm this by profiling the system.  The quick way is:

Truly fascinating!

So my cores that are sucking up CPU on interrupts may be just "spinning" 
waiting for a lock and not doing real work?  That would explain a lot.

> kldload hwpmc
> pmcstat -S unhalted-cycles -T
> (if unhalted-cycles does not work, try instructions).

We'll try this.  It doesn't look like the hwpmc module exists on this 
box but we'll bring it in and try that.

> If _mtx_lock_sleep is the top function then you are hitting mutex
> contention and more cores are unlikely to help.  In this case you might
> actually be able to improve performance by *reducing* the number of
> threads that are contending in pf, for example by using fewer receive
> queues in your NIC.
>
> If this machine is dedicated for pf then setting sysctl net.isr.direct=0
> might also improve performance, by forcing all packets to go through a
> single netisr thread (assuming that net.isr.maxthreads is 1).  Note that
> this will apply to traffic that does not go through pf, so if this
> machine were doing other network things (e.g. serving NFS) then it would
> bottleneck your other traffic behind your pf traffic.

It's only doing pf stuff so this is something we'll try.  Thanks for the 
enlightening information!!