flowtable, collisions, locking and CPU affinity

[Adrian Chadd]

On 7 February 2014 17:05, Ryan Stone <rysto32 at gmail.com> wrote:
> On Fri, Feb 7, 2014 at 7:12 PM, Adrian Chadd <adrian at freebsd.org> wrote:
>> In any case - the reason it's happening above is because there's no
>> actual lock held over the whole lookup/insert path. It's a per-CPU
>> critical enter/exit path, so the only way to guarantee consistency is
>> to use sched_pin() for the entirety of the function.
>
> sched_pin seems like a very heavy hammer for what has to be a very
> rare and mostly harmless race.  Why not redo the lookup after you have
> reacquired the lock, and if you don't have to do the insert anymore
> then don't and move on?

You say "rare and harmless race"; I can trick the system to do this:

flowtable for IPv4:
        57448 lookups
        39543 hits
        17905 misses
        17820 collisions
        382 free checks
        82 frees

.. and it gets stuck in a loop of never quite correctly updating/using
the flowtable entries.

So, it is mostly harmless, except exactly when it bites you in the ass.

Yes, we could just reacquire the lock and insert if required, but I
still have to be absolutely sure that the thread isn't preempted and
migrated to another core. Otherwie we'd have teh same issue again.

I'll keep tickling it.

Thanks,


-a