can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code?

[Ryan Stone]

On Thu, Feb 13, 2014 at 6:57 PM, Adrian Chadd <adrian at freebsd.org> wrote:
> sequentually:
>
> * lookup occurs on CPU A;
> * lookup succeeds on CPU A for some almost-expired entry;
> * preemption occurs, and it gets scheduled to CPU B;
>
> then simultaneously:
>
> * CPU A's flowtable purge code runs, and decides to purge entries
> including the current one;
> * the code now running on CPU B has an item from the CPU A flowtable,
> and dereferences it as it's being freed, leading to potential badness.

This kind of scenario is definitely possible.  All of the FreeBSD
kernel code that deals with lockless per-cpu data structures that I
have seen (e.g. uma) has used critical_enter()/critical_exit() to
prevent preemption, and have been careful to invalidate their
references to the per-cpu data if they have to drop the critical
section.

I don't believe that sched_pin() is good enough because I don't
believe that it handles the scenario when thread A gets a reference
and then is preempted, thread B frees the entry, and then A is
scheduled and uses the now-freed entry.  However I'm really not
familiar at all with flowtable so maybe there's something preventing
that.