How thread-friendly is kevent?

[J David]

On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <jmg at funkthat.com> wrote:
> you
> really need to use one of _DISPATCH or _ONESHOT to ensure that the
> event only gets delivered to a single thread....

That's what one would expect, which is why the observed behavior was
so surprising.  After increasing the testing load considerably, it did
behave as expected (waking more than one thread for one event).  But
even so, the occurrences were very rare.  It would wake up at most one
"extra" thread in slightly less than 1 out of 100,000 events.

> Though if you mean how many threads will be woken up in the kernel
> and find that there are no events remaining as one of the other kernel
> threads has delivered the event, then yes, I have looked at the code,
> and there will be a thundering herd problem...

Thanks for that, that's exactly the kind of information I was hoping to find.

Is that something that can happen without any usermode-visible
effects?  I.e. all the threads wake up, but they almost all go back to
sleep without leaving the kevent() syscall since they can see there's
nothing to do anymore.  If so, that would match the observed behavior,
but could add up to a lot of hidden overhead.

> And if you do, it would make more sense to
> use the recent RSS work that Adrian has been working on, and have one
> kq per CPU w/ the proper cpu binding for that set of sockets...

The most recent information I was able to find:

http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html

suggests that this work, while admirable and important, is quite some
ways away from being production-stable for usermode code:

"hopefully I can get my network / rss library up and running enough to
prototype an RSS-aware memcached and see if it'll handle this
particular workload."

It's definitely something to keep an eye on, but probably not a viable
approach for us right now.

Thanks!