How thread-friendly is kevent?
jmg at funkthat.com
Wed Nov 12 08:49:11 UTC 2014
J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500:
> On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <jmg at funkthat.com> wrote:
> > you
> > really need to use one of _DISPATCH or _ONESHOT to ensure that the
> > event only gets delivered to a single thread....
> That's what one would expect, which is why the observed behavior was
> so surprising. After increasing the testing load considerably, it did
> behave as expected (waking more than one thread for one event). But
> even so, the occurrences were very rare. It would wake up at most one
> "extra" thread in slightly less than 1 out of 100,000 events.
This is odd... I would expect that the event w/o _ONESHOT and _DISPATCH
to be delivered many times... Is it possible you have locks in your
userland side of things that make this less likely?
> > Though if you mean how many threads will be woken up in the kernel
> > and find that there are no events remaining as one of the other kernel
> > threads has delivered the event, then yes, I have looked at the code,
> > and there will be a thundering herd problem...
> Thanks for that, that's exactly the kind of information I was hoping to find.
> Is that something that can happen without any usermode-visible
> effects? I.e. all the threads wake up, but they almost all go back to
> sleep without leaving the kevent() syscall since they can see there's
> nothing to do anymore. If so, that would match the observed behavior,
> but could add up to a lot of hidden overhead.
I have an idea that should only be a few lines of changes that would
prevent all the threads waking up... As we lock the kq before doing
the wakeup, we can change KQ_SLEEP from a flag to a count for how many
threads are sleeping for an event, and if non-zero, do a wakeup_one...
Then when kqueue_scan is about to exit, check to see if there are
still events and threads waiting, and then do another wakeup_one...
Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make
sure everyone wakes up...
Well, if you don't have _ONESHOT and _DISPATCH, any changes I make
should make it more reliable that all threads get the events dispatched
to them... :)
> > And if you do, it would make more sense to
> > use the recent RSS work that Adrian has been working on, and have one
> > kq per CPU w/ the proper cpu binding for that set of sockets...
> The most recent information I was able to find:
> suggests that this work, while admirable and important, is quite some
> ways away from being production-stable for usermode code:
> "hopefully I can get my network / rss library up and running enough to
> prototype an RSS-aware memcached and see if it'll handle this
> particular workload."
> It's definitely something to keep an eye on, but probably not a viable
> approach for us right now.
True... But some of this is making sure you only run enough threads as
necessary... As the kq lock is a single lock, having extra threads that
don't really do much work only increasing contention and other issues...
John-Mark Gurney Voice: +1 415 225 5579
"All that I will do, has been done, All that I have, has not."
More information about the freebsd-questions