How thread-friendly is kevent?

[John-Mark Gurney]

J David wrote this message on Mon, Nov 10, 2014 at 22:00 -0500:
> On Mon, Nov 10, 2014 at 2:13 AM, John-Mark Gurney <jmg at funkthat.com> wrote:
> > you
> > really need to use one of _DISPATCH or _ONESHOT to ensure that the
> > event only gets delivered to a single thread....
> 
> That's what one would expect, which is why the observed behavior was
> so surprising.  After increasing the testing load considerably, it did
> behave as expected (waking more than one thread for one event).  But
> even so, the occurrences were very rare.  It would wake up at most one
> "extra" thread in slightly less than 1 out of 100,000 events.

This is odd...  I would expect that the event w/o _ONESHOT and _DISPATCH
to be delivered many times...  Is it possible you have locks in your
userland side of things that make this less likely?

> > Though if you mean how many threads will be woken up in the kernel
> > and find that there are no events remaining as one of the other kernel
> > threads has delivered the event, then yes, I have looked at the code,
> > and there will be a thundering herd problem...
> 
> Thanks for that, that's exactly the kind of information I was hoping to find.
> 
> Is that something that can happen without any usermode-visible
> effects?  I.e. all the threads wake up, but they almost all go back to
> sleep without leaving the kevent() syscall since they can see there's
> nothing to do anymore.  If so, that would match the observed behavior,
> but could add up to a lot of hidden overhead.

I have an idea that should only be a few lines of changes that would
prevent all the threads waking up...  As we lock the kq before doing
the wakeup, we can change KQ_SLEEP from a flag to a count for how many
threads are sleeping for an event, and if non-zero, do a wakeup_one...
Then when kqueue_scan is about to exit, check to see if there are
still events and threads waiting, and then do another wakeup_one...

Currently, KQ_SLEEP is only a flag, so we have to do wakeup to make
sure everyone wakes up...

Well, if you don't have _ONESHOT and _DISPATCH, any changes I make
should make it more reliable that all threads get the events dispatched
to them... :)

> > And if you do, it would make more sense to
> > use the recent RSS work that Adrian has been working on, and have one
> > kq per CPU w/ the proper cpu binding for that set of sockets...
> 
> The most recent information I was able to find:
> 
> http://adrianchadd.blogspot.com/2014/10/more-rss-udp-tests-this-time-on-dell.html
> 
> suggests that this work, while admirable and important, is quite some
> ways away from being production-stable for usermode code:
> 
> "hopefully I can get my network / rss library up and running enough to
> prototype an RSS-aware memcached and see if it'll handle this
> particular workload."
> 
> It's definitely something to keep an eye on, but probably not a viable
> approach for us right now.

True... But some of this is making sure you only run enough threads as
necessary...  As the kq lock is a single lock, having extra threads that
don't really do much work only increasing contention and other issues...

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."