Thinking about kqueue's and pthread_cond_wait

[John Baldwin]

On Wednesday 10 February 2010 12:46:46 pm Daniel Eischen wrote:
> On Wed, 10 Feb 2010, Randall Stewart wrote:
> 
> >
> > On Feb 10, 2010, at 9:04 AM, Daniel Eischen wrote:
> >
> >> On Wed, 10 Feb 2010, Randall Stewart wrote:
> >> 
> >>> All:
> >>> 
> >>> I have once again come around to thinking about joining pthread cond 
waits 
> >>> and
> >>> kqueue's.
> >>> 
> >>> After thinking about it, I think its doable.. with something like a:
> >>> 
> >>> pthread_cond_wait_kqueue_np(kev, cond, mtx, ucontext)
> >>> 
> >>> Then you can use kev inside a kqueue i.e.
> >>> ret =  kevent(kq, kev, 1, outkev, 1, NULL);
> >>> 
> >>> Now when you saw the event:
> >>> if (kev.filter == EVFILT_UMTX){   /* not sure about the name here */
> >>>     pthread_kqueue_cond_wait_ret_np(kev, cond, mtx, ucontext)
> >>>     do_user_action(cond,mtx, ucontext);
> >>> }
> >>> 
> >>> Which would fill in the cond/mtx and ucontext for the user.
> >>> 
> >>> Now does this sound useful to anyone.. i.e. should I spend the time
> >>> making it work?
> >>> 
> >>> The only down side to this is that it would have to allocate memory so
> >>> one would need to do a:
> >>> 
> >>> pthread_kqueue_cond_wait_free_np(kev)
> >>> 
> >>> After you were done.. and I think it would be best for this to
> >>> be a ONE_SHOT.. i.e. you have to re-arm it if the event happens...
> >>> Of course until you free it that can be as simple as passing the kev
> >>> back down again (i.e. no pthread_cond_wait_kqueue_np() needed).
> >>> 
> >>> Comments? Thoughts?  i.e. especially is it worthwhile doing?
> >> 
> >> Please don't mess with the pthread_ API like that :-)  If you
> >> really want to munge them together, see my email to you a few
> >> weeks ago last time you brought it up.'
> >
> > If I remember right your email was basically don't do it... I will
> > go dig through the archives and re-read it all.
> 
> No, it was to add an interface or two to the kqueue/kevent API, not
> to modify the pthread_ API (which shouldn't know anything about
> kqueues).
> 
> I really think the OS is already given us the tools we need to
> do the job simply enough.  You can easily use a pipe, socketpair,
> or EVFILT_SIGNAL to wakeup a thread stuck in kevent().  You can
> additionally use a mutex to protect data shared between thread
> waiting in kevent() and other threads.
> 
> I don't see what problem this is trying to solve and I think
> whatever solution you come up with involving mutexes/CVs is
> not going to be any simpler and may even be more complex and
> messy.  Mutexes and CVs are userland library thingies, not
> kernel entities.  Yes, the umtx is a kernel entity, but it
> alone does not give you mutexes and CVs.  So when you want
> to mix kqueues and mutexes/CVs, you are involving another
> userland library and this is what makes it messy.

He wants something like 'WaitForMultipleObjects()' in NT.  The fact that these 
are userland primitives is _precisely_ why I think it belongs in the pthread 
library.  What I think is that you want a pthread primitive that is a wrapper 
around a kqueue.  Something like:

/* A lot like a 'struct event' */
struct pthread_event {
};

/* Is actually an int to hold a kqueue fd. */
pthread_event_queue_t;

/* Calls kqueue() to create the event. */
pthread_event_queue_init();

/* Similar to the kevent() call, but works with 'struct pthread_event' arrays 
instead of 'struct event' arrays.  Internally, it maps 'struct pthread_event' 
objects to 'struct event' objects to pass to an internal call to kevent().  
This code can take care of mapping requests to wait for a cv or mutex to the 
appropriate EVFILT_UMTX w/o exposing the EVFILT_UMTX implementation details to 
userland. */
pthread_event_queue_wait();

The user code model would look something like this:

	pthread_event_queue_t queue;
	struct pthread_event event;
	pthread_cond_t cv /* some condition variable */
	int fd;   /* some socket */

	pthread_event_queue_init(&queue);

	/* This is like EV_SET, maybe you would have a EVQ_SET? */
	event.filter = READ;
	event.fd = fd;
	event.flags = EV_ADD;

	/* Register socket. */
	pthread_event_queue_wait(queue, NULL, 0, &event, 1, NULL);

	event.filter = CONDVAR;
	event.cv = cv;
	event.flags = EV_ADD;

	/* Register condvar. */
	pthread_event_queue_wait(queue, NULL, 0, &event, 1, NULL);

	/* Wait for something to happen. */
	for (;;) {
		pthread_event_queue_wait(queue, &event, 1, NULL, 0, NULL);
		switch (event.filter) {
		case READ:
			/* socket is ready to read */
		case CV:
			/* cv is signalled */
		}
	}

To be honest, I find semaphores and mutexes more compelling than condvars for 
this sort of thing (you would do a try-lock to acquire the mutex or semaphore 
perhaps when it is signalled).  If you supported thread objects you could do a 
pthread_join() after a thread exits.

-- 
John Baldwin