Advice on a multithreaded netisr patch?

[Barney Cordoba]

--- On Wed, 4/8/09, Robert Watson <rwatson at FreeBSD.org> wrote:

> From: Robert Watson <rwatson at FreeBSD.org>
> Subject: Re: Advice on a multithreaded netisr  patch?
> To: "Barney Cordoba" <barney_cordoba at yahoo.com>
> Cc: "Ivan Voras" <ivoras at freebsd.org>, freebsd-net at freebsd.org
> Date: Wednesday, April 8, 2009, 9:16 AM
> On Wed, 8 Apr 2009, Barney Cordoba wrote:
> 
> > Is there any work being done on lighter weight locks
> for queues? It seems ridiculous to avoid using queues
> because of lock contention when the locks are only
> protecting a couple lines of code.
> 
> My reading is that there are two, closely related, things
> going on: the first is lock contention, and the second is
> cache line contention.  We have a primitive in 8.x
> (don't think it's been MFC'd yet) for a lockless
> atomic buffer primitive for use in drivers and other parts
> of the stack.  However, that addresses only lock contention,
> not line contention, which at a high PPS will be an issue as
> well.  Only by moving to independent data structures (i.e.,
> on independent cache lines) can we reduce line contention.
> 
> Robert N M Watson
> Computer Laboratory
> University of Cambridge

Are mutexes smart enough to know to yield to higher priority threads
that are waiting immediately? Such as

mtx_unlock()
{
   do_unlock_stuff();
   if (higher_pri_waiting)
      sched_yield()
}

Also is there a way from the structure or flags to determing is some
other thread is waiting on the lock, such as?

mtx_unlock(&mtx);
if (mtx.someone_is_waiting)
  sched_yield();

or better yet

if (higher_priority_is_waiting)
  sched_yield()

I don't quite have a handle on how the turnstile works, but it seems
that there is a lot of time waiting for very short-lived locks. If 
the tasks are on different cpus, what is the granularity of the wait
time for a lock that is cleared almost immediately after trying it?

Also, is the waiting only extended when the threads are running on the
same cpu?

Barney