no more WITNESS errors (was: stable kqueue locking up and running on SMP)

[Brian Fundakowski Feldman]

I know everyone's tired of seeing all the e-mail on the subject, so I'll 
keep it brief ;).  The kqueue fields that just happened to be stored in the
struct filedesc have always been owned by kqueue, and now they are locked
by kqueue, not filedesc, to become more semantically correct.  Just like all 
of the rest of the things (klist, knote, kqueue, fdp->fd_kn*), the lock 
right now is just a shared, global one, and should remain so unless 
profiling proves it unnecessary.

I would like to encourage more widespread testing.  The unused and 
nearly-unimplementable-EVFILT_PROC+NOTE_TRACK is gone, but the ability to 
nest kqueues remains (and will certainly impart difficulties if locking
is extended to separate the individual klists).  I don't think there are
any remaining issues for kqueue unless I've missed replacing some of the
SLIST_INSERT_HEAD()/SLIST_REMOVE() calls with KLIST_INSERT()/KLIST_REMOVE().

"Final" patch, ripe for testing/MUTEX_PROFILING, at:
<http://69.140.204.238/~green/kqueue-giant-locking.3.patch>

-- 
Brian Fundakowski Feldman                           \'[ FreeBSD ]''''''''''\
  <> green at FreeBSD.org                               \  The Power to Serve! \
 Opinions expressed are my own.                       \,,,,,,,,,,,,,,,,,,,,,,\