Darwin SMP (was: Re: ... blah blah blah ...)

[Robert Watson]

On Sun, 19 Sep 2004, Julian Elischer wrote:

> >> Darwin does not 
> >>have this problem whatsoever.
> 
> Dawin was designed from the beginning for SMP. Mach was SMP capable from
> the firt release I ever worked on which was 2.0. 

Actually, Darwin has substantial SMP "problems" in the sense that there is
pretty limited parallelism in the current released version at higher
levels in the kernel.  Lower levels make use of the largely SMP-safe Mach
components (scheduler, virtual memory, memory management, Mach IPC, etc). 
At the BSD layer of the kernel, there are basically two Giant locks
(funnels) associated respectively with "the network stack" and "the rest". 
I.e., Darwin basically runs with pretty similar SMP properties to 5.1 and
5.2 of FreeBSD: low level stuff Giant-free, and a lot of the higher level
stuff (which matters a lot) requiring the logical equivilent of Giant. 
The one difference really lies in their extensive use of Mach IPC, which
was MPSAFE (our pipes, etc were MPSAFE but they're not used in the same
way).

My understanding is that Apple is working hard to move towards a more
fine-grained locking model in Tiger (due mid-next-year) -- some of their
marketing literature suggests that it's based on the FreeBSD 5.x SMP, but
my understanding is that a fair amount of it was developed simultaneous to
our SMPng work, so I'm not sure how much explicit code reuse there will
be.  The low-level Mach synchronization primitives (wait queues, 
mutexes, etc) are pretty similar to the ones we have in SMPng
(not a cooincidence), but they've lacked higher level primitives 
such as condition variables and more general purpose
reader-writer locks.

Obviously, I'd welcome their taking any and all code they want to from
FreeBSD :-).  I welcome a move to more explicit synchronization and
locking in Darwin, as it will help to identify and close a substantial
volume of race conditions that currently exist.  In particular, I've seen
a number of races between the two compartments of the BSD parts of their
kernel where it's not clear whether structure fields are protected by the
kernel or network funnel.  Since we use Darwin extensively at work, we've
run into this a fair amount.  We ported FreeBSD SMPng condition variables
to Darwin as part of our port of the MAC Framework to that platform in
part to get to a more mature synchronization model on Darwin.  We looked
at, but didn't implement, a port of WITNESS to Darwin (where it is sorely
needed, since the lock orders in Mach are basically undocumented). 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Principal Research Scientist, McAfee Research