cvs commit: src/sys/geom/eli g_eli.c

[Robert Watson]

On Mon, 29 Jan 2007, Pawel Jakub Dawidek wrote:

>> Why?  You're proposing yet another intrusive change to the kernel to handle 
>> yet another one-off requirement of your code.  Why not do what I suggested 
>> before with hooking the appropriate SYSINIT in your module? Or why not 
>> follow Robert's suggestion and implement a simple event mechanism so that 
>> any module can know when a CPU has come online or offline.  Heck, you 
>> probably don't even need to implement a new mechanism, just hook the 
>> existing EVENTHANLER mechanism.  That's what it's designed for!!
>
> I'm afraid Scott that your proposals are hacks. As a GEOM class I should not 
> use SYSINIT, EVENTHANDLER, etc. I shouldn't bother if CPUs are online or 
> not. All events I need to implement a GEOM class I should receive from the 
> infrastructure. Also I shouldn't be called by the infrastructure when the 
> system is not yet ready for my activity, that's why I proposed to implement 
> this functionality in the infrastructure (ie. delay GEOM tasting machanism), 
> that hack SYSINITs in every single GEOM class that need to bind to a CPU.

I guess I'm not sure I entirely agree.  I think that we lack some important 
infrastructure, which we've been talking on and off for a dev summit or two 
now, for handling the arrival and departure of CPU resources ("dynamic 
reconfiguration").  While once this wasn't really an issue on PC hardware, it 
now is, with the advent of hypervisors, virtualization, not to mention more 
multiprocessing, etc.  We have quite a few algorithms and data structures that 
assume that the set of CPUs is static, and fail quite badly (i.e., memory 
leaks, work lost, etc) if a CPU were to stop scheduling threads.  Geli is not 
alone in wanting to know what and when CPUs are available for concurrent work, 
and like other pieces of code (UMA is the piece I have particular familiarity 
with), finds our infrastructure lacking.  I'm also not entirely convinced I 
agree with you as by the same token that you might claim sysinits and event 
handlers shouldn't be used by GEOM modules, perhaps kthreads should also not 
be used :-).  Sysinits, eventhandlers, and kthreads are all ways for 
scheduling and dispatching work.

So perhaps we need to start having the conversation about CPU events more 
seriously now.  What do you think of the idea of the following: two event 
handlers, a CPU start event and a CPU stop event, which are guaranteed to run 
on each CPU as as the CPU comes online, and just before the CPU goes offline. 
Kernel subsystems could use these events to determine when CPU resources were 
arriving and departing in some serious sense (not just "busy") in order to 
initialize and tear down per-CPU data structures, rebalance workloads, start 
or stop per-CPU works, etc.  The example I have in mind here is the network 
stack, which might reasonably wish to have per-CPU netisr (worker) threads. 
When the set of CPUs changes, it would like to increase or decrease the number 
of workers -- having the same number of workers compressed down to a smaller 
number of CPUs by migration would be a disaster for performance.

How to handle the boot processor is an interesting question -- are we 
interested in configuring away the boot processor at run-time?  If not, we 
probably want to handle it as a special case via sysinit.  If all CPUs are 
equal and any may go away, then we might need to rework our notion of 
shutdown, and provide these same events for the boot CPU (which does sound 
desirable so as not to end up with lots of special casing in subsystems). 
Regardless, we are hardly the first OS to try to address these issues via a 
clean architectural solution, and my thinking is we should do a bit of 
research.  A first place to look would definitely be OpenSolaris.

Robert N M Watson
Computer Laboratory
University of Cambridge