New cpu_switch() and cpu_throw().

[Matthew Dillon]

     I haven't read your code to see what you've done exactly but I
     had a similar issue in DragonFly and I solved it by having a bit
     of code that set a bit in the previous thread's flags AFTER having
     switched to the new thread.

     e.g. the switch restore code for the new thread (%eax) is also
     responsible for cleaning up the old thread (%ebx):

        /*
         * Clear TDF_RUNNING flag in old thread only after cleaning up
         * %cr3.  The target thread is already protected by being TDF_RUNQ
         * so setting TDF_RUNNING isn't as big a deal.
         */
        andl    $~TDF_RUNNING,TD_FLAGS(%ebx)
        orl     $TDF_RUNNING,TD_FLAGS(%eax)

	(from /usr/src/sys/platform/pc32/i386/swtch.s on DragonFly)

    The exit code can then interlock on TDF_RUNNING without there being
    a race against the old thread's stack.  The condition occured so rarely
    I didn't even bother using a lock... I just test it in exit (well,
    really the reaper) and sleep for one tick in a loop if the race was
    detected.  The same feature is used for thread migration between cpu's...
    the target cpu gets an IPI message with the thread being migrated, and
    spins waiting for TDF_RUNNING to clear, indicating that the thread has
    been completely switched out by the originating cpu.

    FreeBSD uses a more complex arrangement so it might not be applicable,
    but it is still a nice trick.

						-Matt