Re3: [CHECKER] bugs in FreeBSD

[Matthew Dillon]

    More research...  correct me if I am wrong but it appears that the 5.x
    kmem_malloc() code may have some issues.  If you look down at around line
    349 there is a comment:

        /*
         * Note: if M_NOWAIT specified alone, allocate from
         * interrupt-safe queues only (just the free list).  If
         * M_USE_RESERVE is also specified, we can also
         * allocate from the cache.  Neither of the latter two
         * flags may be specified from an interrupt since interrupts
         * are not allowed to mess with the cache queue.
         */

        if ((flags & (M_NOWAIT|M_USE_RESERVE)) == M_NOWAIT)
                pflags = VM_ALLOC_INTERRUPT | VM_ALLOC_WIRED;
        else
                pflags = VM_ALLOC_SYSTEM | VM_ALLOC_WIRED;

    Here's the problem... the problem is that malloc(...M_NOWAIT) is used
    by interrupts not only to avoid blocking, but also to avoid messing with
    the VM Page 'cache' queue.

    But in 5.x it is possible for non-interrupt threads to preempt other
    non-interrupt threads indirectly (due to an interrupt trying to get a
    mutex that a non-interrupt thread currently holds).  Am I correct?

    But the non-interrupt thread will almost certainly be making 
    memory allocations with M_WAITOK, which means that a preempting thread
    *CAN* wind up pulling pages out of the 'cache' queue.

    Now, my understanding is that 5.x's mutexes around the VM system means
    that this, in fact, will work just fine.

    So, that means that the above comment is no longer correct, right?  In
    fact, interrupts *should* be able to allocate pages from the VM page
    'cache' queue in 5.x now.

    This leads to the obvious conclusion that 'critical' code, such as the CAM
    code, which cannot afford to block but which also does terrible things
    when an M_NOWAIT allocation fails should be able to use 
    (M_WAITOK|M_USE_RESERVE|M_USE_INTERRUPT_RESERVE) and this would result
    in far safer operation then the current M_NOWAIT use results in.

    (M_USE_INTERRUPT_RESERVE would be a new M_* flag that allows the system
    to exhaust the entire free page reserve if necessary and has the same
    effect as M_NOWAIT had before, but the combination of flags would now
    allow interrupt-time allocations to also allocate from the cache queue
    making it virtually impossible for such allocations to fail and that,
    combined with M_WAITOK, would allow all NULL checks to be removed.  It
    could actually be considered a critical error for the above flags
    combination to deadlock.

						-Matt