[CHECKER] bugs in FreeBSD

[Matthew Dillon]

    Well, this is fun.  There are over 460 files in the 5.x source tree 
    (360 in DFly) that make calls to malloc(... M_NOWAIT), and so far 
    about 80% of the calls that I've reviewed generate inappropriate
    side effects when/if a failure occurs.  CAM is the biggest violator...
    it even has a few panic() conditionals if a malloc(... M_NOWAIT) fails.
    Not Fun!

    The only reason it works at all is because M_NOWAIT actually does appear
    to allow malloc() to block in a number of situations (such as on VM object
    and map mutexes), and M_NOWAIT triggers VM_ALLOC_INTERRUPT which allows
    kmem_malloc() to dig into the free page reserve.  So in 5.x M_NOWAIT
    allocations will actually work most of the time.. well, at least until
    something exhausts the free page reserve at just the wrong time, which
    is quite possible to do considering how much code is being allowed to
    dig into the reserve now.

    M_NOWAIT is being used pretty much as if it were M_WAITOK|M_USE_RESERVE
    most of the time, especially considering the side effect situation when
    such allocations fail.  I don't think M_WAITOK|M_USE_RESERVE would be 
    any less reliable, actually.  It looks like the whole paradigm has 
    shifted away from the original definition of M_NOWAIT to something that
    is more like a cross between M_NOWAIT, M_WAITOK, and M_USE_RESERVE.

    This creates a conundrum for me.  In DFly M_NOWAIT really means M_NOWAIT,
    so I am going to have to do something about all the improper M_NOWAIT
    use in the source base.  I'm amazed we haven't had more crashes but
    even in DFly M_NOWAIT failures due to, e.g. not being able to get the
    kernel_map lock non-blocking, do not occur all that often.

						-Matt