Memory reserves or lack thereof

[Andre Oppermann]

On 11.11.2012 22:40, Alan Cox wrote:
> On Sat, Nov 10, 2012 at 7:20 AM, Konstantin Belousov <kostikbel at gmail.com>wrote:
>> Your analysis is right, there is nothing to add or correct.
>> This is the reason to strongly prefer M_WAITOK.
>>
>
> Agreed.  Once upon time, before SMPng, M_NOWAIT was rarely used.  It was
> well understand that it should only be used by interrupt handlers.
>
> The trouble is that M_NOWAIT conflates two orthogonal things.  The obvious
> being that the allocation shouldn't sleep.  The other being how far we're
> willing to deplete the cache/free page queues.
>
> When fine-grained locking got sprinkled throughout the kernel, we all to
> often found ourselves wanting to do allocations without the possibility of
> blocking.  So, M_NOWAIT became commonplace, where it wasn't before.

Yes, we have many places where we don't want to sleep for example in
the network code.  There we simply want to be told that we've run out
of memory and handle the failure.  It's expected to happen from time
to time.  We don't need or want to dig deep or into reserves.  Packets
are expected to get lost from time to time and upper layer protocols
will handle retransmits just fine.  What we *don't* want normally is to
get blocked on a failing memory allocation.  We'd rather drop this one
and go on with the next packet to avoid the head of line blocking
problem where everything cascades to a total halt.

As a side note we don't do many, if any, true interrupt time allocations
anymore.  Usually the interrupt is just acknowledged in interrupt
context and a taskqueue or ithread is scheduled to do all the hard work.
Neither runs in interrupt context.

> This had the unintended consequence of introducing a lot of memory
> allocations in the top-half of the kernel, i.e., non-interrupt handling
> code, that were digging deep into the cache/free page queues.
>
> Also, ironically, in today's kernel an "M_NOWAIT | M_USE_RESERVE"
> allocation is less likely to succeed than an "M_NOWAIT" allocation.
> However, prior to FreeBSD 7.x, M_NOWAIT couldn't allocate a cached page; it
> could only allocate a free page.  M_USE_RESERVE said that it ok to allocate
> a cached page even though M_NOWAIT was specified.  Consequently, the system
> wouldn't dig as far into the free page queue if M_USE_RESERVE was
> specified, because it was allowed to reclaim a cached page.
>
> In conclusion, I think it's time that we change M_NOWAIT so that it doesn't
> dig any deeper into the cache/free page queues than M_WAITOK does and
> reintroduce a M_USE_RESERVE-like flag that says dig deep into the
> cache/free page queues.  The trouble is that we then need to identify all
> of those places that are implicitly depending on the current behavior of
> M_NOWAIT also digging deep into the cache/free page queues so that we can
> add an explicit M_USE_RESERVE.

I don't think many places depend on M_NOWAIT digging deep.  I'm
perfectly happy with having M_NOWAIT give up on first try.  Only
together with M_TRY_REALLY_HARD it would dig into reserves.

PS: We have a really nasty namespace collision with the mbuf flags
which use the M_* prefix as well.

-- 
Andre