memory allocation / deallocation within the kernel

[Robert Watson]

On Tue, 2 May 2006, Bharma Ji wrote:

> I am trying to understand the impact of memory allocation / deallocation
> within the kernel. As I understand

I can't answer all your questions, but can point at a few examples in current 
kernel code.

> a) Kernel memory is not pageable ie the one you get from using kernel 
> malloc(there may be exceptions)

In general, we guarantee that kernel memory is accessible fault-free so that 
locks can be held over kernel memory use.  In particular, both kernel malloc 
and the kernel slab allocator allocate both address space and locked memory 
that is non-pageable.  Two canonical exceptions:

- Kernel thread stacks, which may be swapped out.  See PS_INMEM.
- Pipe buffers, which use pageable kernel memory.

> b) Does this imply that if I have 1 GB of RAM - then I cannot reserve more 
> than 1 GB of kernel virtual address space? The reason is that if at any 
> point of time, the kernel has to allocate all of its virtual address space 
> i.e. if it needs to allocate more than 1 GB of address space, there won't be 
> any physical RAM memory to allocate from and thus this scenario is not 
> allowed as a configuration?

Address space and memory are separable.  The kernel memory allocators ask for 
memory from the VM system as other consumers do, so the kernel address space 
can be much larger than available memory.  On the other hand, you as long as 
you're dealing with kernel memory allocated using standard allocators, such as 
malloc and UMA, you are allocating both address space and memory, so you can't 
use more memory than address space.  Some components, such as md disk devices, 
allocate using swap-backed memory that is pageable, so interfaces exist to use 
pageable memory in kernel, and are used.  You do have to be pretty careful 
though, as if you fault in an address in kernel, you may have to wait for it 
to arrive, which involves long sleeps.  Holding kernel locks, especially 
mutexes, across faults is not a good thing, and our invariants checkers will 
generate errors when that happens.

> c) Another scenario is that assume that the kernel has 512 MB of virtual 
> address space with 1 GB of RAM. Now assume that the entire 1 GB of RAM is 
> used up by the kernel and other userland process that are running - with the 
> kernel taking 256 MB and the rest allocated to the processes. Now if the 
> kernel needs to allocate more memory, will some of the processes be swapped 
> out to make way for the kernel(since the kernel can take upto 512 MB)

If we're talking about non-pageable kernel memory, then user space will be 
operating in a memory-starved environment, and likely thrash.  If we're 
talking about pageable kernel memory, then the kernel threads and user threads 
will compete for physical memory.

Robert N M Watson