kernel address space and auto-sizing

[Nathan Whitehorn]

This is bringing back bad memories. Let me explain about
VM_MAX_SAFE_KERNEL_ADDRESS and mmu_oea64.c briefly.

mmu_oea64 is the PMAP code for both PPC32 and PPC64 running on 64-bit
CPUs. Both AIM32 and AIM64 in general have a direct map. *However*,
32-bit kernels running on 64-bit CPUs do *not* have a direct map.

The memory layout when the kernel is booted is has the low 4 GB
organized into 16 256 MB segments. We keep all firmware mappings and try
to fit in a direct map. Segments 12 and 13 (0xc, 0xd) are guaranteed to
be free. OF allocates its own mappings at the end of segment 14 (0xe).
Everything else can't be touched. Without a direct map, having only 512
MB of KVA causes the kernel to regularly run out of VA space -- this was
crashing package building. To ameliorate that, I added the
MAX_SAFE_ADDRESS stuff and it then expands the KVA space from there into
segment 14 until it finds memory the firmware is actually using and ends
KVA there. I suspect this patch will cause a return to the crashing. Did
I miss something?
-Nathan

On 02/24/13 18:08, Alan Cox wrote:
> After the fallout from the recent auto-sizing changes, I've been
> reviewing the kernel address space configuration on all of our supported
> architectures.  This review turned up a couple of odd things on the AIM
> platform:
> 
> 1. Everywhere else, except for 32- and 64-bit AIM, VM_MAX_KERNEL_ADDRESS
> denotes the end of the kernel map.  Instead, 32- and 64-bit AIM define
> VM_MAX_SAFE_KERNEL_ADDRESS and uses this to initialize the end of the
> kernel map.  On occasion, we use VM_MAX_KERNEL_ADDRESS to implement
> auto-sizing based on the size of the kernel map, so this difference
> interferes with that.
> 
> 2. The size of the kmem submap on 32-bit AIM is hardwired to only 12
> MB!  ARM had the same issue.  There, the consequences were much worse
> because ARM, unlike 32-bit AIM, doesn't have a direct map, so
> uma_small_alloc() can't be used on arm.  However, even with
> uma_small_alloc(), this 12 MB cap on the kmem submap still unnecessarily
> limits multipage allocations by the kernel.  Even with a direct map,
> those come from the kmem submap.
> 
> The attached patch does the following things to AIM:
> 
> 1. On 32-bit AIM, it redefines VM_MAX_KERNEL_ADDRESS to be the address
> that should be the end of the kernel map and eliminates
> VM_MAX_SAFE_KERNEL_ADDRESS.  On 64-bit AIM, VM_MAX_KERNEL_ADDRESS and
> VM_MAX_SAFE_KERNEL_ADDRESS were the same, so it simply eliminates
> VM_MAX_SAFE_KERNEL_ADDRESS.
> 
> 2. It enables auto-sizing of the kmem submap on 32-bit AIM, but places a
> cap on its maximum size so that it will not consume the entire kernel
> map.  (64-bit AIM already has auto-sizing and a cap for the kmem submap.)
> 
> I would appreciate review and/or testing of this patch.  In particular,
> I'd like to see the output from two sysctl's before and after applying
> the patch: vm.max_kernel_address, vm.kmem_size, and vm.kmem_size_max
> 
> Thanks,
> Alan
> 
> P.S. On 64-bit AIM, the following bit of code in the pmap bootstrap
> function was already a NOP because VM_MAX_KERNEL_ADDRESS and
> VM_MAX_SAFE_KERNEL_ADDRESS were the same.  I didn't touch it, but I
> thought that I would mention it.
> 
>         #ifndef __powerpc64__   /* KVA is in high memory on PPC64 */
>         PMAP_LOCK(kernel_pmap);
>         while (virtual_end < VM_MAX_KERNEL_ADDRESS &&
>             moea64_pvo_find_va(kernel_pmap, virtual_end+1) == NULL)
>                 virtual_end += PAGE_SIZE;
>         PMAP_UNLOCK(kernel_pmap);
>         #endif
> 
>