Increasing KVM on amd64

[Alan Cox]

Artem Belevich wrote:

>Alan,
>
>Thanks a lot for the patch. I've applied it to RELENG_7 and it seems
>to work great - "make -j8 buildworld" succeeds, linux emulation seems
>to work well enough to run linux-sun-jdk14 binaries, ZFS ARC size is
>bigger, too. So far I didn't see any ZFS-related KVM shortages either.
>
>The only problem is that everything is fine as long as vm.kmem_size is
>set to less or equal to 4096M. As soon as I set it to 4100M or
>anything larger, kernel crashes on startup. I'm unable to capture
>exact crash messages as they keep scrolling really fast on the screen
>for a few seconds until the box reboots. Unfortunately  the box does
>not have built-in serial ports, so the messages are gone before I can
>see them. :-(
>
>  
>

There are two underlying causes.  First, the size of the kmem map, which 
holds the kernel's heap, is recorded in a 32-bit int.  So, setting 
vm.kmem_size to 4100M is leading to integer overflow.  The following 
change addresses this issue:

sys/kern/kern_malloc.c

Revision *1.167*: download 
<http://www.freebsd.org/cgi/cvsweb.cgi/%7Echeckout%7E/src/sys/kern/kern_malloc.c?rev=1.167;content-type=text%2Fplain> 
- view: text 
<http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?rev=1.167;content-type=text%2Fplain>, 
markup 
<http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?rev=1.167;content-type=text%2Fx-cvsweb-markup>, 
annotated 
<http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?annotate=1.167> 
- select for diffs 
<http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?r1=1.167#rev1.167>
/Sat Jul 5 19:34:33 2008 UTC/ (2 months ago) by /alc/
Branches: MAIN 
<http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?only_with_tag=MAIN>
CVS tags: HEAD 
<http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c?only_with_tag=HEAD>
Diff to: previous 1.166: preferred 
<http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c.diff?r1=1.166;r2=1.167>, 
colored 
<http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_malloc.c.diff?r1=1.166;r2=1.167;f=h>
Changes since revision 1.166: +11 -11 lines

SVN rev 180308 on 2008-07-05 19:34:33Z by alc

Enable the creation of a kmem map larger than 4GB.
Submitted by: Tz-Huan Huang

Make several variables related to kmem map auto-sizing static.
Found by: CScout


Second, there is no room for a kmem map greater than 4GB unless the 
overall KVM size is greater than 6GB.  Specifically, a >4GB kmem map 
isn't possible with 6GB KVM because the kmem map would overlap the 
kernel's code, data, and bss segment.

If you're able to apply the above kern_malloc.c change to your kernel, 
then I should be able to describe how to increase your KVM beyond 6GB.

Regards,
Alan