real vs. avail memory

[Peter Edwards]

David Schultz wrote:

>On Sun, Oct 12, 2003, Dag-Erling Smrgrav wrote:
>  
>
>>I've gotten used to the fact that there is a small discrepancy between
>>real and available memory, but I was surprised to see the following in
>>dmesg on a new P4 system:
>>
>>real memory  = 1073676288 (1023 MB)
>>avail memory = 1037799424 (989 MB)
>>
>>That's a full 40 MB difference...  where does that memory go?  is it
>>used for page maps or something like that?
>>    
>>
>
>Unless this is related to Peter's recent machdep.c changes, the
>difference is probably just random chunks of memory that the BIOS
>decided to use.  This could include a shadow copy of the BIOS, the
>BIOS data segment, maybe a frame buffer for a cheap integrated
>video card, etc.  If you do a verbose boot, you'll get a list of
>the chunks of memory that are taken according to the BIOS.
>
>  
>
This piqued my interest, so I studied it a bit. (My 512MB machine was 
showing a 24MB difference)

Most of the memory is swallowed by two functions called from vm_mem_init():

The kernel allocates a vm_page structure for every page of available 
memory early on. These are 72 bytes in size. (see vm_page_startup() in 
vm_page.c)

Later, the pmap code preallocates a pv_entry structure for each vm_page 
(I assume it can probably allocate more later, but will need at least 
this many if each page is mapped once). These are 28 bytes each. (See 
pmap_init() in pmap.c)

The kernel itself will come out of real memory too. The kernel is loaded 
into 4MB pages, and any residue in the last 4MB page appears to be lost 
at the moment.  I'm not sure if this was always the case (well, since 
PSE was introduced) or if it's just an issue since Bosko's fix for the 
weird Pentium bugs was committed. If you look at the "phys_avail" array 
of a running system, you can see how much the "hole" for the kernel 
takes up:

 > petere at hippo# gdb -k /usr/src/sys/i386/compile/HIPPO/kernel.debug 
/dev/mem
 > GNU gdb 5.2.1 (FreeBSD)
 > [snip]
 > (kgdb) p phys_avail
 > $1 = {4096, 651264, 1048576, 4194304, 12738560, 526987264, 0, 0, 0, 0}
 > (kgdb)

This array shows the spans of physical memory that were found at boot 
time. The even elements (counting from 0) are the start of the span, 
with odd elements being the end. My box shows a hole between 4194304 and 
12738559: This is where the kernel was loaded. Assuming your phys_avail 
map had a similar 8MB hole for the kernel, then the minimum estimate of 
the amount of space consumed before printing the "avail memory" message 
can be calculated as follows:

Available memory in pages = 107366288 / 4096 = 253369

Space Lost to struct "vm_page"s:    253369 * 72      = 18242568
Space Lost to struct "pv_entry"s:   253369 * 28      =  7094332
Space Lost to kernel "hole":      12738560 - 4194304 =  8544256
                                                       --------
                                                       33881156

that's within 2MB of what you're seeing. You can probably take out some 
more space for preloaded kld admin overhead, and the message buffer, and 
probably plenty more I can't think of (BIOS and WITNESS SYSINITS at 
least happen between the final print out and the VM initialisation: that 
probably accounts for a lot of the remainder)