Tracking down what has inact pages locked up

[Karl Denninger]

On 3/18/2014 4:30 PM, John Baldwin wrote:
> On Tuesday, March 18, 2014 3:36:04 pm Karl Denninger wrote:
>> On 3/18/2014 2:05 PM, John Baldwin wrote:
>>> On Sunday, March 16, 2014 4:36:06 pm Karl Denninger wrote:
>>>> Is there a reasonable way to determine who or what has that memory
>>>> locked up -- and thus why the vm system is not demoting that space into
>>>> the cache bucket so it can be freed (which, if my understanding is
>>>> correct, should be happening long before now!)
>>> I have a hackish thing (for 8.x, might work on 10.x) to let you figure out
>>> what is using up RAM.  This should perhaps go into the base system at some
>>> point.
>>>
>>> Grab the bits at http://people.freebsd.org/~jhb/vm_objects/
>>>
>>> You will want to build the kld first and use 'make load' to load it.  It adds
>>> a new sysctl that dumps info about all the VM objects in the system.  You can
>>> then build the 'vm_objects' tool and run it.  It can take a while to run if
>>> you have NFS mounts, so I typically save its output to a file first and then
>>> use sort on the results.  sort -n will show you the largest consumer of RAM,
>>> sort -n -k 3 will show you the largest consumer of inactive pages.  Note
>>> that 'df' and 'ph' objects are anonymous, and that filename paths aren't
>>> always reliable, but this can still be useful.
>>>
>> Thanks.
>>
>> I suspect the cause of the huge inact consumption is a RAM leak in the
>> NAT code in IPFW.  It was not occurring in 9.2-STABLE, but is on
>> 10.0-STABLE, and reverting to natd in userland stops it -- which
>> pretty-well isolates where it's coming from.
> Memory for in-kernel NAT should be wired pages, not inactive.
Yeah, should be. :-)

But..... it managed to lock up 19GB of the 24GB the system has in inact 
pages over 12 hours, and dropping the system to single user and 
unloading the modules did not release the RAM...... which is why the 
question (on how to track down what the hell is going on.)

Changing the config back to natd as opposed to in-kernel NAT, however, 
made the problem disappear.

-- 
-- Karl
karl at denninger.net


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2711 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20140318/b3eddfb2/attachment.bin>