malloc+utrace, tracking memory leaks in a running program.

[Alfred Perlstein]

On 1/10/13 1:41 AM, Alfred Perlstein wrote:
> On 12/23/12 12:28 PM, Jason Evans wrote:
>> On Dec 21, 2012, at 7:37 PM, Alfred Perlstein <bright at mu.org> wrote:
>>> So the other day in an effort to debug a memory leak I decided to 
>>> take a look at malloc+utrace(2) and decided to make a tool to debug 
>>> where leaks are coming from.
>>>
>>> A few hours later I have:
>>> 1) a new version of utrace(2) (utrace2(2)) that uses structured data 
>>> to prevent overloading of data. (utrace2.diff)
>>> 2) changes to ktrace and kdump to decode the new format. (also in 
>>> utrace2.diff)
>>> 3) changes to jemalloc to include the new format AND the function 
>>> caller so it's easy to get the source of the leaks. (also in 
>>> utrace2.diff)
>>> 4) a program that can take a pipe of kdump(1) and figure out what 
>>> memory has leaked. (alloctrace.py)
>>> 5) simple test program (test_utrace.c)
>>>
>>> […]
>> Have you looked at the heap profiling functionality built into 
>> jemalloc?  It's not currently enabled on FreeBSD, but as far as I 
>> know, the only issue keeping it from being useful is the absence of a 
>> Linux-compatible /proc/<pid>/maps (and the gperftools folks may 
>> already have a solution for that; I haven't looked).  I think it 
>> makes more sense to get that sorted out than to develop a separate 
>> trace-based leak checker.  The problem with tracing is that it 
>> doesn't scale beyond some relatively small number of allocator events.
>
> I have looked at some of this functionality (heap profiling) but alas 
> it is not implemented yet.  In addition the dtrace work appears to be 
> quite away from a workable solution with too many performance 
> penalties until some serious hacking is done.
>
> I am just not sure how to proceed, on one hand I do not really have 
> the skill to fix the /proc/pid/maps problem, nor figure out how to get 
> dtrace into the system in any time frame that is reasonable.
>
> All a few of us need is the addition of the trace back into the 
> existing utrace framework.
>
>>> Is it time to start installing with some form of debug symbols? This 
>>> would help us also with dtrace.
>> Re: debug symbols, frame pointers, etc. necessary to make userland 
>> dtrace work by default, IMO we should strongly prefer such defaults.  
>> It's more reasonable to expect people who need every last bit of 
>> performance to remove functionality than to expect people who want to 
>> figure out what the system is doing to figure out what functionality 
>> to turn on.
>>
>
> This is very true.  I'm going to continue to work towards this end 
> with a few people and get up to speed on it so that hopefully we can 
> get to this point hopefully in the next release cycle or two.
>
> If you have a few moments, can you have a look at the "utrace2" 
> branches here:
> https://github.com/alfredperlstein/freebsd/tree/utrace2
>
> This branch contains the addition of the utrace2 system call which is 
> needed to structure data via utrace(2).  The point of this is to avoid 
> kdump(1) needing to discern type of ktrace records based on arbitrary 
> size or other parameters and introduces an extensible protocol for new 
> types of utrace data.
>
> The utrace2 branch here augments jemalloc to use utrace2 to pass the 
> old utrace records, but in addition to pass the return address along 
> with the type and size of the allocation:
> https://github.com/alfredperlstein/jemalloc/tree/utrace2
>
> -Alfred

Jason,

Here are more convenient links that give diffs against FreeBSD and 
jemalloc for the proposed changes:

FreeBSD:
https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2

jemalloc:
https://github.com/alfredperlstein/jemalloc/compare/master...utrace2

-Alfred