malloc+utrace, tracking memory leaks in a running program.

[Jason Evans]

On Dec 21, 2012, at 7:37 PM, Alfred Perlstein <bright at mu.org> wrote:
> So the other day in an effort to debug a memory leak I decided to take a look at malloc+utrace(2) and decided to make a tool to debug where leaks are coming from.
> 
> A few hours later I have:
> 1) a new version of utrace(2) (utrace2(2)) that uses structured data to prevent overloading of data.   (utrace2.diff)
> 2) changes to ktrace and kdump to decode the new format. (also in utrace2.diff)
> 3) changes to jemalloc to include the new format AND the function caller so it's easy to get the source of the leaks. (also in utrace2.diff)
> 4) a program that can take a pipe of kdump(1) and figure out what memory has leaked. (alloctrace.py)
> 5) simple test program (test_utrace.c)
> 
> […]

Have you looked at the heap profiling functionality built into jemalloc?  It's not currently enabled on FreeBSD, but as far as I know, the only issue keeping it from being useful is the absence of a Linux-compatible /proc/<pid>/maps (and the gperftools folks may already have a solution for that; I haven't looked).  I think it makes more sense to get that sorted out than to develop a separate trace-based leak checker.  The problem with tracing is that it doesn't scale beyond some relatively small number of allocator events.

> Is it time to start installing with some form of debug symbols? This would help us also with dtrace.

Re: debug symbols, frame pointers, etc. necessary to make userland dtrace work by default, IMO we should strongly prefer such defaults.  It's more reasonable to expect people who need every last bit of performance to remove functionality than to expect people who want to figure out what the system is doing to figure out what functionality to turn on.

Thanks,
Jason