Mem leak : malloc/free + pthreads = leakage?

[Eric Anderson]

On Mar 5, 2011, at 10:44 AM, Deomid Ryabkov wrote:

> On 03/05/2011 04:02 AM, Eric Anderson wrote:
>> Hi all,
>> 
>> I have a moderately threaded userland program (all C) I am working on (using pthreads, freebsd 8.1 64bit).  It seems to leak memory (using standard malloc/free) badly.
> as opposed to what? OpenBSD? Linux? Windows? why do you think your problem is specific to FreeBSD (as evidenced by your post to a FreeBSD-specific list) or is related to threaded programs?

OpenSolaris and Mac OS X.  I didn't really assume or state it was specific to FreeBSD, just that this scenario was on FreeBSD. I happen to do most of the development and testing on OS X and FreeBSD, and I've enjoyed the FreeBSD community for a very long time.



> 
>>   I am using pcap to capture packets and process them. I have a handful of libs statically linked in (pcap is one, the rest don't seem to matter - I can remove them and still see the leak).
>> 
>> Does anyone know of issues regarding malloc/free on multithreaded userland apps?
> hell yeah. it goes like this: you malloc() then forget to free() - boom, you have a memory leak.
> 
> you're welcome.


Thanks, very insightful.


> 
> 
> sarcasm aside, those questions still remain: why do you think os/libraries are the problem and not your code?

Because I am tracking all malloc and free calls within the application code (aside from libraries) and I can account for all malloc'ed memory and freed memory in both count and by bytes, yet looking at ps output shows a very different story, and if I leave it run long enough, will consume all memory and swap in the system and then be killed off.  I wrapped malloc/free in a function, and record all memory alloc'ed and free'd.  The only memory I cannot track is memory alloced and freed by libraries I am pulling in (well, can't track easily anyway without hacking through all of their source code).



> you can't post all of it, ok, and we don't want all of it either. can you isolate a specific example of where valid usage of a library causes a leak?


Not really - if I could, I would have fixed it by now.  It's a non-trivial issue - which is why I am beginning to suspect something more complicated than a "oops I forgot to free" kind of error.  Plus, I have seen a few people elsewhere on various forums/mailing lists with similar issues claiming that switching to the Hoard allocator fixed the problem (which doesn't seem to be happy with 32bit FreeBSD - tried it).  I have also seen various comments about pthreads and memory allocators having apparent leaks at some threading level, but not sure.

Thanks,
Eric