Python malloc issue?

[Andrew MacIntyre]

Amit Dev wrote:
> I'm observing a strange memory usage pattern with python strings on
> Freebsd. Consider
> the following session. Idea is to create a list which holds some
> strings so that cumulative characters in the list is 100MB.
> 
>>>> l = []
>>>> for i in xrange(100000):
> ...  l.append(str(i) * (1000/len(str(i))))
> 
> This uses around 100MB of memory as expected and 'del l' will clear that.
> 
>>>> for i in xrange(20000):
> ...  l.append(str(i) * (5000/len(str(i))))
> 
> This is using 165MB of memory. I really don't understand where the
> additional memory usage is coming from. [Size of both lists are same]
> 
> Python 2.6.4 on FreeBSD 7.2. On Linux and windows both uses around
> 100mb memory only.

A lot depends on the behavior of the system malloc(), and also the 
reporting tools.

Python's internal allocator is only used for objects up to 256 bytes; 
over that the request is passed to the platform malloc().

At a guess, a lot of your requests in the second case are over 4096 
bytes (a single page of memory) and so the allocator is probably padding 
the requests out to 2 pages, which is roughly in proportion with the 
memory consumption you report.

jemalloc's behaviour can be tailored a bit - you might want to read the 
man page.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia