Swap exhaustion

[Eduardo Morras]

On Sat, 8 Oct 2016 18:33:47 -0700
Doug Hardie <doug at mail.sermon-archive.info> wrote:

> 
> > On 27 May 2015, at 16:49, Doug Hardie <bc979 at lafn.org> wrote:
> > 
> > I have a process that is eating up 6 GB of swap space.  At that
> > point, FreeBSD 9.3 terminates a process.  However, occasionally its
> > not the one eating up the space.  When I manually quit the process
> > then the swap space returns to a few KB used.  The system runs fine
> > after that.
> > 
> > I have very little knowledge of what this process is doing
> > internally but would like to know what might be causing this
> > issue.  There are 5 of these processes running (parent plus 4
> > children).  Normally each uses about 90 MB RES/SIZE.  However when
> > the problem occurs they are about 2GB RES/SIZE each.  The system
> > has 4 GB memory.  I thought that a malloc would not be able to grab
> > that much memory, even with swapping as it has to be in memory.
> > Could a malloc cause this growth in process size or need I look
> > elsewhere?
> 
> I have been trying periodically to resolve this issue.  I modified
> the application to log (syslog) every memory location it allocates.
> Interestingly enough they are all around 0x100F380.  However, the
> segments that are increasing in number all have the highest order
> address bit set, i.e., 0x83ac00000.  These are shown by procstat as
> type "df".  The number of these increases with time.  The sizes vary
> from what appears to be small to over 5000 resident pages.  They
> never seem to go away unless I restart the process.  Right after a
> restart there are 7 "df"s.  However, with time there can eventually
> be hundreds.  Since these do not have any file backing them, they eat
> up swap space and thus cause the system to run out of swap and start
> killing processes.
> 
> What things allocate memory at the top of the address space?  The
> application mallocs and mmaps all allocate much lower in the address
> space.

FreeBSD 9.3 uses (afair) jemalloc 3.4. There are tons of bug fixes in
newer versions, last one 4.2.1. Perhaps you were bitten by one.
There is not a port :( but compiles nicely under FreeBSD 10.3 (don't
tried under 9.3).

https://github.com/jemalloc/jemalloc/blob/dev/ChangeLog
http://jemalloc.net/

---   ---
Eduardo Morras <emorrasg at yahoo.es>