Swap exhaustion

[Doug Hardie]

> On 27 May 2015, at 16:49, Doug Hardie <bc979 at lafn.org> wrote:
> 
> I have a process that is eating up 6 GB of swap space.  At that point, FreeBSD 9.3 terminates a process.  However, occasionally its not the one eating up the space.  When I manually quit the process then the swap space returns to a few KB used.  The system runs fine after that.
> 
> I have very little knowledge of what this process is doing internally but would like to know what might be causing this issue.  There are 5 of these processes running (parent plus 4 children).  Normally each uses about 90 MB RES/SIZE.  However when the problem occurs they are about 2GB RES/SIZE each.  The system has 4 GB memory.  I thought that a malloc would not be able to grab that much memory, even with swapping as it has to be in memory.  Could a malloc cause this growth in process size or need I look elsewhere?

I have been trying periodically to resolve this issue.  I modified the application to log (syslog) every memory location it allocates.  Interestingly enough they are all around 0x100F380.  However, the segments that are increasing in number all have the highest order address bit set, i.e., 0x83ac00000.  These are shown by procstat as type "df".  The number of these increases with time.  The sizes vary from what appears to be small to over 5000 resident pages.  They never seem to go away unless I restart the process.  Right after a restart there are 7 "df"s.  However, with time there can eventually be hundreds.  Since these do not have any file backing them, they eat up swap space and thus cause the system to run out of swap and start killing processes.

What things allocate memory at the top of the address space?  The application mallocs and mmaps all allocate much lower in the address space.