Observations on virtual memory operations

[doug at safeport.com]

On Tue, 29 Dec 2020, Pete Wright wrote:

>
>
> On 12/29/20 9:40 AM, doug at safeport.com wrote:
>> On Tue, 29 Dec 2020, Michael Schuster wrote:
>>
>>> On Tue, Dec 29, 2020, 00:37 Pete Wright <pete at nomadlogic.org> wrote:
>>>
>>>>
>>>>
>>>> On 12/28/20 3:25 PM, doug wrote:
>>>>> I have two servers running jails that "routinely" run out of 
>>>>> swapspace with no demand paging activity. To try and get a handle on 
>>>>> VM/swapspace management I have been tracking swapinfo vs memory use 
>>>>> as measured by top. The numbers do not exactly add up but I assume 
>>>>> that is not involved in my issue.
>>>>>
  <snip>
>> Thank you all for the information and thoughts. If vmstat produces 
>> correct infomation there is no demand paging. The limiting condition 
>> on these systems is swapfile space rather than real memory. There are 
>> 69 sysctl elements dealing with paging and swapfile. If there is 
>> documentation (other than in C) on these that would be helpful 
>> perhaps. Most are totals, demand paging rates may be in this set, but 
>> not so as I can tell.
>>
>> The one time I caught the system dying the limiting resource was 
>> swapspace. There was no paging (last vmstat) and about 670MB left in 
>> the swapfile. In this state I could recover by restarting apache.
>
> I wouldn't go down that rabbit hole just yet.  If the issue is with 
> apache-httpd causing your memory to run away I would instead focus on 
> trying to determine *why* httpd is doing that.  Generally a well behaved 
> process should not need to page out to disk if the system is 
> appropriately sized and configured.  As such I would suggest starting at 
> the application layer before trying to tweak how FreeBSD manages paging 
> out to disk.
>
> For example, I remember issues back in the day where httpd would consume 
> tons of memory if people were uploading files.  We were able to address 
> this by being more aggressive in how we wrote files to disk in chunks 
> during the upload process.

I do not seem to be able to say clearly enough, there is no memory problem, 
the is a problem with the paging subsystem filling up swapspace. I have 
monitored memory with vmstat, systat, top, a perl program I found that 
makes a report using sysctl, and a python program I wrote to see how swap 
space is allocated among the jails. As far as I can see other than forking, 
there is no paging and assuming 100% of active memory is backed up, there 
is ample swap space. All that said there are so many data elements that 
sorta match that making definitive statements with a pretty good 
understanding of the code is risky.

when the system starts with the 'out of swap space' message being logged, 
the console is totally locked up and pretty much the rest of these system 
along with it. I keep an xterm with an apache shutdown command handy as one 
enter key will get thourgh quicker than I can enlist the data center 
personel to do a manual reboot. It would appear at this point this issue is 
unique to me.

In reading Dillon's writeup on VM processing it seems possbile that space 
allocated to the inactive and/or laundry queues is not being cleared. I 
found one guy suggesting that directing all console output to a file at 
least prevents losing the serial console. I wonder if anyone has experience 
with that. If there is a program that will fill up the swap file I could 
test this. I am somewhat reluctant to try this on a production system 
without having any experience with the fall out of doing this.