Observations on virtual memory operations
doug at safeport.com
doug at safeport.com
Wed Dec 30 01:03:09 UTC 2020
On Tue, 29 Dec 2020, Pete Wright wrote:
> On 12/29/20 9:40 AM, doug at safeport.com wrote:
>> On Tue, 29 Dec 2020, Michael Schuster wrote:
>>> On Tue, Dec 29, 2020, 00:37 Pete Wright <pete at nomadlogic.org> wrote:
>>>> On 12/28/20 3:25 PM, doug wrote:
>>>>> I have two servers running jails that "routinely" run out of
>>>>> swapspace with no demand paging activity. To try and get a handle on
>>>>> VM/swapspace management I have been tracking swapinfo vs memory use
>>>>> as measured by top. The numbers do not exactly add up but I assume
>>>>> that is not involved in my issue.
>> Thank you all for the information and thoughts. If vmstat produces
>> correct infomation there is no demand paging. The limiting condition
>> on these systems is swapfile space rather than real memory. There are
>> 69 sysctl elements dealing with paging and swapfile. If there is
>> documentation (other than in C) on these that would be helpful
>> perhaps. Most are totals, demand paging rates may be in this set, but
>> not so as I can tell.
>> The one time I caught the system dying the limiting resource was
>> swapspace. There was no paging (last vmstat) and about 670MB left in
>> the swapfile. In this state I could recover by restarting apache.
> I wouldn't go down that rabbit hole just yet. If the issue is with
> apache-httpd causing your memory to run away I would instead focus on
> trying to determine *why* httpd is doing that. Generally a well behaved
> process should not need to page out to disk if the system is
> appropriately sized and configured. As such I would suggest starting at
> the application layer before trying to tweak how FreeBSD manages paging
> out to disk.
> For example, I remember issues back in the day where httpd would consume
> tons of memory if people were uploading files. We were able to address
> this by being more aggressive in how we wrote files to disk in chunks
> during the upload process.
I do not seem to be able to say clearly enough, there is no memory problem,
the is a problem with the paging subsystem filling up swapspace. I have
monitored memory with vmstat, systat, top, a perl program I found that
makes a report using sysctl, and a python program I wrote to see how swap
space is allocated among the jails. As far as I can see other than forking,
there is no paging and assuming 100% of active memory is backed up, there
is ample swap space. All that said there are so many data elements that
sorta match that making definitive statements with a pretty good
understanding of the code is risky.
when the system starts with the 'out of swap space' message being logged,
the console is totally locked up and pretty much the rest of these system
along with it. I keep an xterm with an apache shutdown command handy as one
enter key will get thourgh quicker than I can enlist the data center
personel to do a manual reboot. It would appear at this point this issue is
unique to me.
In reading Dillon's writeup on VM processing it seems possbile that space
allocated to the inactive and/or laundry queues is not being cleared. I
found one guy suggesting that directing all console output to a file at
least prevents losing the serial console. I wonder if anyone has experience
with that. If there is a program that will fill up the swap file I could
test this. I am somewhat reluctant to try this on a production system
without having any experience with the fall out of doing this.
More information about the freebsd-questions