Sudden grow of memory in "Laundry" state

[Robert]

Sorry, let me be more specific.

Please look into: 
https://docs.google.com/spreadsheets/d/152qBBNokl4mJUc6T6wVTcxaWOskT4KhcvdpOL68gEUM/edit?usp=sharing 
(wait until charts fully loaded).

These are all memory states and mmap\munmap stats collected. Y axis is 
in MBs, X is a timeline.

It's not a problem to understand which process produces allocations and 
is being swapped. I know this for sure.

The issue is: I strongly believe that by some reason FreeBSD kernel 
fails to reuse deallocated memory properly.

Looking into graphs we can see following:

1. When process allocates memory (mmap), "Active" memory increases, 
"Free" memory decreases (that's expected).

2. When process deallocates memory (munmap), "Inactive" memory 
increases, "Active" memory decreases.

Memory never returns into "Free" state. That's kind of expected as well.

3. At some point, when sum of "Active" and "Inactive" memory exceeds 
some upper memory limits,

OS starts to push "Inactive" memory into "Laundry" and "Swap". This 
happens very quick and unexpectedly.

Now why OS doesn't reuse huge amounts of "Inactive" memory when calling 
mmap?

Or my assumption about availability of "Inactive" memory is wrong? Which 
one is free for allocations then?

Thanks.


On 10/24/18 11:58 AM, Mark Millard wrote:
> On 2018-Oct-24, at 1:25 PM, Robert <robert.ayrapetyan at gmail.com> wrote:
>
>> Sorry, that wasn't my output, mine (related to the screenshot I've sent earlier) is:
> No screen shot made it through the list back out to those that
> get messages from the freebsd-hackers at freebsd.org reference
> in the CC. The list limits itself to text as I understand.
>
>> Mem: 1701M Active, 20G Inact, 6225M Laundry, 2625M Wired, 280M Free
>> ARC: 116M Total, 6907K MFU, 53M MRU, 544K Anon, 711K Header, 55M Other
>>       6207K Compressed, 54M Uncompressed, 8.96:1 Ratio
>> Swap: 32G Total, 15G Used, 17G Free, 46% Inuse
> Relative to my limited point: I do not know the status of
> mutually-exclusive categorizations vs. not for ZFS ARC and
> Mem.
>
> Unfortunately, as I understand things, it is questionable if
> adding -S to the top command gives you swap information that
> can point to what makes up the 15G swapped out by totaling
> the sizes listed. But you might at least be able to infer
> what processes became swapped out even if you can not get
> a good size for the swap space use for each.
>
> Using -ores does seem like it puts the top users of resident
> memory at the top of top's process list.
>
> Sufficient Active RAM use by processes that stay active will
> tend to cause inactive processes to be swapped out. FreeBSD
> does not swap out processes that stay active: it pages those
> as needed instead of swapping.
>
> So using top -Sores  might allow watching what active
> process(es) grow and stay active and what inactive processes
> are swapped out at the time of the activity.
>
> I do infer that the 15G Used for Swap is tied to processes
> that were not active when swapped out.
>
>> I'm OK with a low "Free" memory if OS can effectively allocate from "Inactive",
>>
>> but I'm worrying about a sudden move of a huge piece of memory into "Swap" without any relevant mmap calls.
>>
>>
>> My question is: what else (except mmap) may reduce "Free" memory and increase "Laundry"\"Swap" in the system?
>>
>> Thanks.
>>
>>
>> On 10/24/18 9:34 AM, Mark Millard wrote:
>>> On 2018-Oct-24, at 11:12 AM, Rozhuk Ivan <rozhuk.im at gmail.com> wrote:
>>>
>>>> On Wed, 24 Oct 2018 10:19:20 -0700
>>>> Robert <robert.ayrapetyan at gmail.com> wrote:
>>>>
>>>>> So the issue is still happening. Please check attached screenshot.
>>>>> The green area is "inactive + cached + free".
>>>>>
>>>>>   . . .
>>>> +1
>>>> Mem: 845M Active, 19G Inact, 4322M Laundry, 6996M Wired, 1569M Buf, 617M Free
>>>> Swap: 112G Total, 19M Used, 112G Free
>>> Just a limited point based on my understanding of "Buf" in
>>> top's display . . .
>>>
>>> If "cached" means "Buf" in top's output, my understanding of Buf
>>> is that it is not a distinct memory area. Instead it totals the
>>> buffer space that is spread across multiple states: Active,
>>> Inactive, Laundry, and possibly Wired(?).
>>>
>>> In other words: TotalMemory = Active+Inact+Laundry+Wired+Free.
>>> If Buf is added to that then there is double counting of
>>> everything included in Buf and the total will be larger
>>> than the TotalMemory.
>>>
>>> Also Inact+Buf+Free may double count some of the Inact space,
>>> the space that happens to be inactive buffer space.
>>>
>>> I may be wrong, but that is my understanding.
>>>
>
> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
>