Re[3]: Weird change of behavior of VSZ allocation in 14.2 compared to 12.4

> 
> 
> > > 
> > > 
> > > Hi Konstantin,
> > > 
> > > Thanks a lot for such a detailed answer and suggestions regarding custom allocators.
> > > We were suspecting that this is a new reality of VSZ management by the allocator and there is nothing wrong with it. And you have confirmed it.
> > > 
> > > So, we're going to come up with another strategy to safeguard against stray memory consumption.
> > > 
> > > Regarding defragmentation working, it doesn't seem to be the case. During the previous restart there indeed were several ranges, hinting at some defragmentation logic working. But, since the last restart (continuation of data sent in a previous message) we now observe:
> > > 
> > > 44608 0x861f75000 0x861f95000 rw- 3 3 1 0 ---D- sw 0.125 0.0117188
> > > 44608 0x1e853c200000 0x1e853c3e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875
> > > 44608 0x1e853c400000 0x1e85403eb000 rw- 13103 13103 1 0 ----- sw 63.918 51.1836
> > > 44608 0x1e8540400000 0x1e8f08a21000 rw- 4122954 4122954 1 0 --S-- sw 40070.1 16105.3
> > > 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625
> > > 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125
> > > 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0
> > > 
> > > One huge chunk of almost 40GiB while only 16GiB are actually in use. And it only grows from there.
> > > 
> > I have seen this sort of behavior in long-lived processes myself; what I suspect is going on (and its substantiated by changes I made that addressed it) is that something being allocated is "blocking" consolidation/defragmentation and, in some cases, this becomes pathological.
> > For example, if I use some of Openssl's EVP hash generator functions (which requires I get a context) and then free the context after use this can happen.
> > I've gotten rather strategic about this sort of use to evade that in things like a fastcgi app that may have a lifetime of "since the machine booted" and "millions of potential transactions" over that time -- for example, if I have a function that does hashes in there for some reason and uses openssl's EVP functions to do so I'll initialize the context on application start and then "reset" it after each use rather than free and get another one next time through. This puts the RAM grab for it near the "front" of what the application allocates and thus avoids the risk. If I don't do this I run the risk that defrag/reuse gets blocked (e.g. there's nothing contiguous of the allocation request available, thus a new block of storage is allocated) and VSS will expand every time that occurs even though there is no actual leak of RAM. It is unlikely to be trouble if the allocations are modest in size but if you need to allocate a large buffer it can become a significant issue.
> > If you don't have control over the allocation or when it occurs as is the case in many languages that abstract that away from you as a programmer.....
> > -- 
> > Karl Denninger
> > karl@denninger.net
> > The Market Ticker
> > [S/MIME encrypted email preferred]
> > 
> 
> 
> Hi Karl,
> 
> Thanks for your input. This is indeed a process that lives as long as possible, which every server app strives for. And it does all sorts of allocations, external libraries included. Though it doesn't leak. Two servers loaded roughly the same level exhibit roughly the same amount of RES consumption. While on 14.2 the VSZ just grows and grows, basically forever, on 12.4 it stays very close to RES, basically the same + a small margin. There it runs for months without issues.
> 
> The funny thing is that there seem to be some connection to how long the memory stays 'occupied' (time in between malloc() and free()). So, basically, the bulk of the memory is some cache that is evicted by LRU as well as amount of time the record stays unused. Normally the 'unused time' threshold is large and we just evict by LRU (via count limit). And this is the result that we observe: a creeping growth of VSZ. However, when 'unused time' threshold is set to a smaller value like 5min the efficiency of cache, of course, drops and the memory consumption (RES) drops by 25% (less records in cache at any time). The VSZ seems to stop growing indefinitely: it settles at some ratio like x2 of RES and stays like that for many days.
> 
> Maybe it was just a lucky coincidence. We're going to repeat this experiment again and share the results.
> 

Hi,

Reporting back to share our findings. Unfortunately, it *was* a lucky coincidence. Even with 'unused time' threshold eviction of 5min we observe the same unlimited growth of VSZ, since yesterday's restart we're already at 3:1 ratio of VSZ to RSS:


22227 0x801c484f000 0x801c4851000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125
22227 0x1f7ed1400000 0x1f7ed15e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875
22227 0x1f7ed1600000 0x1f7ed59e7000 rw- 14665 14665 1 0 --S-- sw 67.9023 57.2852
22227 0x1f7ed5a00000 0x1f7ee5f42000 rw- 63610 63610 1 0 ----- sw 261.258 248.477
22227 0x1f7ee6000000 0x1f873b200000 rw- 3021161 3021161 1 0 --S-- sw 34130 11801.4
22227 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0

As before, there's one huge slab that isn't getting defragmented. Allocated memory is mostly constantly recycled and there aren't many allocations that are kept for a long time. Especially overnight. This is also confirmed by the picture seen on 12.4-STABLE. So we can't understand what prevents defragmentation from happening. This VSZ growth is also typical in other processes: the longer they run (or more likely the more allocations they make during their lifetime) the larger VSZ gets.