Re[2]: Weird change of behavior of VSZ allocation in 14.2 compared to 12.4
Date: Tue, 01 Jul 2025 11:35:33 UTC
> > There is per-uid swap limit. I highly suspect that this is what you want, > if the goal is to limit the amount of anonymous memory allocated by specific > user. > > No idea about VSZ, and nobody would have an idea. Look at the procstat vm > output of the process before and after the peak that bothers you. > Hi Konstantin, Thank you for your answer. Technically we have swap disabled by having no swap device configured in the first place. Unfortunately limiting by UID won't work for us as many, completely different processes with their own memory usage pattern run under the same UID. So we stick with a 'hard' per-process limit, so that OS will kill any stray process before it manages to cause a memory pressure. The solution that has worker in 12.4-STABLE was setting RLIMIT_AS via set_rlimit(). Unfortunately, in 14.2-STABLE VSZ now behaves unpredictably, hence we can no longer rely on RLIMIT_AS large virtual space no longer means a large actual physical memory consumption (RES). It seems that the `procstat vm` output has changed in between 12.4 and 14.2. On 12.4 we observe 680K output lines for a process. Looks like the output in more fine-grained in 12.4, showing each contiguous range of resident pages even for a singe large chunk of a virtual space. For example: ... 99821 0x80fb8b000 0x80fb8f000 rw- 4 24939 2651 0 ----- df 0.015625 99821 0x80fb90000 0x80fbb5000 rw- 37 24939 2651 0 ----- df 0.144531 99821 0x80fbb6000 0x80fbd5000 rw- 31 24939 2651 0 ----- df 0.121094 99821 0x80fbd7000 0x80fbdf000 rw- 8 24939 2651 0 ----- df 0.03125 99821 0x80fbe1000 0x80fbef000 rw- 14 24939 2651 0 ----- df 0.0546875 99821 0x80fbf0000 0x80fbf6000 rw- 6 24939 2651 0 ----- df 0.0234375 99821 0x80fbf7000 0x80fbfc000 rw- 5 24939 2651 0 ----- df 0.0195312 99821 0x80fbfc000 0x80fc17000 rw- 27 83406 24833 0 ----- df 0.105469 99821 0x80fc19000 0x80fd75000 rw- 348 83406 24833 0 ----- df 1.35938 99821 0x80fd76000 0x80fda5000 rw- 47 83406 24833 0 ----- df 0.183594 99821 0x80fda6000 0x80fdaa000 rw- 4 83406 24833 0 ----- df 0.015625 99821 0x80fdab000 0x80fdbe000 rw- 19 83406 24833 0 ----- df 0.0742188 99821 0x80fdc0000 0x80fdf5000 rw- 53 83406 24833 0 ----- df 0.207031 99821 0x80fdf5000 0x80fe09000 rw- 20 85 8 0 ----- df 0.078125 99821 0x80fe0a000 0x80fe0b000 rw- 1 85 8 0 ----- df 0.00390625 99821 0x80fe0e000 0x80fe26000 rw- 24 85 8 0 ----- df 0.09375 99821 0x80fe28000 0x80fe29000 rw- 1 85 8 0 ----- df 0.00390625 99821 0x80fe2b000 0x80fe37000 rw- 12 85 8 0 ----- df 0.046875 99821 0x80fe3a000 0x80fe44000 rw- 10 85 8 0 ----- df 0.0390625 99821 0x80fe47000 0x80fe56000 rw- 15 85 8 0 ----- df 0.0585938 99821 0x80fe59000 0x80fe5b000 rw- 2 85 8 0 ----- df 0.0078125 99821 0x80fe5b000 0x80fe9c000 rw- 65 83406 24833 0 ----- df 0.253906 99821 0x80fe9d000 0x80ff47000 rw- 170 83406 24833 0 ----- df 0.664062 99821 0x80ff48000 0x80ff6c000 rw- 36 83406 24833 0 ----- df 0.140625 99821 0x80ff6d000 0x8100d7000 rw- 362 83406 24833 0 ----- df 1.41406 99821 0x8100d8000 0x8100d9000 rw- 1 83406 24833 0 ----- df 0.00390625 ... the last column was added via awk, as the size of address range in Megabytes. Judging by the repeating numbers in RES and PRES we assume that each such line just a piece of a larger virt range. Probably, here we can only see resident sub-ranges. On 14.2-STABLE, however the picture is completely different. Here we can see only a few of huge ranges that definitely belong to a heap: ... 91683 0x8616d1000 0x8618b1000 --- 0 0 0 0 ----- gd 1.875 0 91683 0x8618b1000 0x8618d1000 rw- 3 3 1 0 ---D- sw 0.125 0.0117188 91683 0x7545ca68000 0x7545ca69000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 91683 0x7545ca69000 0x7545ca6b000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 91683 0x1e3e72600000 0x1e3e727e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875 91683 0x1e3e72800000 0x1e408fd6b000 rw- 1942903 1942903 1 0 --S-- sw 8661.42 7589.46 91683 0x1e408fe00000 0x1e46bb200000 rw- 2191861 2191861 1 0 --S-- sw 25268 8561.96 91683 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 the last two column were added via awk, as the size of address range and resident size in Megabytes. `vn` ranges were filtered out as they are irrelevant here. When observed via `procstat vm`, memory dynamics after restart looks like: 44608 0x861f75000 0x861f95000 rw- 2 2 1 0 ---D- sw 0.125 0.0078125 44608 0x1e853c200000 0x1e853c33b000 rw- 244 244 1 0 ----- sw 1.23047 0.953125 44608 0x1e853c400000 0x1e8540280000 rw- 12699 12699 1 0 ----- sw 62.5 49.6055 44608 0x1e8540400000 0x1e8597e00000 rw- 189350 189350 1 0 ----- sw 1402 739.648 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 2 2 1 0 ---D- sw 0.125 0.0078125 44608 0x1e853c200000 0x1e853c39e000 rw- 345 345 1 0 ----- sw 1.61719 1.34766 44608 0x1e853c400000 0x1e8540280000 rw- 12699 12699 1 0 ----- sw 62.5 49.6055 44608 0x1e8540400000 0x1e8640c00000 rw- 883532 883532 1 0 --S-- sw 4104 3451.3 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 2 2 1 0 ---D- sw 0.125 0.0078125 44608 0x1e853c200000 0x1e853c3e0000 rw- 439 439 1 0 ----- sw 1.875 1.71484 44608 0x1e853c400000 0x1e85402a1000 rw- 12699 12699 1 0 ----- sw 62.6289 49.6055 44608 0x1e8540400000 0x1e8715600000 rw- 1713386 1713386 1 0 --S-- sw 7506 6692.91 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 2 2 1 0 ---D- sw 0.125 0.0078125 44608 0x1e853c200000 0x1e853c3e0000 rw- 439 439 1 0 ----- sw 1.875 1.71484 44608 0x1e853c400000 0x1e85402c2000 rw- 12723 12723 1 0 ----- sw 62.7578 49.6992 44608 0x1e8540400000 0x1e8784600000 rw- 1934867 1934867 1 0 --S-- sw 9282 7558.07 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 3 3 1 0 ---D- sw 0.125 0.0117188 44608 0x1e853c200000 0x1e853c3e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875 44608 0x1e853c400000 0x1e85402e3000 rw- 12786 12786 1 0 ----- sw 62.8867 49.9453 44608 0x1e8540400000 0x1e87e7e00000 rw- 2356654 2356654 1 0 --S-- sw 10874 9205.68 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 3 3 1 0 ---D- sw 0.125 0.0117188 44608 0x1e853c200000 0x1e853c3e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875 44608 0x1e853c400000 0x1e8540346000 rw- 12882 12882 1 0 ----- sw 63.2734 50.3203 44608 0x1e8540400000 0x1e885a600000 rw- 2798472 2798472 1 0 --S-- sw 12706 10931.5 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 3 3 1 0 ---D- sw 0.125 0.0117188 44608 0x1e853c200000 0x1e853c3e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875 44608 0x1e853c400000 0x1e8540367000 rw- 12922 12922 1 0 ----- sw 63.4023 50.4766 44608 0x1e8540400000 0x1e88bc200000 rw- 2904271 2904271 1 0 --S-- sw 14270 11344.8 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 3 3 1 0 ---D- sw 0.125 0.0117188 44608 0x1e853c200000 0x1e853c3e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875 44608 0x1e853c400000 0x1e8540367000 rw- 12933 12933 1 0 ----- sw 63.4023 50.5195 44608 0x1e8540400000 0x1e88fc200000 rw- 3007464 3007464 1 0 --S-- sw 15294 11747.9 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 3 3 1 0 ---D- sw 0.125 0.0117188 44608 0x1e853c200000 0x1e853c3e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875 44608 0x1e853c400000 0x1e8540388000 rw- 12972 12972 1 0 ----- sw 63.5312 50.6719 44608 0x1e8540400000 0x1e899c200000 rw- 3268072 3268072 1 0 --S-- sw 17854 12765.9 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 44608 0x861f75000 0x861f95000 rw- 3 3 1 0 ---D- sw 0.125 0.0117188 44608 0x1e853c200000 0x1e853c3e0000 rw- 440 440 1 0 ----- sw 1.875 1.71875 44608 0x1e853c400000 0x1e8540388000 rw- 12978 12978 1 0 ----- sw 63.5312 50.6953 44608 0x1e8540400000 0x1e8a5f200000 rw- 3526850 3526850 1 0 --S-- sw 20974 13776.8 44608 0x1fb104d95000 0x1fb104d96000 r-- 1 3 2 0 ----- sw 0.00390625 0.00390625 44608 0x1fb104d96000 0x1fb104d98000 rw- 2 3 2 0 ----- sw 0.0078125 0.0078125 44608 0x7ffffffff000 0x800000000000 --- 0 0 0 0 ----- gd 0.00390625 0 Not much different from what we see via `ps`. We're still not sure, why the gap between VSZ and RSS grows and VSZ never goes down, even if memory is released. Seems like there was a change in system allocator and this is its new strategy?