Re: Weird change of behavior of VSZ allocation in 14.2 compared to 12.4
Date: Tue, 01 Jul 2025 03:51:42 UTC
On Mon, Jun 30, 2025 at 06:20:15PM +0300, Paul wrote: > Hi! > > After a recent upgrade to 14.2-STABLE we've noticed a consistent change of ratio between VSZ and RSS across every running process in the system. It affects our custom software, as well as open source software such as nginx. The change is pretty dramatic. As before, the both sized were close to each other, having a seemingly fixed difference, probably a reflection of shared mmaped regions, such a libraries that don't count as RSS as well as slabs that system allocator allocates in. Now, the gap starts small and then grows spontaneously. > > We also have few servers running 13.1-STABLE that seem to behave similar to 12.4-STABLE. > > > Below is the observation of evolution of values, gathered with `ps`. Column `V-R` is a difference between VSZ and RSS. Application is a server after a graceful restart. It slowly grows a set of clients as they migrate over from the old instance. Hence the gradual rise of memory usage. Collection begun few minutes after its start. > > > TIME VSZ RSS V-R COMMENT > Fri Jun 27 10:45:22 EEST 2025 14733672 12785888 1947784 > Fri Jun 27 10:45:27 EEST 2025 14733672 12813680 1919992 > ... > Fri Jun 27 10:52:39 EEST 2025 14733668 13230424 1503244 > Fri Jun 27 10:52:44 EEST 2025 14733668 13238628 1495040 > ... > Fri Jun 27 11:04:06 EEST 2025 14733668 13735400 998268 > Fri Jun 27 11:04:11 EEST 2025 14733668 13745444 988224 > Fri Jun 27 11:04:16 EEST 2025 14995812 13750084 1245728 <- Size increase > Fri Jun 27 11:04:21 EEST 2025 14995812 13760968 1234844 > Fri Jun 27 11:04:26 EEST 2025 14995812 13762616 1233196 > ... > Fri Jun 27 11:12:38 EEST 2025 14995812 14289904 705908 > Fri Jun 27 11:12:43 EEST 2025 14995812 14303756 692056 > ... > Fri Jun 27 11:14:48 EEST 2025 14995788 14400636 595152 > Fri Jun 27 11:14:53 EEST 2025 14995788 14402060 593728 > Fri Jun 27 11:14:58 EEST 2025 16306508 14412192 1894316 <- Size increase > Fri Jun 27 11:15:03 EEST 2025 16306536 14416660 1889876 > Fri Jun 27 11:15:08 EEST 2025 16306536 14416664 1889872 > ... > Fri Jun 27 11:18:49 EEST 2025 16306532 14448932 1857600 > Fri Jun 27 11:18:54 EEST 2025 16306532 14455744 1850788 > Fri Jun 27 11:18:59 EEST 2025 17617252 14461104 3156148 <- Size increase (strange) > Fri Jun 27 11:19:04 EEST 2025 17617228 14466204 3151024 > Fri Jun 27 11:19:09 EEST 2025 17617252 14473432 3143820 > ... > Fri Jun 27 11:35:42 EEST 2025 17617256 14439060 3178196 > Fri Jun 27 11:35:47 EEST 2025 17617256 14448720 3168536 > ... > Fri Jun 27 11:54:21 EEST 2025 17617252 14547200 3070052 > Fri Jun 27 11:54:26 EEST 2025 17617252 14547428 3069824 > ... > Fri Jun 27 12:18:51 EEST 2025 17617384 14490896 3126488 > Fri Jun 27 12:18:56 EEST 2025 17617384 14493676 3123708 > ... > Fri Jun 27 12:21:11 EEST 2025 17617388 14655232 2962156 > Fri Jun 27 12:21:16 EEST 2025 17617388 14663356 2954032 > Fri Jun 27 12:21:21 EEST 2025 19239400 14674992 4564408 <- Size increase (strange) > Fri Jun 27 12:21:27 EEST 2025 19239400 14677000 4562400 > Fri Jun 27 12:21:32 EEST 2025 19239400 14677592 4561808 > ... > Fri Jun 27 12:23:12 EEST 2025 19239400 14705036 4534364 > Fri Jun 27 12:23:17 EEST 2025 19239400 14707284 4532116 > Fri Jun 27 12:23:22 EEST 2025 20812264 14725088 6087176 <- Size increase (strange) > Fri Jun 27 12:23:27 EEST 2025 20812264 14726720 6085544 > Fri Jun 27 12:23:32 EEST 2025 20812264 14727972 6084292 > ... > Fri Jun 27 12:40:20 EEST 2025 20812264 15194128 5618136 > Fri Jun 27 12:40:25 EEST 2025 20812264 15183644 5628620 > ... > Fri Jun 27 12:42:50 EEST 2025 20812264 15266600 5545664 > Fri Jun 27 12:42:55 EEST 2025 20812264 15268296 5543968 > Fri Jun 27 12:43:00 EEST 2025 21139944 15283636 5856308 <- Size increase (strange) > Fri Jun 27 12:43:06 EEST 2025 21139944 15284760 5855184 > Fri Jun 27 12:43:11 EEST 2025 21139944 15290556 5849388 > ... > Fri Jun 27 14:16:49 EEST 2025 21139944 15604296 5535648 > Fri Jun 27 14:16:54 EEST 2025 21139944 15606432 5533512 > Fri Jun 27 14:16:59 EEST 2025 22974952 15620564 7354388 <- Size increase (strange) > Fri Jun 27 14:17:04 EEST 2025 22974952 15626492 7348460 > Fri Jun 27 14:17:09 EEST 2025 22974952 15626556 7348396 > ... > Fri Jun 27 14:20:10 EEST 2025 22974956 15754320 7220636 > Fri Jun 27 14:20:15 EEST 2025 22974928 15758468 7216460 > Fri Jun 27 14:20:20 EEST 2025 24809960 15771748 9038212 <- Size increase (strange) > Fri Jun 27 14:20:25 EEST 2025 24809960 15758564 9051396 > Fri Jun 27 14:20:30 EEST 2025 24809960 15744344 9065616 > ... > ... <- Drift continues and settles around 11GiB: > Sat Jun 28 00:27:03 EEST 2025 29946320 18199812 11746508 > Sat Jun 28 00:27:08 EEST 2025 29946320 18208100 11738220 > > > We don't understand the reasons behind this large size gap. Given exactly the same source code of this server on 12.4-STABLE the gap is always floating around 1GiB: goes up and down, up and down. > Another observation is that on 12.4-STABLE VSZ could go down as application frees memory. On 14.2-STABLE this doesn't happen and there it can only go up. > Application uses strictly free/malloc from the system libc. > > Having a predictable value of VSZ is very important as this is the only practical way of limiting memory consumption of stray applications. Limit on RSS is just a hint for a memory manager and won't prevent some applications from causing the memory pressure in the first place. > There is per-uid swap limit. I highly suspect that this is what you want, if the goal is to limit the amount of anonymous memory allocated by specific user. No idea about VSZ, and nobody would have an idea. Look at the procstat vm output of the process before and after the peak that bothers you.