Re: Wired Memory Increasing about 500MBytes per day

From: Mark Johnston <markj_at_freebsd.org>
Date: Tue, 3 Aug 2021 10:30:38 -0400
On Tue, Aug 03, 2021 at 04:59:34PM +0300, Andrey V. Elsukov wrote:
> 03.08.2021 16:47, Mark Johnston пишет:
> >> We noticed the same problem, I'm not sure the exact version, but you can
> >> check the output:
> >> # vmstat -z | egrep "ITEM|pgcache"
> >>
> >> The page cache grows until lowmem is not reached. Then it automatically
> >> cleans and begins to grow again.
> > 
> > The pgcache zones simply provide a per-CPU cache and allocator for
> > physical page frames.  The sizes of the caches are bounded.  The numbers
> > of "used" items from the pgcache zones do not really tell you anything
> > since those pages may be allocated for any number of purposes, including
> > for other UMA zones.  For instance, if ZFS allocates a buffer page from
> > its ABD UMA zone, and that zone's caches are empty, UMA may allocate a
> > new slab using uma_small_alloc() -> vm_page_alloc() -> pgcache zone.
> > 
> > So if there is some wired page leak, the pgcache zones are probably not
> > directly responsible.
> 
> We don't see any leaks, but our monitoring shows that "free" memory
> migrates to "wired" and only these zones are grow.

How are you measuring this?  USED or USED+FREE?

> So, we have on the
> graphs linear growing of wired memory over 7 days. When free memory
> reaches ~4% all returns to normal, and then again linear growing for 7
> days. And pgcache zones reset their number of USED items to low value.
> This is on the server with 256G RAM.
> 
> E.g. This is when 9% of free memory left:
> 
> $ vmstat -z | egrep "ITEM|pgcache"
> ITEM                   SIZE  LIMIT     USED     FREE      REQ
> FAILSLEEP XDOMAIN
> vm pgcache:            4096,      0,    5225,     139,  412976,   0,
> 0,   0
> vm pgcache:            4096,      0,28381269,      77,190108006,  24,
> 0,   0
> vm pgcache:            4096,      0,  166358,   11523,1684567513,3054,
>  0,   0
> vm pgcache:            4096,      0,29548679,     576,780034183,1730,
> 0,   0
> $ bc
> >>> 5225+28381269+166358+29548679
> 58101531
> >>> 58101531*4096/1024/1024/1024
> 221
> >>>
> 
> This is when lowmem triggered:
> % vmstat -z | egrep "ITEM|pgcache"
> ITEM                   SIZE  LIMIT     USED     FREE      REQ
> FAILSLEEP XDOMAIN
> vm pgcache:            4096,      0,    5336,     337,  410052,   0,
> 0,   0
> vm pgcache:            4096,      0, 3126129,     117,56689945,  24,
> 0,   0
> vm pgcache:            4096,      0,   49771,    3910,413657845,1828,
> 0,   0
> vm pgcache:            4096,      0, 4249924,     706,224519238, 562,
> 0,   0
> % bc
> >>> 5336+3126129+49771+4249924
> 7431160
> >>> 7431160*4096/1024/1024/1024
> 28
> >>>
> 
> Look at the graph:
> https://imgur.com/yhqK1p8.png
> 
> -- 
> WBR, Andrey V. Elsukov
> 
Received on Tue Aug 03 2021 - 14:30:38 UTC

Original text of this message