11.2-STABLE kernel wired memory leak

[Mike Tancsa]

On 2/12/2019 11:49 AM, Eugene Grosbein wrote:
> 12.02.2019 23:34, Mark Johnston wrote:
>
>> I suspect that the "leaked" memory is simply being used to cache UMA
>> items.  Note that the values in the FREE column of vmstat -z output are
>> quite large.  The cached items are reclaimed only when the page daemon
>> wakes up to reclaim memory; if there are no memory shortages, large
>> amounts of memory may accumulate in UMA caches.  In this case, the sum
>> of the product of columns 2 and 5 gives a total of roughly 4GB cached.
> Forgot to note, that before I got system to single user mode, there was heavy swap usage (over 3.5GB)
> and heavy page-in/page-out, 10-20 megabytes per second and system was crawling slow due to pageing.

I just ran into this issue on a RELENG12 box I was getting ready for the
FreeBSD netperf cluster. It seems pretty easy to trigger. I created a 12
disk raidz pool and did a simple dd test and the box started to run out
of memory

pid 776 (rpc.statd), jid 0, uid 0, was killed: out of swap space
pid 784 (rpc.lockd), jid 0, uid 0, was killed: out of swap space

CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 1120K Active, 628K Inact, 264K Laundry, 30G Wired, 26M Buf, 1133M Free
ARC: 28G Total, 73K MFU, 28G MRU, 32K Anon, 56M Header, 1713K Other
     27G Compressed, 27G Uncompressed, 1.00:1 Ratio

zpool create tanker raidz1 da0p1 da1p1 da2p1 da3p1 da4p1 da5p1 da6p1
da7p1 da8p1 da9p1 da10p1 da11p1

dd if=/dev/zero of=/tanker/test bs=1m count=100000

zpool destroy tanker

last pid:  1078;  load averages:  0.37,  1.32, 
0.84                                                                up
0+00:11:44  19:22:03
32 processes:  1 running, 31 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 564K Active, 792K Inact, 1000K Laundry, 30G Wired, 26M Buf, 1046M Free
Swap:

# vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort
-k1,1 -rn | head
   73899.1 mbuf_cluster:          2048
     23693 mbuf_packet:            256
   4912.17 socket:                 872
   4719.04 unpcb:                  256
   147.354 udpcb:                   32
   147.345 udp_inpcb:              488
   28.8717 tcpcb:                  976
   28.8717 tcp_inpcb:              488
   11.6294 mbuf_jumbo_page:       4096
   2.98672 ripcb:                  488
#





> -- 
> -------------------
> Mike Tancsa, tel +1 519 651 3400 x203
> Sentex Communications, mike at sentex.net
> Providing Internet services since 1994 www.sentex.net
> Cambridge, Ontario Canada