11.2-STABLE kernel wired memory leak

[Mike Tancsa]

On 2/19/2019 2:35 PM, Mike Tancsa wrote:
> dd if=/dev/zero of=/tanker/test bs=1m count=100000

The box has 32G of RAM. If I do a

# sysctl -w vfs.zfs.arc_max=12224866304
vfs.zfs.arc_max: 32224866304 -> 12224866304
#

after WIRED memory is at 29G, it doesnt immediately reclaim it and there
is memory pressure.  Booting the box with
vfs.zfs.arc_max=12224866304
keeps WIRED at 15G


last pid:  1049;  load averages:  2.02,  2.30, 
1.07                                                                up
0+00:04:21  20:17:43
38 processes:  2 running, 36 sleeping
CPU:  3.6% user,  0.0% nice,  5.9% system,  0.0% interrupt, 90.5% idle
Mem: 39M Active, 6212K Inact, 15G Wired, 30M Buf, 16G Free
ARC: 11G Total, 363K MFU, 11G MRU, 117M Anon, 23M Header, 850K Other
     11G Compressed, 11G Uncompressed, 1.01:1 Ratio
Swap:


Re-running the test but this time with swap,

# zpool create tanker raidz1 da0p1 da1p1 da2p1 da3p1 da4p1 da5p1 da6p1
da7p1 da8p1 da9p1 da10p1 da11p1 da12p1 da13p1 da14p1
# pstat -T
120/1043936 files
0M/65536M swap space
#

There is still 30G of Wired memory ?

last pid:  1096;  load averages:  0.78,  2.13, 
1.25                                                                up
0+00:06:42  20:33:21
35 processes:  1 running, 34 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 976K Active, 274M Inact, 88M Laundry, 30G Wired, 1341M Buf, 722M Free
ARC: 1227K Total, 775K MFU, 339K MRU, 32K Anon, 9216 Header, 69K Other
     176K Compressed, 1179K Uncompressed, 6.70:1 Ratio
Swap: 64G Total, 64G Free

I start a new dd on a freshly created pool,

CPU:  0.0% user,  0.0% nice, 31.4% system,  0.8% interrupt, 67.8% idle
Mem: 1396K Active, 460K Inact, 18M Laundry, 30G Wired, 1341M Buf, 720M Free
ARC: 27G Total, 54K MFU, 26G MRU, 1380M Anon, 56M Header, 3459K Other
     26G Compressed, 26G Uncompressed, 1.00:1 Ratio
Swap: 64G Total, 73M Used, 64G Free

and the box starts to swap

last pid:  1104;  load averages: 11.61,  6.10, 
3.04                                                                up
0+00:09:32  20:36:11
36 processes:  2 running, 34 sleeping
CPU:  0.1% user,  0.0% nice, 95.3% system,  2.1% interrupt,  2.4% idle
Mem: 1212K Active, 124K Inact, 688K Laundry, 30G Wired, 1341M Buf, 819M Free
ARC: 27G Total, 54K MFU, 24G MRU, 1620M Anon, 58M Header, 5572K Other
     24G Compressed, 24G Uncompressed, 1.01:1 Ratio
Swap: 64G Total, 90M Used, 64G Free



(avg's oneline script shows)
61,724,160      arc_buf_hdr_t_full          55,263,744      6,460,416
111,783,936     512                         109,534,208     2,249,728
555,884,000     UMA_Slabs                   553,374,640     2,509,360
1,280,966,656   zio_data_buf_131072         905,576,448     375,390,208
28,142,604,288  abd_chunk                   27,965,505,536  177,098,752
30,431,435,848  TOTAL                       29,663,389,468  768,046,380

# vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n",
$2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
       358 zio_data_buf_131072
   168.895 abd_chunk
   18.5738 zio_cache
     14.25 zio_buf_786432
        14 zio_buf_1048576
    13.987 BUF TRIE
    13.125 zio_buf_655360
     9.625 zio_buf_917504
      8.75 zio_buf_458752
         8 zio_buf_524288
#

As the box is netbooted over nfs, when it swaps out some of the nfs
helper apps, the box gets a bit unresponsive.  e.g. if I do a find
/usr/src -type f |xargs md5 the box will stall out for periods of time. 
If I limit ARC to say 12G, all is well.


    ---Mike


-- 
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada