[Bug 277389] Reproduceable low memory freeze on 14.0-RELEASE-p5

From: <bugzilla-noreply_at_freebsd.org>
Date: Sun, 17 Mar 2024 05:45:48 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277389

--- Comment #20 from Mark Millard <marklmi26-fbsd@yahoo.com> ---
(In reply to Mark Millard from comment #19)

I updated to a PkgBase vintage of:

# uname -apKU
FreeBSD 7950X3D-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT main-n268827-75464941dc17
GENERIC-NODEBUG amd64 amd64 1500015 1500015

and added to /boot/loader.conf :

nullfs_load="YES"
vfs.nullfs.cache_vnodes=0

I reran the test for such a boot context and the vmstat -z
had a notable difference for VNODE, but the Wired value stayed
huge. No hangup or OOM activity happened.

I'll remind that my prior report had (after the rm):

VNODE:                  448,      0, 1756979,  325855,17353842,   0,   0,   0

but this test got:

VNODE:                  448,      0,     836,    1846,   43727,   0,   0,   0

Yet: 183850Mi Wired still.

[Note added later, well after the below vmstat -z output: it has slowly dropped
to
151961Mi Wired and continues to decrease. I'll do another vmstat -z later.]

For reference (183850Mi time frame):

# rm iozone.DUMMY.*
# vmstat -z
ITEM                   SIZE   LIMIT     USED     FREE      REQ FAIL SLEEP XDOM
kstack_cache:         16384,      0,    1865,       4,    1933,   0,   0,   0
buffer arena-40:       4096,      0,       0,       0,       0,   0,   0,   0
buffer arena-81:       8192,      0,       0,       0,       0,   0,   0,   0
buffer arena-12:      12288,      0,       0,       0,       0,   0,   0,   0
buffer arena-16:      16384,      0,      10,       0,      10,   0,   0,   0
buffer arena-20:      20480,      0,       0,       0,       0,   0,   0,   0
buffer arena-24:      24576,      0,       0,       0,       0,   0,   0,   0
buffer arena-28:      28672,      0,       0,       0,       0,   0,   0,   0
buffer arena-32:      32768,      0,       1,       0,       1,   0,   0,   0
buffer arena-36:      36864,      0,       0,       0,       0,   0,   0,   0
buffer arena-40:      40960,      0,       0,       0,       0,   0,   0,   0
buffer arena-45:      45056,      0,       0,       0,       0,   0,   0,   0
buffer arena-49:      49152,      0,       0,       0,       0,   0,   0,   0
buffer arena-53:      53248,      0,       0,       0,       0,   0,   0,   0
buffer arena-57:      57344,      0,       0,       0,       0,   0,   0,   0
buffer arena-61:      61440,      0,       0,       0,       0,   0,   0,   0
buffer arena-65:      65536,      0,       0,       0,       0,   0,   0,   0
buf free cache:         432,      0,      11,     199,      11,   0,   0,   0
vm pgcache:            4096,      0,  170057,   18447,  315805, 632,   0,   0
vm pgcache:            4096,      0,45110709,    5482,45353298,2948,   0,   0
UMA Kegs:               384,      0,     152,       1,     152,   0,   0,   0
UMA Zones:             4608,      0,     178,       0,     178,   0,   0,   0
UMA Slabs 0:             80,      0,43974304,      40,44081542,   0,   0,   0
UMA Slabs 1:            112,      0,      19,      16,      19,   0,   0,   0
UMA Hash:               256,      0,       0,       0,       0,   0,   0,   0
2 Bucket:                32,      0,     506,    9700,   65241,   0,   0,   0
4 Bucket:                48,      0,     299,    7345,    2335,   0,   0,   0
8 Bucket:                80,      0,     791,    4309,   31017,  13,   0,   0
16 Bucket:              144,      0,     690,    2950, 1560027,   0,   0,   0
32 Bucket:              256,      0,     603,    3762, 1367241, 145,   0,   0
64 Bucket:              512,      0,   19701,     803,  644895,  21,   0,   0
128 Bucket:            1024,      0,   16902,     421,  394790, 109,   0,   0
256 Bucket:            2048,      0,  182806,     210, 1993341,7129,   0,   0
SMR SHARED:              24,      0,       7,     760,       7,   0,   0,   0
SMR CPU:                 32,      0,       7,     760,       7,   0,   0,   0
vmem:                  1856,      0,       1,       7,       1,   0,   0,   0
vmem btag:               56,      0,  163197,    9655,  170132,1200,   0,   0
VM OBJECT:              264,      0,    1344,    6486,   68963,   0,   0,   0
RADIX NODE:             144,      0,  143672,   46332,  340396,   0,   0,   0
KMAP ENTRY:              96,      0,      60,      21,      61,   0,   0,   0
MAP ENTRY:               96,      0,    1434,   22632,  231811,   0,   0,   0
VMSPACE:                616,      0,      30,    1044,    4748,   0,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0,   0
pcpu-4:                   4,      0,       0,       0,       0,   0,   0,   0
pcpu-8:                   8,      0,    4182,    6058,    4184,   0,   0,   0
pcpu-16:                 16,      0,      96,    5792,      96,   0,   0,   0
pcpu-32:                 32,      0,       0,       0,       0,   0,   0,   0
pcpu-64:                 64,      0,     471,    1321,     471,   0,   0,   0
malloc-16:               16,      0,   19551,    6405,265166105,   0,   0,   0
malloc-32:               32,      0,   14392,  123074,25514596,   0,   0,   0
malloc-64:               64,      0,   20549,   55681,273950445,   0,   0,   0
malloc-128:             128,      0,   48447,  541824,317116223,   0,   0,   0
malloc-256:             256,      0,    4288,  267902,28517402,   0,   0,   0
malloc-384:             384,      0,    2379, 1369911,12869523,   0,   0,   0
malloc-512:             512,      0,     115,   24733, 4566575,   0,   0,   0
malloc-1024:           1024,      0,    2037,     987,   30539,   0,   0,   0
malloc-2048:           2048,      0,     993,    1189,17374795,   0,   0,   0
malloc-4096:           4096,      0,    1234,     665,   27051,   0,   0,   0
malloc-8192:           8192,      0,     144,      15,    1210,   0,   0,   0
malloc-16384:         16384,      0,      23,      94,  228721,   0,   0,   0
malloc-32768:         32768,      0,      16,      14,     870,   0,   0,   0
malloc-65536:         65536,      0,      18,      14,     820,   0,   0,   0
SLEEPQUEUE:              80,      0,    1993,    3063,    1993,   0,   0,   0
kenv:                   258,      0,      23,     907,    8562,   0,   0,   0
Files:                   80,      0,     113,    3937,   27558,   0,   0,   0
filedesc0:             1072,      0,      49,     791,    4765,   0,   0,   0
PWD:                     40,      0,      17,    7356,     165,   0,   0,   0
rl_entry:                40,      0,     733,    7448,     737,   0,   0,   0
rangeset pctrie nodes:  144,      0,       0,       0,       0,   0,   0,   0
TURNSTILE:              136,      0,    1993,    1241,    1993,   0,   0,   0
lkpicurr:               168,      0,       2,    3118,       2,   0,   0,   0
lkpimm:                  56,      0,       1,    3095,       1,   0,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0,   0
umtx_shm:                88,      0,       0,       0,       0,   0,   0,   0
ertt:                    72,      0,      10,     886,      13,   0,   0,   0
ertt_txseginfo:          40,      0,       0,    5858,     828,   0,   0,   0
swrbuf:                2360,    512,       0,       0,       0,   0,   0,   0
swwbuf:                2360,    256,       0,       0,       0,   0,   0,   0
nfspbuf:               2360,    512,       0,       0,       0,   0,   0,   0
vnpbuf:                2360,   2048,       0,       0,       0,   0,   0,   0
clpbuf:                2360,    512,       0,       0,       0,   0,   0,   0
pbuf:                  2360,     16,       0,    3090,       0,   0,   0,   0
FPU_save_area:         2440,      0,    1867,      89,    3366,   0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0,   0
PROC:                  1376,      0,      48,     744,    4764,   0,   0,   0
PGRP:                   120,      0,      29,    3007,     105,   0,   0,   0
THREAD:                1824,      0,    1865,     127,    1933,   0,   0,   0
cpuset:                 200,      0,     129,    1183,     129,   0,   0,   0
domainset:               40,      0,       0,       0,       0,   0,   0,   0
pkru ranges:             24,      0,       0,       0,       0,   0,   0,   0
audit_record:          1280,      0,       0,       0,       0,   0,   0,   0
mbuf_packet:            256,      0,       0,    6350,     770,   0,   0,   0
mbuf:                   256,78114771,       1,    8139,    5249,   0,   0,   0
mbuf_cluster:          2048,12205433,    6350,     762,    6359,   0,   0,   0
mbuf_jumbo_page:       4096,6102716,       0,     508,       4,   0,   0,   0
mbuf_jumbo_9k:         9216,1808212,       0,       0,       0,   0,   0,   0
mbuf_jumbo_16k:       16384,1017119,       0,       0,       0,   0,   0,   0
epoch_record pcpu:      256,      0,       4,      60,       4,   0,   0,   0
ttyinq:                 160,      0,     150,     975,     450,   0,   0,   0
ttyoutq:                256,      0,      80,    1045,     240,   0,   0,   0
IOMMU_MAP_ENTRY:        104,      0,       0,       0,       0,   0,   0,   0
g_bio:                  376,      0,       0,    2490,50797174,   0,   0,   0
DEVCTL:                1024,   1000,       0,     684,     667,   0,   0,   0
linux_dma_pctrie:       144,      0,       0,       0,       0,   0,   0,   0
linux_dma_object:        32,      0,       0,       0,       0,   0,   0,   0
vtnet_tx_hdr:            24,      0,       0,     168,       0,   0,   0,   0
cryptop:                280,      0,       0,       0,       0,   0,   0,   0
taskq_zone:             160,      0,       0,    5628,   58365,   0,   0,   0
VNODE:                  448,      0,     836,    1846,   43727,   0,   0,   0
BUF TRIE:               144,      0,       5,  105947,       5,   0,   0,   0
S VFS Cache:            104,      0,    1195,    3095,    1840,   0,   0,   0
STS VFS Cache:          144,      0,       0,       0,       0,   0,   0,   0
L VFS Cache:            320,      0,       0,      36,       1,   0,   0,   0
LTS VFS Cache:          360,      0,       0,       0,       0,   0,   0,   0
NAMEI:                 1024,      0,       0,     512,   40505,   0,   0,   0
NCLNODE:                608,      0,       0,       0,       0,   0,   0,   0
TMPFS node:             232,      0,       0,       0,       0,   0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0,   0
pipe:                   728,      0,       2,     471,    2043,   0,   0,   0
AIO:                    208,      0,       0,       0,       0,   0,   0,   0
AIOCB:                  552,      0,       0,       0,       0,   0,   0,   0
AIOLIO:                 272,      0,       0,       0,       0,   0,   0,   0
Mountpoints:           2816,      0,      40,      56,      40,   0,   0,   0
zfs_btree_leaf_cache:  4096,      0,    2093,    2930,   24200,   0,   0,   0
metaslab_alloc_trace_cache:72,      0,       0,       0,       0,   0,   0,   0
brt_entry_cache:         40,      0,       0,       0,       0,   0,   0,   0
brt_pending_entry_cache:160,      0,       0,       0,       0,   0,   0,   0
ddt_cache:            24800,      0,       5,       1,       5,   0,   0,   0
ddt_entry_cache:        392,      0,       0,       0,       0,   0,   0,   0
zio_cache:             1216,      0,      36,  799014,310814940,   0,   0,   0
zio_link_cache:          48,      0,       0,  798588,42176120,   0,   0,   0
zio_buf_comb_512:       512,      0,      83,   19237, 4276735,   0,   0,   0
zio_buf_comb_1024:     1024,      0,      96,     400,     187,   0,   0,   0
zio_buf_comb_1536:     1536,      0,      67,     208,     333,   0,   0,   0
zio_buf_comb_2048:     2048,      0,      26,     118,      57,   0,   0,   0
zio_buf_comb_3072:     3072,      0,      32,      48,      59,   0,   0,   0
zio_buf_comb_4096:     4096,      0,    2307,     931,  193961,   0,   0,   0
zio_buf_comb_6144:     6144,      0,       1,      21,      43,   0,   0,   0
zio_buf_comb_8192:     8192,      0,       2,      67,   15859,   0,   0,   0
zio_buf_comb_12288:   12288,      0,       0,      49,     120,   0,   0,   0
zio_buf_comb_16384:   16384,      0,     551,     298,   18392,   0,   0,   0
zio_buf_comb_24576:   24576,      0,       0,      55,     157,   0,   0,   0
zio_buf_comb_32768:   32768,      0,       0,      40,      96,   0,   0,   0
zio_buf_comb_49152:   49152,      0,       3,      51,     201,   0,   0,   0
zio_buf_comb_65536:   65536,      0,       0,      42,     129,   0,   0,   0
zio_buf_comb_98304:   98304,      0,       2,      49,     471,   0,   0,   0
zio_buf_comb_131072: 131072,      0,     318,   48097,21075243,   0,   0,   0
zio_buf_comb_196608: 196608,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_262144: 262144,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_393216: 393216,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_524288: 524288,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_786432: 786432,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_1048576:1048576,      0,       0,       1,      17,   0,   0,   0
zio_buf_comb_1572864:1572864,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_2097152:2097152,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_3145728:3145728,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_4194304:4194304,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_6291456:6291456,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_8388608:8388608,      0,       0,       0,       0,   0,   0,   0
zio_buf_comb_12582912:12582912,      0,       0,       0,       0,   0,   0,  
0
zio_buf_comb_16777216:16777216,      0,       0,       0,       0,   0,   0,  
0
abd_chunk:             4096,      0,    6050,43797698,401901671,   0,   0,   0
sa_cache:               264,      0,     719,     871,    1303,   0,   0,   0
dnode_t:                768,      0,    1662,     773,    4431,   0,   0,   0
arc_buf_hdr_t_full:     232,      0,    3038, 1389585,12609871,   0,   0,   0
arc_buf_hdr_t_l2only:    96,      0,       0,       0,       0,   0,   0,   0
arc_buf_t:               32,      0,     929,   75931,20908852,   0,   0,   0
dmu_buf_impl_t:         296,      0,    1903,   68726,12532431,   0,   0,   0
zil_lwb_cache:          376,      0,       3,     897,     230,   0,   0,   0
zil_zcw_cache:           80,      0,       0,    3950,    1736,   0,   0,   0
sio_cache_0:            136,      0,       0,       0,       0,   0,   0,   0
sio_cache_1:            152,      0,       0,       0,       0,   0,   0,   0
sio_cache_2:            168,      0,       0,       0,       0,   0,   0,   0
zfs_znode_cache:        464,      0,     719,     905,    1303,   0,   0,   0
ksiginfo:               112,      0,     741,    3435,   17755,   0,   0,   0
itimer:                 352,      0,       0,       0,       0,   0,   0,   0
KNOTE:                  160,      0,      11,     239,     267,   0,   0,   0
socket:                 960,6264358,      42,    7578,    5966,   0,   0,   0
IPsec SA lft_c:          16,      0,       0,       0,       0,   0,   0,   0
unpcb:                  256,6264358,      12,    7608,    5697,   0,   0,   0
ripcb:                  392,      0,       1,      35,       1,   0,   0,   0
ripcb ports:             32,      0,       0,       0,       0,   0,   0,   0
tcp_inpcb:             1312,      0,      10,      47,      13,   0,   0,   0
tcp_inpcb ports:         32,      0,       4,     878,       4,   0,   0,   0
udp_inpcb:              424,      0,      17,     766,     185,   0,   0,   0
udp_inpcb ports:         32,      0,       7,    4151,      55,   0,   0,   0
udplite_inpcb:          424,      0,       0,       0,       0,   0,   0,   0
udplite_inpcb ports:     32,      0,       0,       0,       0,   0,   0,   0
routing nhops:          256,      0,       8,     367,      13,   0,   0,   0
tcpreass:                48, 762839,       0,       0,       0,   0,   0,   0
tcp_log:                408,5000000,       0,       0,       0,   0,   0,   0
tcp_log_id_bucket:      176,      0,       0,       0,       0,   0,   0,   0
tcp_log_id_node:        120,      0,       0,       0,       0,   0,   0,   0
rtentry:                168,      0,      11,     373,      13,   0,   0,   0
ipq:                     56,  51200,       0,       0,       0,   0,   0,   0
syncache:               168,  15360,       0,     240,       2,   0,   0,   0
hostcache:               64,  15360,       0,       0,       0,   0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0,   0
tfo:                      4,      0,       0,       0,       0,   0,   0,   0
tfo_ccache_entries:      80,      0,       0,       0,       0,   0,   0,   0
swpctrie:               144,24410876,       0,       0,       0,   0,   0,   0
swblk:                  136,24410866,       0,       0,       0,   0,   0,   0
ada_ccb:                272,      0,       0,       0,       0,   0,   0,   0
da_ccb:                 536,      0,       0,       0,       0,   0,   0,   0
NetGraph items:          72,   4096,       0,     448,      60,   0,   0,   0
NetGraph data items:     72,   4096,       0,     576,      54,   0,   0,   0

-- 
You are receiving this mail because:
You are on the CC list for the bug.