Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.

[Lev Serebryakov]

 I have server which is mostly torrent box. It uses ZFS and equipped
with 16GiB of physical memory. It is running 11-STABLE (r339914 now).

 I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago.

 I was used to see 13-14GiB of memory in ZFS ARC and it was Ok.
Sometimes it "locks" under heavy disk load due to ARC memory pressure,
but it was bearable, and as ZFS is main reason this server exists, I
didn't limit ARC.

 But new revision (r339914) shows very strange behaivor: ARC is no more
than 4GiB, but kernel has 15GiB wired:

Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free
ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other
     2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio

 It is typical numbers for last week: 15G wired, 237M Free, but only
4252M ARC!

 Where is other 11G of memory?!

I've checked USED and FREE in "vmstat -z" output and got this:

$ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n |
tail -20
23001088 9171456 MAP ENTRY
29680800 8404320 VM OBJECT
34417408 10813952 256
36377964 2665656 S VFS Cache
50377392 53856 sa_cache
50593792 622985216 zio_buf_131072
68913152 976896 mbuf_cluster
73543680 7225344 mbuf_jumbo_page
92358552 67848 zfs_znode_cache
95731712 51761152 4096
126962880 159581760 dmu_buf_impl_t
150958080 233920512 mbuf_jumbo_9k
165164600 92040 VNODE
192701120 30350880 UMA Slabs
205520896 291504128 zio_data_buf_1048576
222822400 529530880 zio_data_buf_524288
259143168 293476864 zio_buf_512
352485376 377061376 zio_buf_16384
376109552 346474128 dnode_t
2943016960 5761941504 abd_chunk
$

 And total USED/FREE numbers is very strange for me:

$ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}'
5717965420 9328951088
$

 So, only ~5.7G is used and 9.3G is free! But why this memory is not
used by ARC anymore and why is it wired and not free?

-- 
// Lev Serebryakov

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20181120/ea6352aa/attachment.sig>