Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.

[Mark Johnston]

On Tue, Nov 20, 2018 at 03:42:24PM +0300, Lev Serebryakov wrote:
> 
>  I have server which is mostly torrent box. It uses ZFS and equipped
> with 16GiB of physical memory. It is running 11-STABLE (r339914 now).
> 
>  I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago.
> 
>  I was used to see 13-14GiB of memory in ZFS ARC and it was Ok.
> Sometimes it "locks" under heavy disk load due to ARC memory pressure,
> but it was bearable, and as ZFS is main reason this server exists, I
> didn't limit ARC.
> 
>  But new revision (r339914) shows very strange behaivor: ARC is no more
> than 4GiB, but kernel has 15GiB wired:
> 
> Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free
> ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other
>      2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio
> 
>  It is typical numbers for last week: 15G wired, 237M Free, but only
> 4252M ARC!
> 
>  Where is other 11G of memory?!
> 
> I've checked USED and FREE in "vmstat -z" output and got this:
> 
> $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n |
> tail -20
> 23001088 9171456 MAP ENTRY
> 29680800 8404320 VM OBJECT
> 34417408 10813952 256
> 36377964 2665656 S VFS Cache
> 50377392 53856 sa_cache
> 50593792 622985216 zio_buf_131072
> 68913152 976896 mbuf_cluster
> 73543680 7225344 mbuf_jumbo_page
> 92358552 67848 zfs_znode_cache
> 95731712 51761152 4096
> 126962880 159581760 dmu_buf_impl_t
> 150958080 233920512 mbuf_jumbo_9k
> 165164600 92040 VNODE
> 192701120 30350880 UMA Slabs
> 205520896 291504128 zio_data_buf_1048576
> 222822400 529530880 zio_data_buf_524288
> 259143168 293476864 zio_buf_512
> 352485376 377061376 zio_buf_16384
> 376109552 346474128 dnode_t
> 2943016960 5761941504 abd_chunk
> $
> 
>  And total USED/FREE numbers is very strange for me:
> 
> $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}'
> 5717965420 9328951088
> $
> 
>  So, only ~5.7G is used and 9.3G is free! But why this memory is not
> used by ARC anymore and why is it wired and not free?

Could you show the output of "vmstat -s" when in this state?