All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

[Nagy László Zsolt]

I have experienced a very similar thing. After upgrading my machine from
11.1-R to 11.2-R, the swap space is filled up to about 66% in about
every 2 days. First I tought that it was PostgreSQL, and lowered the
shared_buffers setting, but it only postponed the problem for another day.

The only thing that has changed is the OS version 11.1-R -> 11.2-R. Here
is the top of top:

last pid: 50425;  load averages:  0.19,  0.16, 
0.17                                                       up
15+23:02:21  06:18:18
45 processes:  1 running, 43 sleeping, 1 zombie
CPU:     % user,     % nice,     % system,     % interrupt,     % idle
Mem: 81M Active, 91M Inact, 1577M Laundry, 14G Wired, 226M Free
ARC: 9598M Total, 90M MFU, 8715M MRU, 105K Anon, 199M Header, 594M Other
     8085M Compressed, 15G Uncompressed, 1.84:1 Ratio
Swap: 4096M Total, 3103M Used, 993M Free, 75% Inuse

The ARC value seem to be growing for a while, then it starts to use the
swap heavily.

But this might be unrelated because the swap usage does not go above
80%. (E.g. it does not crash, but it is clearly using swap when it
should not.)

  Laszlo

> On 01/08/2018 07:24, Mark Martinec wrote:
>> I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE
>> and the situation has not improved. Also turned off all services.
>> ZFS is still leaking memory about 30 MB per hour, until the host
>> runs out of memory and swap space and crashes, unless I reboot it
>> first every four days.
>>
>> Any advise before I try to get rid of that faulted disk with a pool
>> (or downgrade to 10.3, which was stable) ?
>>
>>   Mark
>>
>>
>> 2018-07-23 17:12, myself wrote:
>>> After upgrading an older AMD host from FreeBSD 10.3 to 11.1-RELEASE-p11
>>> (amd64), ZFS is gradually eating up all memory, so that it crashes every
>>> few days when the memory is completely exhausted (after swapping heavily
>>> for a couple of hours).
>>>
>>> This machine has only 4 GB of memory. After capping up the ZFS ARC
>>> to 1.8 GB the machine can now stay up a bit longer, but in four days
>>> all the memory is used up. The machine is lightly loaded, it runs
>>> a bind resolver and a lightly used web server, the ps output
>>> does not show any excessive memory use by any process.
> When you say all used up - you mean the amount of wired ram goes higher
> than about 90% physical ram? You can watch the wired amount in top, or
> calculate it as vm.stats.vm.v_wire_count * hw.pagesize
>
> ZFS ARC is marked as wired, there is also vm.max_wired which limits how
> much the kernel can wire, this defaults to 30% ram, so about 1.2G for
> you. It seems these two wired values don't interact and can add up to
> more than physical ram. I have reported this in bug 229764
>
> Try the patch at
> https://reviews.freebsd.org/D7538
> it has given me the best arc related memory improvements I have seen
> since 10.1, I now see arc being released instead of swap being used.
>