ZFS exhausts kernel memory just by importing zpools

[Nagy, Attila]

Hi,

Running latest stable/12 on amd64 with 64 GiB memory on a machine with 
44 4T disks. Each disks have its own zpool on it (because I solve the 
redundancy between machines and not locally with ZFS).

One example zpool holds 2.2 TiB of data (according to df) and have 
around 75 million files in hashed directories, this is the typical usage 
on them.

When I import these zpools, top says around 50 GiB wired memory (ARC is 
minimal, files weren't yet touched) and after I start to use (heavy 
reads/writes) the pools, the free memory quickly disappears (ARC grows) 
until all memory is gone and the machine starts to kill processes, ends 
up in a deadlock, where nothing helps.

If I import the pools one by one, each of them adds around 1-1.5 GiB of 
wired memory.

Top shows this, right after it came to a halt and nothing else works (I 
can't log in even on the console):

last pid: 61878;  load averages:  5.05,  4.42,  2.50    up 0+01:07:23  
15:45:17
171 processes: 1 running, 162 sleeping, 1 stopped, 1 zombie, 6 waiting
CPU:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
Mem: 7716K Active, 8192 Inact, 84K Laundry, 57G Wired, 180M Buf, 14M Free
ARC: 21G Total, 10G MFU, 4812M MRU, 4922M Anon, 301M Header, 828M Other
      5739M Compressed, 13G Uncompressed, 2.35:1 Ratio
Swap:

   PID USERNAME    THR PRI NICE   SIZE    RES STATE    C TIME    WCPU 
COMMAND
61412 root          1  20    0    14M  3904K CPU14   14 0:06   1.55% top
57569 redis        57  20    0  1272M    64M uwait   22 4:28   0.24% consul
  5574 root          1  20    0    13M  3440K nanslp  10 0:02   0.05% gstat
  5557 root          1  20    0    20M  7808K select  20 0:00   0.01% sshd
  5511 root          1  20    0    20M  7808K select   4 0:01   0.01% sshd
  4955 root          1  20    0    10M  1832K select   9 0:00   0.01% 
supervis
  5082 root          1  20    0    25M    14M select   0 0:00   0.00% perl
  4657 _pflogd       1  20    0    12M  2424K bpf      1 0:00   0.00% pflogd
  5059 elasticsea    2  20  -20  6983M   385M STOP     5 1:29   0.00% java
61669 root          1  26    0    23M      0 pfault   4 0:14   0.00% 
<python3
61624 root          1  20  -20    24M    14M buf_ha   9 0:09   0.00% 
python3.
61626 root          1  20  -20    23M    16K pfault   0 0:08   0.00% 
python3.
61651 root          1  20  -20    23M    14M buf_ha  10 0:08   0.00% 
python3.
61668 root          1  20  -20    23M    13M buf_ha  20 0:08   0.00% 
python3.

I've already tried to shrink ARC and vm.kmem_size without too much success.

Any ideas what causes this?