ZFS exhausts kernel memory just by importing zpools

[Nagy, Attila]

Hi,

Oh, should've written about that: no I don't use (and never used) dedup.

On 2019. 07. 02. 18:13, Sam Fourman Jr. wrote:
> Hello,
>
> My initial guess is that you may have de-duplication enabled on one 
> (or more) of the underlying datasets.
> **if** this is the case, a simple solution is to add more memory to 
> the machine. (64GB of memory is not sufficient for dedup to be enabled )
>
> -- Sam Fourman Jr.
>
> On Tue, Jul 2, 2019 at 10:59 AM Nagy, Attila <bra at fsn.hu 
> <mailto:bra at fsn.hu>> wrote:
>
>     Hi,
>
>     Running latest stable/12 on amd64 with 64 GiB memory on a machine
>     with
>     44 4T disks. Each disks have its own zpool on it (because I solve the
>     redundancy between machines and not locally with ZFS).
>
>     One example zpool holds 2.2 TiB of data (according to df) and have
>     around 75 million files in hashed directories, this is the typical
>     usage
>     on them.
>
>     When I import these zpools, top says around 50 GiB wired memory
>     (ARC is
>     minimal, files weren't yet touched) and after I start to use (heavy
>     reads/writes) the pools, the free memory quickly disappears (ARC
>     grows)
>     until all memory is gone and the machine starts to kill processes,
>     ends
>     up in a deadlock, where nothing helps.
>
>     If I import the pools one by one, each of them adds around 1-1.5
>     GiB of
>     wired memory.
>
>     Top shows this, right after it came to a halt and nothing else
>     works (I
>     can't log in even on the console):
>
>     last pid: 61878;  load averages:  5.05,  4.42,  2.50    up 0+01:07:23
>     15:45:17
>     171 processes: 1 running, 162 sleeping, 1 stopped, 1 zombie, 6 waiting
>     CPU:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
>     Mem: 7716K Active, 8192 Inact, 84K Laundry, 57G Wired, 180M Buf,
>     14M Free
>     ARC: 21G Total, 10G MFU, 4812M MRU, 4922M Anon, 301M Header, 828M
>     Other
>           5739M Compressed, 13G Uncompressed, 2.35:1 Ratio
>     Swap:
>
>        PID USERNAME    THR PRI NICE   SIZE    RES STATE    C TIME    WCPU
>     COMMAND
>     61412 root          1  20    0    14M  3904K CPU14   14 0:06 1.55% top
>     57569 redis        57  20    0  1272M    64M uwait   22 4:28 0.24%
>     consul
>       5574 root          1  20    0    13M  3440K nanslp  10 0:02  
>     0.05% gstat
>       5557 root          1  20    0    20M  7808K select  20 0:00  
>     0.01% sshd
>       5511 root          1  20    0    20M  7808K select   4 0:01  
>     0.01% sshd
>       4955 root          1  20    0    10M  1832K select   9 0:00   0.01%
>     supervis
>       5082 root          1  20    0    25M    14M select   0 0:00  
>     0.00% perl
>       4657 _pflogd       1  20    0    12M  2424K bpf      1 0:00  
>     0.00% pflogd
>       5059 elasticsea    2  20  -20  6983M   385M STOP     5 1:29  
>     0.00% java
>     61669 root          1  26    0    23M      0 pfault   4 0:14 0.00%
>     <python3
>     61624 root          1  20  -20    24M    14M buf_ha   9 0:09 0.00%
>     python3.
>     61626 root          1  20  -20    23M    16K pfault   0 0:08 0.00%
>     python3.
>     61651 root          1  20  -20    23M    14M buf_ha  10 0:08 0.00%
>     python3.
>     61668 root          1  20  -20    23M    13M buf_ha  20 0:08 0.00%
>     python3.
>
>     I've already tried to shrink ARC and vm.kmem_size without too much
>     success.
>
>     Any ideas what causes this?
>
>     _______________________________________________
>     freebsd-fs at freebsd.org <mailto:freebsd-fs at freebsd.org> mailing list
>     https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>     To unsubscribe, send any mail to
>     "freebsd-fs-unsubscribe at freebsd.org
>     <mailto:freebsd-fs-unsubscribe at freebsd.org>"
>
>
>
> -- 
>
> Sam Fourman Jr.