[Bug 235125] Process was killed: out of swap space on gmirror + zfs

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235125

            Bug ID: 235125
           Summary: Process was killed: out of swap space on gmirror + zfs
           Product: Base System
           Version: 11.2-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: alaa.alassafin at card-1.com

Hello,

We've got FreeBSD 11.2p8 installed on 2 SSDs which are mirrored using gmirror.
we also have a zpool which consists of 3x7 raidz1 drives, 2x Cache and 2x SLOG
and 1x spare.
dual Xeon CPUs and 128GB of Ram.

The first time this Problem accrued right after the 11.2p8 upgrade from p4:

----
Jan 14 17:04:22 san2 zfsd: POLLHUP detected on devd socket.
Jan 14 17:04:22 san2 kernel: pid 606 (devd), uid 0, was killed: out of swap
space
Jan 14 17:04:22 san2 kernel: Jan 14 17:04:22 san2 kernel: pid 606 (devd), uid
0, was killed: out of swap space
Jan 14 17:04:22 san2 zfsd: Disconnecting from devd.
Jan 14 17:04:22 san2 zfsd: ConnectToDevd: Connecting to devd.
----

we had to restart the machine. After 3 days we had the same Problem, but this
time multiple processes were killed:

------
Jan 19 10:49:49 san2 kernel: pid 610 (devd), uid 0, was killed: out of swap
space
Jan 19 10:49:49 san2 kernel: Jan 19 10:49:49 san2 kernel: pid 610 (devd), uid
0, was killed: out of swap space
Jan 19 11:09:49 san2 kernel: pid 835 (zabbix_agentd), uid 122, was killed: out
of swap space
Jan 19 11:09:49 san2 kernel: Jan 19 11:09:49 san2 kernel: pid 835
(zabbix_agentd), uid 122, was killed: out of swap space
Jan 19 11:10:48 san2 kernel: pid 847 (bareos-fd), uid 0, was killed: out of
swap space
Jan 19 11:10:48 san2 kernel: Jan 19 11:10:48 san2 kernel: pid 847 (bareos-fd),
uid 0, was killed: out of swap space
Jan 19 11:11:15 san2 kernel: pid 838 (ntpd), uid 233, was killed: out of swap
space
Jan 19 11:11:15 san2 kernel: Jan 19 11:11:15 san2 kernel: pid 838 (ntpd), uid
233, was killed: out of swap space
Jan 19 11:11:29 san2 kernel: pid 802 (ctld), uid 0, was killed: out of swap
space
Jan 19 11:11:29 san2 kernel: Jan 19 11:11:29 san2 kernel: pid 802 (ctld), uid
0, was killed: out of swap space
Jan 19 11:11:45 san2 kernel: pid 116 (adjkerntz), uid 0, was killed: out of
swap space
Jan 19 11:11:45 san2 kernel: Jan 19 11:11:45 san2 kernel: pid 116 (adjkerntz),
uid 0, was killed: out of swap space
Jan 19 11:12:15 san2 kernel: pid 971 (getty), uid 0, was killed: out of swap
space
Jan 19 11:12:15 san2 kernel: Jan 19 11:12:15 san2 kernel: pid 971 (getty), uid
0, was killed: out of swap space
Jan 19 11:12:29 san2 kernel: pid 32950 (getty), uid 0, was killed: out of swap
space
Jan 19 11:12:29 san2 kernel: Jan 19 11:12:29 san2 kernel: pid 32950 (getty),
uid 0, was killed: out of swap space
Jan 19 11:12:46 san2 kernel: pid 32951 (getty), uid 0, was killed: out of swap 
-----

Messages kept on repeating until we restarted the machine.
We tried disabling zfsd, but that didn't help.

This Machine is in production its really frustrating to have this behavior. I
will gladly provide any more info/tests when needed.

Thank you


Here is some more info:
------

root at san2:~ # gmirror status
       Name    Status  Components
mirror/boot  COMPLETE  gpt/boot0 (ACTIVE)
                       gpt/boot1 (ACTIVE)
mirror/swap  COMPLETE  gpt/swap0 (ACTIVE)
                       gpt/swap1 (ACTIVE)
mirror/root  COMPLETE  gpt/root0 (ACTIVE)
                       gpt/root1 (ACTIVE)

zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH 
ALTROOT
san2pool  37.9T  17.4T  20.4T        -         -     2%    46%  1.00x  ONLINE 
-


root at san2:~ # vmstat
procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr ad0 ad1   in    sy    cs us sy id
0 0 2 409M  3.2G  1170   1   5   4  1569 1355   0   0 3837   800  9140  0  7 93


Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap   8388604    26528  8362076     0%

-- 
You are receiving this mail because:
You are the assignee for the bug.