[Bug 232800] Processes killed out of swap space

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232800

--- Comment #2 from teksimian at gmail.com ---
(In reply to Mark Millard from comment #1)
Thank you for your feedback. I've added vm.pageout_oom_seq=120 to sysctl.conf.

Both of the following messages have been logged;

Oct 30 23:08:20 host kernel: swap_pager: indefinite wait buffer: bufobj: 0,
blkno: 1143, size: 16384
Oct 30 23:16:52 host kernel: swap_pager: I/O error - pageout failed; blkno
477,size 69632, error 12

swap is filebacked;

[user at host /var/log]$ cat /etc/fstab
# Device        Mountpoint      FStype  Options Dump    Pass#
...
md99    none    swap    sw,late,file=/usr/swap0 0       0

the swap file is in /usr which is the same fs as / (it was provisioned this way
by the cloud provider.)  I can roll back the vm to a 10.4 snapshot. I'm
confident the issue wouldnt be present.  The 10.4 install has been solid, and
this began /immediately/ after the upgrade.



Reboot 1

[user at host /var/log]$ sudo less all.log

...

Oct 30 23:05:58 host kernel: Starting jails:
Oct 30 23:08:20 host kernel:
Oct 30 23:08:20 host kernel: swap_pager: indefinite wait buffer: bufobj: 0,
blkno: 1143, size: 16384
Oct 30 23:08:20 host last message repeated 2 times
Oct 30 23:08:20 host kernel: pid 2532 (mysqld), uid 88, was killed: out of swap
space
Oct 30 23:08:20 host kernel: swap_pager: indefinite wait buffer: bufobj: 0,
blkno: 1143, size: 16384
Oct 30 23:08:20 host last message repeated 2 times
Oct 30 23:08:20 host kernel: pid 791 (ntpd), uid 0, was killed: out of swap
space
Oct 30 23:08:20 host kernel: Oct 30 23:08:20 host kernel: pid 2532 (mysqld),
uid 88, was killed: out of swap space
Oct 30 23:08:20 host kernel: Oct 30 23:08:20 host kernel: pid 791 (ntpd), uid
0, was killed: out of swap space

...

[user at host /var/log]$ uname -a
FreeBSD host.domain.com 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27
08:16:24 UTC 2018    
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
[user at host /var/log]$ swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/md99          524288     5296   518992     1%
[user at host /var/log]$ uptime
11:13PM  up 8 mins, 2 users, load averages: 0.06, 0.11, 0.08
[user at host /var/log]$ sudo service ntpd start
Starting ntpd.
[user at host /var/log]$ sudo jail -c sql
sql: created
[user at host /var/log]$ swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/md99          524288    59912   464376    11%
[user at host /var/log]$ jls
   JID  IP Address      Hostname                      Path
     2  127.0.0.253     php.domain.com                 /usr/local/jails/php
     3  127.0.0.254     www.domain.com                 /usr/local/jails/www
     4  127.0.0.252     sql.domain.com                 /usr/local/jails/sql



Reboot 2

[user at host /var/log]$ sudo less all.log

Oct 30 23:16:50 host kernel: Starting jails:
Oct 30 23:16:52 host kernel:
Oct 30 23:16:52 host kernel: swap_pager: I/O error - pageout failed; blkno
445,size 131072, error 12
Oct 30 23:16:52 host kernel: swap_pager: I/O error - pageout failed; blkno
477,size 69632, error 12
Oct 30 23:16:52 host kernel: swap_pager: I/O error - pageout failed; blkno
494,size 8192, error 12
...
Oct 30 23:16:52 host kernel: swap_pager: I/O error - pageout failed; blkno
946,size 69632, error 12
Oct 30 23:16:52 host kernel: swap_pager: I/O error - pageout failed; blkno
963,size 36864, error 12
Oct 30 23:16:52 host kernel: swap_pager: I/O error - pageout failed; blkno
972,size 12288, error 12
Oct 30 23:18:12 host kernel: pid 2532 (mysqld), uid 88, was killed: out of swap
space
Oct 30 23:18:12 host kernel: Oct 30 23:18:12 host kernel: pid 2532 (mysqld),
uid 88, was killed: out of swap space

[user at host ~]$ uptime
11:18PM  up 2 mins, 1 user, load averages: 0.23, 0.25, 0.11
[user at host ~]$ jls
   JID  IP Address      Hostname                      Path
     2  127.0.0.253     php.domain.com                 /usr/local/jails/php
     3  127.0.0.254     www.domain.com                 /usr/local/jails/www
[user at host ~]$ swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/md99          524288     5752   518536     1%
[user at host ~]$ sudo jail -c sql
sql: created
[user at host ~]$ swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/md99          524288    54272   470016    10%

-- 
You are receiving this mail because:
You are the assignee for the bug.