[Bug 235748] Using swap with free memory available

Thu Feb 14 17:42:41 UTC 2019

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235748

            Bug ID: 235748
           Summary: Using swap with free memory available
           Product: Base System
           Version: 11.2-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: info at murrawhip.net

Since updating to FreeBSD 11.2 from 11.1 I've been experiencing swap usage
growing consistently over several weeks until there's none left, despite having
free ram available. I detailed the issue on one of the smaller servers here:
https://serverfault.com/questions/953819/freebsd-swap-usage-with-mysql

I have a larger server exhibiting the same issue but it's at the point where
it's logging the following 20-40 times in a second, every couple minutes:
Feb 14 09:31:48 rad-myecr kernel: swap_pager_getswapspace(6): failed

MySQL is the process using the memory, and its data directory is on a zvol with
 vfs.zfs.arc_max="256M"

root at rad-myecr:/var/log # vmstat
procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr da0 da1   in    sy    cs us sy id
2 0 1 2.1T  434M   850 886  10   5   569 1103   0   0  827   759   320  2  1 97

root at rad-myecr:/var/log # zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH 
ALTROOT
data  1.12T   345G   799G        -         -    70%    30%  1.00x  ONLINE  -

last pid: 75850;  load averages:  1.80,  1.71,  1.54                           
             up 42+15:02:10  09:38:24
29 processes:  1 running, 28 sleeping
CPU:  2.6% user,  0.0% nice,  1.5% system,  0.0% interrupt, 96.0% idle
Mem: 17G Active, 9096M Inact, 2219M Laundry, 3295M Wired, 1551M Buf, 347M Free
ARC: 252M Total, 110M MFU, 31M MRU, 1380K Anon, 4992K Header, 104M Other
     30M Compressed, 198M Uncompressed, 6.54:1 Ratio
Swap: 20G Total, 16G Used, 3638M Free, 82% Inuse, 20K In

last pid: 75910;  load averages:  1.85,  1.76,  1.58                           
             up 42+15:03:57  09:40:11
29 processes:  1 running, 28 sleeping
CPU:  6.0% user,  0.0% nice,  1.1% system,  0.1% interrupt, 92.8% idle
Mem: 17G Active, 8662M Inact, 2218M Laundry, 3294M Wired, 1551M Buf, 772M Free
ARC: 253M Total, 110M MFU, 37M MRU, 1008K Anon, 4984K Header, 100M Other
     34M Compressed, 202M Uncompressed, 5.87:1 Ratio
Swap: 20G Total, 16G Used, 3642M Free, 82% Inuse, 152K In

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
10914 mysql        43  22    0 63717M 27468M select  4 487.4H 148.46% mysqld
75880 root          1  20    0  7920K  3628K CPU16  16   0:00   0.08% top
  125 root          2  20    0 32472K  4520K select  4  46:58   0.07% vmtoolsd
  452 root          1  20    0  9184K   696K select 12   4:41   0.02% devd
36456 root          1  28    0   534M   408M nanslp  3   5:29   0.02% php
75614 dfairley      1  20    0 13164K  8084K select  2   0:00   0.02% sshd
33400 root          1  23    0   255M 37376K select  9  13:15   0.02% snmpd
  619 root          1  20    0  6416K  2008K select  7   3:04   0.00% syslogd
  709 root          1  20    0 12456K 12552K select 15   3:51   0.00% ntpd
 1273 root          1  20    0 10456K  5092K select 19   1:02   0.00% sendmail
 1280 root          1  20    0  6468K   768K nanslp  0   0:14   0.00% cron
  401 root          1  42    0  6548K  1912K select  7   0:02   0.00% dhclient
 1276 smmsp         1  20    0 10456K   876K pause  17   0:01   0.00% sendmail
  447 _dhcp         1  20    0  6548K  2004K select 21   0:01   0.00% dhclient
 1270 root          1  20    0 12852K  5264K select 13   0:00   0.00% sshd
75619 root          1  20    0  7416K  4336K pause  20   0:00   0.00% csh
10428 mysql         1  52    0  7064K     0K wait   18   0:00   0.00% <sh>
75615 dfairley      1  20    0  7416K  4360K pause   6   0:00   0.00% csh
75612 root          1  23    0 13164K  7964K select 22   0:00   0.00% sshd
75617 root          1  22    0  7612K  3832K select  9   0:00   0.00% sudo
75618 root          1  24    0  6936K  3104K wait    1   0:00   0.00% su
 1335 root          1  52    0  6408K  1796K ttyin   2   0:00   0.00% getty
 1333 root          1  52    0  6408K  1796K ttyin   5   0:00   0.00% getty
 1334 root          1  52    0  6408K  1796K ttyin   0   0:00   0.00% getty
 1336 root          1  52    0  6408K  1796K ttyin  12   0:00   0.00% getty
 1332 root          1  52    0  6408K  1796K ttyin  11   0:00   0.00% getty
 1331 root          1  52    0  6408K  1796K ttyin   3   0:00   0.00% getty
 1329 root          1  52    0  6408K  1796K ttyin   7   0:00   0.00% getty
 1330 root          1  52    0  6408K  1796K ttyin  21   0:00   0.00% getty

-- 
You are receiving this mail because:
You are the assignee for the bug.