swapspace grows with no return

[Doug Denault]

This is a smallish server, 8G memory, normally running 8 jails. When this 
issue started I stopped three of the jails which were used for testing and 
development. What's left are jails running apache 2.4, wordpress, and MySQL 
5.7. One jail, camden is testing a new version of squirrelmail and 
roundcube.

>From swapinfo and top:

Device          1K-blocks     Used    Avail Capacity
/dev/aacd0p3      4194304   831572  3362732    20%

[ 0, 2010408]   root
[13, 3359976]   camden
[15, 5039344]   bassharbor
[14, 5201500]   monhegan
[12, 5776608]   newharbor
[ 4, 6870432]   pemaquid

total:  28258268

So about 3% of the virtual storage allocated is currently written to the 
swap file. Two times in the last several weeks the swap file has run out of 
space. This is Sunday, there is as close to zero activity as these guys 
get. The 20% will only grow from here.

I ran vmstat over night; there were about 10 pageins and no pageouts. top 
and 'systat vmstat' also show no paging. The swap file when from 18% to 20% 
over that time frame. The obvious question is: how come?

sysctl swap counts:

vm.swap_enabled: 1
vm.disable_swapspace_pageouts: 0
vm.swap_idle_enabled: 0
vm.stats.vm.v_swappgsout: 631602
vm.stats.vm.v_swappgsin: 46818
vm.stats.vm.v_swapout: 53454
vm.stats.vm.v_swapin: 9842
vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
vm.nswapdev: 1
vm.swap_async_max: 4
vm.swap_maxpages: 32444512
vm.swap_reserved: 30920429568
vm.swap_total: 4294967296

Seems ok to me. We had to reboot about five days ago. Twice since then we 
just rebooted the offending jail.

Past what is going on: The default installation will allocate a 4G 
swapfile. I am pretty sure I read from a FreeBSD source that the 2 x 
physical memory rule is not needed. Several threads on questions suggest 
otherwise. What is the 'best practice' here?

I also read somewhere that the VM system will preactively pageout changed 
pages. That would seem to be the case here, but they are never freed. 
Lastly, can an application lock memory?

Thanks for any suggestions,
Doug

_____
Douglas Denault
http://www.safeport.com
doug at safeport.com
Voice: 301-217-9220
   Fax: 301-217-9277