Performance problems with pagedaemon

[Artem Belevich]

On Mon, Jan 2, 2012 at 5:41 AM, Victor Balada Diaz <victor at bsdes.net> wrote:
...
> System wide totals computed every five seconds: (values in kilobytes)
> ===============================================
> Processes:              (RUNQ: 2 Disk Wait: 0 Page Wait: 0 Sleep: 51)
> Virtual Memory:         (Total: 1098017100K, Active 24065448K)
> Real Memory:            (Total: 21157668K Active 20971144K)
> Shared Virtual Memory:  (Total: 27740K Active: 8632K)
> Shared Real Memory:     (Total: 6128K Active: 4928K)
> Free Memory Pages:      315636K

On a system with 24GB of RAM you seem to have almost all of it active.
It appears that you're simply on the edge of running out of memory and
thus page daemon wakes up constantly trying to find more pages...

...
> Top:
>
> last pid: 24777;  load averages:  3.26,  4.07,  4.49                                                                                                             up 34+19:43:58  14:32:05
> 66 processes:  5 running, 61 sleeping
> CPU:  0.2% user,  0.0% nice, 37.6% system,  0.0% interrupt, 62.2% idle
> Mem: 18G Active, 1908M Inact, 3008M Wired, 73M Cache, 2465M Buf, 232M Free
> Swap: 4096M Total, 4096M Free
>
>  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
>  1059 mysql      22  44    0 21004M 19741M ucond   3  71.2H 124.51% mysqld

Mysql uses more than 20G of RAM. You may want to tune it down a bit so
that there is a bit of free RAM around.

Page daemon is trying to maintain v_free_target + v_cache_min.

>vm.v_free_target: 161771
>vm.v_cache_min: 161771

In your case that would be about 1.2GB. If 'v_free_count +
v_cache_count' are below that page daemon will periodically wake up
and will start scanning active/inactive lists trying to find the pages
it could use. In your case, when most of the memory is in active use,
page daemon's job will be almost pointless and would just waste CPU
time.

On large memory systems default tuning for v_free/cache_min/target is
probably somewhat conservative. You may try setting them somewhat
lower via sysctl and see if you can find an equilibrium with mysql
happy, pagedaemon sleeping and the system up and running. The danger
of tuning these parameters too low is that is you don't have enough
memory available for allocation without having to sleep, things will
start falling apart and will eventually hang or crash your box.

--Artem