Unresponsive system, ZFS ARK forcing proceses into swap

[James Morris]

Hi,

I have a freebsd 10.3 zfs server running with multiple zfs jails.

I run samba, syslog-ng, a web server and a few assorted other services in each jail.

The machine is not used much, but I have noticed that I can not ssh into the base system or reach a web server that I run on the base system after a few days of running.

On connecting via the console I notice that arc is taking up a bulk of the ram and it seems processes like sshd and nginx are frozen.

Also the swap space is utilised and the ARC is taking up the majority of my RAM.

When I try and run an interactive command like top, less, ... It loads, but the system takes exceedingly long 10min+ to close these.

The same symptoms are true when I try to use ^c to quit a running process like ping.

Any help triaging this issue will be much appreciated.

I believe that the ZFS ARC has something to do with it, but I do not want to turn it off altogether.

Is there a way to make sure ZFS does not push my other processes (sshd, nginx, ...) into swap where they are unresponsive.

Output of top while in this state is below

last pid: 31465;  load averages:  0.43,  0.56,  0.54  up 1+08:18:48    13:59:44
319 processes: 1 running, 308 sleeping, 1 stopped, 3 zombie, 6 waiting

Mem: 82M Active, 336K Inact, 15G Wired, 76M Cache, 12M Free
ARC: 12G Total, 3503M MFU, 26M MRU, 13M Anon, 22M Header, 8572M Other
Swap: 2048M Total, 1064M Used, 984M Free, 51% Inuse

Thanks in advance,

James