CPU limit for Jails(patch for ULE scheduler)
Меньшиков Константин
kostjn at peterhost.ru
Wed May 20 15:37:52 UTC 2009
Меньшиков Константин wrote:
> Hello all!
> Many users want have limits on resourse for jail, for examle cpu and
> memory limit.
> I`m rewrire original cdjones patch for cpu limit for jail under ULE
> scheduler.
> So, this work simple.
Hi.
I`m rewrite jail limit patch under CURRENT.
New patch limited CPU, memory, filedesc, process.
And allow change limit on the fly
You can download tar.gz from
http://kostjn.spb.ru/patch-jail-limit-8CURRENT.tar.gz
===========================================================================
How to use.
===========================================================================
Build
cvsup CURRENT
cd /usr/src
patch -p0 < patch-jail-limit-8CURRENT
make buildkernel
make buildworld
make installkernel
reboot
make installworld
Create new entry in login.conf, for example class jail128
jail128:\
:cputime=10:\
:memoryuse=128M:\
:maxproc=256:\
:openfiles=1024:\
:tc=default:
Cputime is percent on 1 core.
Openfiles is sum filedesc for all proc in jail.
Create new jail.
...
Add in /etc/rc.conf
jail_test_flags="-Ljail128"
Run new jail
/etc/rc.d/jail start test
===========================================================================
Sysctl
===========================================================================
Added sysctl
[root at book ~]# sysctl security.jail.limit
security.jail.limit.enable: 1
security.jail.limit.memory_exceed_kill: 0
[root at book ~]# sysctl -d security.jail.limit
security.jail.limit: Jail limit
security.jail.limit.enable: Enable jail limit
security.jail.limit.memory_exceed_kill: Kill biggest proc in jail, if
jail excee
d memory limit
===========================================================================
Jset and Jget
===========================================================================
jset and jget is program for set new jail limit and get current limit
Example
[root at book ~]# cat /etc/rc.conf | grep jail2
jail_list="jail1 jail2 jail3 jail4 jail5 jail6 jail7 jail8 jail9 jail10"
jail_jail2_rootdir="/usr/jails/jail2/"
jail_jail2_hostname="jail2.book.pht"
jail_jail2_interface="re0"
jail_jail2_ip="192.168.200.22"
jail_jail2_flags="-Ljail64"
[root at book ~]# /etc/rc.d/jail start jail2
Configuring jails:.
Starting jails: jail2.book.pht.
[root at book ~]# cd ~kostjn/
[root at book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
[root at book /home/kostjn]# ./jset.o 1 jail2048
Set new jail limits, jid = 1
Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
[root at book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 30, MEM 2048M, NPROC 1024, NOFILE 2048
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
You see that new limit is set.
===========================================================================
Test
===========================================================================
Cpu limit
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Script
[root at book /home/kostjn]# cat test.sh
#!/bin/sh
for i in `jot 8 1`; do cpuset -l0 jexec 1 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 2 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 3 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 4 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 5 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 6 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 7 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 8 /a.out & done
for i in `jot 8 1`; do cpuset -l0 jexec 9 /a.out & done
cpuset -l0 jexec 10 /a.out &
Set class for all jail.
[root at book /home/kostjn]# for i in `jot 10 1`; do ./jset.o $i jail128 ;done
Set new jail limits, jid = 1
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 2
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 3
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 4
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 5
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 6
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 7
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 8
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 9
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
Set new jail limits, jid = 10
Limits: CPU 10, MEM 128M, NPROC 256, NOFILE 1024
[root at book /home/kostjn]# jexec 1 bash
[root at jail1 /]# cat cpu.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <math.h>
int main(int argc,char *argv[]){
int64_t i,j=0;
char *s;
for (;;){
}
}
Run test.sh
Result
top
last pid: 3513; load averages: 70.87, 37.58, 16.40 up 0+00:44:02
14:19:46
185 processes: 74 running, 111 sleeping
CPU: 49.9% user, 0.0% nice, 0.0% system, 0.2% interrupt, 49.9% idle
Mem: 139M Active, 24M Inact, 47M Wired, 192K Cache, 29M Buf, 1785M Free
Swap: 4044M Total, 4044M Free
PID JID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAN
3502 10 root 1 97 0 1480K 1244K CPU0 0 0:13 8.79%
a.out
3474 6 root 1 97 0 1480K 1244K RUN 0 0:04 4.69%
a.out
3431 2 root 1 96 0 1480K 1244K RUN 0 0:03 4.30%
a.out
3454 4 root 1 97 0 1480K 1244K RUN 0 0:03 4.05%
a.out
3422 1 root 1 96 0 1480K 1244K RUN 0 0:04 3.86%
a.out
3482 7 root 1 97 0 1480K 1244K RUN 0 0:03 3.86%
a.out
3447 3 root 1 97 0 1480K 1244K RUN 0 0:03 3.86%
a.out
3429 1 root 1 96 0 1480K 1244K RUN 0 0:03 3.66%
a.out
3485 8 root 1 97 0 1480K 1244K RUN 0 0:05 3.56%
a.out
3424 1 root 1 96 0 1480K 1244K RUN 0 0:04 3.56%
a.out
3464 5 root 1 97 0 1480K 1244K RUN 0 0:02 3.56%
a.out
3438 2 root 1 96 0 1480K 1244K RUN 0 0:03 3.47%
a.out
3494 9 root 1 96 0 1480K 1244K RUN 0 0:03 3.27%
a.out
3497 9 root 1 97 0 1480K 1244K RUN 0 0:05 3.17%
a.out
3433 2 root 1 96 0 1480K 1244K RUN 0 0:03 2.88%
a.out
3428 1 root 1 96 0 1480K 1244K RUN 0 0:02 2.88%
a.out
3487 8 root 1 97 0 1480K 1244K RUN 0 0:04 2.78%
a.out
ps auxwwww -ojid | more
root 3502 9.0 0.1 1480 1244 v2 RJ 2:15PM 0:07.40 /a.out
10
root 3476 4.4 0.1 1480 1244 v2 RJ 2:15PM 0:04.38 /a.out
7
root 3480 4.1 0.1 1480 1244 v2 RJ 2:15PM 0:03.02 /a.out
7
root 3498 3.9 0.1 1480 1244 v2 RJ 2:15PM 0:04.00 /a.out
9
root 3429 3.7 0.1 1480 1244 v2 RJ 2:15PM 0:01.38 /a.out
1
root 3487 3.6 0.1 1480 1244 v2 RJ 2:15PM 0:03.32 /a.out
8
root 3452 3.5 0.1 1480 1244 v2 RJ 2:15PM 0:01.37 /a.out
4
root 3463 3.5 0.1 1480 1244 v2 RJ 2:15PM 0:01.65 /a.out
5
root 3472 3.3 0.1 1480 1244 v2 RJ 2:15PM 0:02.63 /a.out
6
root 3437 3.2 0.1 1480 1244 v2 RJ 2:15PM 0:01.93 /a.out
2
root 3494 3.0 0.1 1480 1244 v2 RJ 2:15PM 0:02.92 /a.out
9
root 3500 3.0 0.1 1480 1244 v2 RJ 2:15PM 0:03.63 /a.out
9
We see that jail 10 (1 thread), used ~10 % cpu under heavy load.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Resourse compute
[root at book /home/kostjn]# ./jset.o 1 jail64
Set new jail limits, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
[root at book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 6M, NPROC 9, NOFILE 65
[root at book /home/kostjn]#
[root at book /home/kostjn]# jexec 1 bash
[root at jail1 /]# apachectl stop
/usr/local/sbin/apachectl stop: httpd stopped
[root at jail1 /]# exit
[root at book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Resource limit
[root at book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
[root at book /home/kostjn]# jexec 1 bash
[root at jail1 /]# cat mem.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <math.h>
int main(int argc,char *argv[]){
int64_t i,j=0;
char *s;
for (i=0; i < 1000 ;i++){
s = malloc(100000 * sizeof(char));
}
sleep(1000);
}
[root at jail1 /]# cc mem.c && ./a.out &
[1] 1320
[root at jail1 /]# ls
bash: fork: Cannot allocate memory
[root at jail1 /]# exit
[root at book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 1, MEM 103M, NPROC 5, NOFILE 31
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
We see that jail exceed memory limit. And new fork, mmap syscall not
permitted.
If you set sysctl
[root at book /home/kostjn]# sysctl security.jail.limit.memory_exceed_kill=1
security.jail.limit.memory_exceed_kill: 1 -> 1
[root at book /home/kostjn]# ./jget.o 1
Jail limits and rusage, jid = 1
Limits: CPU 5, MEM 64M, NPROC 128, NOFILE 512
Usage: CPU 0, MEM 3M, NPROC 3, NOFILE 24
[root at book /home/kostjn]# jexec 1 bash
[root at jail1 /]# ./a.out
Killed: 9
[root at jail1 /]# exit
[root at book /home/kostjn]# tail -n 1 /var/log/messages
May 20 14:10:17 book kernel: pid 1337 (a.out), uid 0, jid 1 was killed:
Prison e
xceed memory limit
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
If you attempt set nonexisten class, limit set to infinity.
[root at book /home/kostjn]# ./jset.o 1 jail123
Set new jail limits, jid = 1
Limits: CPU 9223372036854775807, MEM 20M, NPROC 9223372036854775807,
NOFILE 9223
372036854775807
===========================================================================
Problem
===========================================================================
If you have problem in this patch.
Add to kernel config
options KTR
options KTR_ENTRIES=1024
options
KTR_COMPILE=(KTR_PROC|KTR_JAIL|KTR_SCHED|KTR_RUNQ|KTR_LOCK|KTR_CONTENTIO N)
options KTR_MASK=KTR_JAIL
options KTR_CPUMASK=0x3
options KTR_VERBOSE
options PRINTF_BUFR_SIZE=128
Rebuild kernel.
Reboot.
Set sysctl
sysctl debug.ktr.mask=65536
and check /var/log/messages
More information about the freebsd-jail
mailing list