[SoC] Jail Resource Limits

[Chris Jones]

On 21-Aug-06, at 9:37 AM, Remko Lodder wrote:
> Before my little question, i would like to take the opportunity to
> thank you for the hardwork, this is a much requested feature and
> will greatly appriciated by the crowd :-) Cheers for that!.

Thanks!  I do some web hosting with jails on the side, and've often  
wished this sort of thing were around.  I suppose necessity truly is  
the mother of invention. ;)

> My little question: How does this impact on performance? I noticed
> all your p4 commits and saw some pieces of code that constantly
> monitor the usage of the jail (wrt CPU and Memory); doesn't that
> reduce our performance? Is that performance impact (if any) also
> growing when using more jails?

The changes can be broken out into three major groups: the memory- 
limiter (kern_jail:jpager_td), the CPU usage monitor  
(sched_4bsd:schedcpu), and the process priority resetter  
(sched_4bsd:resetpriority).

The memory-limiter essentially idles if the sysctl for memory limits  
is turned off or the jail doesn't have a memory limit; otherwise, we  
need to go over all processes and check whether they're in the jail  
and sum their RSS (in prison_memory).  This could really use having a  
quick way of getting a list of processes in a jail (perhaps a new  
entry in the proc struct for next-process-in-this-jail?).

If we're over the limit, we then have to go over all processes again  
and check whether they're in the jail and amenable to being pushed  
out.  If so, we then go and push them out in the same way that the  
pagedaemon does.  This bit's going to be relatively slow, and also  
could use that quick get-processes-in-jail method.

The CPU usage monitor keeps track of the number of CPU usage shares  
allocated in schedcpu_thread, which runs every second and then calls  
schedcpu (see below).  This one's linear with the number of jails and  
again could probably benefit from a quick check to see whether we're  
enforcing CPU shares at all.  An even better method would be to just  
recalculate the total outstanding shares when the jail or  
jail_set_resource_limits syscalls are called or the sysctl for  
unjailed shares changes, rather than all the time.  The amount of CPU  
time each KSE uses is already collected by the system, so we just re- 
use that.

schedcpu needs to lock the allprison mutex and reset each prison's  
estimated CPU time at line 466, so that'll be about linear with the  
number of jails.  Hmm, looking at it, there's a really obvious  
optimization that I forgot to put in (skip the lock & reset if we  
have no jails), which I'll want to add.

Next, it will increment the CPU time if the sysctl is set and the  
process is jailed.  We're looking at about eight loads if the  
compiler's smart, three compares and two ors, an if, and maybe an  
add.  Unfortunately, this time's linear with the number of processes.

The big hit comes in the process priority resetter, which runs when  
we consider what priority to run a process at (this runs  
intermittently for each active process).  If the sysctl is set and  
the process is jailed, we do some reasonably hefty (some multiplies)  
calculations to change it.  This'll be roughly linear with the number  
of jailed processes.

Overall, I don't think there'll be a noticeable performance impact  
for non-pathological numbers of processes or jails, but I don't have  
any stats to back that up.

> I think these things are simple, but important to know; I do not
> have a full featured setup (sadly) to measure the performance
> impact of this, If you are able to find someone, that would be
> really great imo.

Consider this a general appeal for anyone who can help me do some  
analysis on the performance testing, or give me some pointers on how  
to collect the relevant data. :)

Cheers,

Chris