limitations on jail style virtualization

[Julian Elischer]

On 11/13/10 8:27 PM, Bjoern A. Zeeb wrote:
> On Sat, 13 Nov 2010, Julian Elischer wrote:
>
>> On 11/13/10 2:13 PM, Julian Elischer wrote:
>>> On 11/13/10 1:55 PM, Brandon Gooch wrote:
>>>> On Sat, Nov 13, 2010 at 2:59 PM, Julian 
>>>> Elischer<julian at freebsd.org> wrote:
>>>>
>>>> Was this brought up in any of the discussions?
>>>>
>>>> http://www.7he.at/freebsd/vps/
>>>
>>> no it was not brought up..
>>> it was an unofficial non-planned discussion that errupted pretty much
>>> spontaneously an a couple of couches so it was not prepared in 
>>> advance..
>>>
>>> Since I don't know much about that project I didn't bring it up.
>>> However I should take the time to read it..
>>> thanks for bringing it up!
>>
>> replying to myself:
>>
>> Now that I have looked at the reference above I realise that I did 
>> look
>> at it before but during a particularly hectic pereiod of my life so 
>> I didn't
>> really take it in as much as I should have.
>>
>> I really do like this and I seen no reason why it can not be all 
>> linked together.
>> Since I last heard about it, it appears that they have advanced 
>> somewhat.
>> They are already integrating the  VIMAGE code into their project, 
>> and they have
>> extended the VIMAGE mechanism to suite their own purposes.
>>
>> It would be nice if they would show up in -virtualization sometimes 
>> so we could
>> include them into our discussions.
>>
>>>
>>>> I'm not sure if the VPS project pertains directly to what you're
>>>> talking about, but perhaps some of the code or ideas from the 
>>>> project
>>>> might?
>>>>
>>>> Even if it doesn't, it's still an exciting project that adds a 
>>>> ton of
>>>> value to FreeBSD's light-weight virtualization strategy. What do 
>>>> think
>>>> about the VPS concept in relation to the current virtualization 
>>>> effort
>>>> being put in to jails? It seems to me that recent efforts at
>>>> virtualizing kernel-level objects makes VPS the future of FreeBSD's
>>>> virtualization, leaving jails as a great way to isolate
>>>> applications...
>>>
>>> it the approaches are compatible I see no reason to not combine
>>> but I'll know more when I have read it.
>>
>> the approaches are different but not necessarily incompatible
>> In particular I'd like to hear what Jamie (James Gritton)
>> thinks about it, as he has most of the jail direction in his head :-)
>>
>> I'd certainly like to be able do do many of the things they hope to 
>> do (or have a start at).
>
> we had Klaus at the DevSummit in Karlsruhe and he gave a talk at
> EuroBSDCon as well.  See the wiki and the 2010.eurobsdcon.org web
> page (thought the material should be on his weg page as well).
>
> So, yes, we are talking to him, even though I got busy the last weeks.
> As he's piggybacking on VNET/VIMAGE and there'll me more things he and
> we'll do, there's certainly code going to be shared (I would hope). \

this sort of dovetails into something I've been thinking about for a 
while,
which is NUMA support.

Now, you may well ask "what has this to do with NUMA?" so I will explain.

It seems to me that as the speed limits for single CPUS seem to be 
getting reached,
we are seeing the transfer from speed to breadth. i.e more and more CPUs.
We will shortly see the intruduction of systems that support not 2 or 
4 threads
of operation  but thousands of threads of operation. We are already 
seeing some early
implementations of this. A single image OS will have great trouble 
handling this
sort of system, but the people who are hoping to make it all work are 
teh people like
VMWARE.  They will take systems with thousands of threads and break 
them up
in a more-or less dynamic manner, keeping the NUMA layout of the 
system in mind.

I have been wondering if we might not be able to do something similar.
where we one one BSD system which overall may not be that efficient, 
but break
it up along the NUMA fault lines  using something like VPS to present 
a whole bunch of
independent systems that can run very efficiently (on local NUMA RAM)
an yet be 'tightly coupled' by the fact that they are running in one 
overall system
with full cross connectivity (even though it be slightly less 
efficient than the NUMA
couplings).

We could, scale our systems up to tens of thousands of CPUS if we were 
willing
to break them up to a large number of 'jails'/VPSs, each with only 32 
or so threads
(or whatever the hardware effieciently supports). Then the overall system
could be responsible mostly for inter-system communications and 
housekeeping.

Duplicated OS images and seaprated data ranges might all come together
to form this "cluster of systems in a single OS".

Anyhow, that the way I see things progressing and the OS that can
harness all these CPUs in a useful way will have a lot if demand.
whether it will be ESX+OS-de-jour or Linux++ or ClusterBSD will remain 
to be seen.

Julian

>
> /bz
>