Stack virtualization (was: running out of mbufs?)

[Marko Zec]

On Tuesday 09 August 2005 14:41, Andre Oppermann wrote:
> Marko Zec wrote:
> > On Monday 08 August 2005 18:47, Andre Oppermann wrote:
> > > Marko Zec wrote:
> > > > On Monday 08 August 2005 12:32, Andre Oppermann wrote:
...
> > > > > There is a patch doing that for FreeBSD 4.x.  However while
> > > > > interesting it is not the way to go.  You don't want to have
> > > > > multiple parallel stacks but just multiple routing tables and
> > > > > interface groups one per jail. This gives you the same
          ^^^^^^^^^^^^^^^^
> > > > > functionality as Cisco VRF but is far less intrusive to the
> > > > > kernel.
...
> I don't want to have non-global interface lists in the kernel.

But sooner or later you _will_ end up with some sort of non-global 
interface lists after all, just as you stated yourself at the beginning 
of this tread.  Of course one can still maintain all interfaces linked 
in one list and introduce another set of separated lists on per-stack 
basis which will be used to logically group interfaces into smaller 
sets, but that's really just a question of coding / design style.

...
> > > Having multiple stacks duplicates a lot of structures for each
> > > stack which don't have to be duplicated.  With your approach you
> > > need a new jail for every new stack.  In each jail you have to
> > > run a new instance of a routing daemon (if you do routing).  And
> > > it precludes having one routing daemon managing multiple routing
> > > tables.  While removing one limitation you create some new ones
> > > in addition to the complexity.
> >
> > Bemusingly, none of the above claims are true.
> >
> > A new jail for each network stack instance is NOT required.  Inside
> > the kernel what could be considered "per-jail" and per-network
> > stack structures are cleanly separated and independent.  In fact,
> > one can run multiple jails bound to a single network stack
> > instance, if desired.
>
> Ok.
>
> > Furthermore, a single process can simultaneously attach to multiple
> > network stacks, thus potentially allowing a single routing daemon
> > to manage multiple separated routing tables and interface groups. 
> > The entity that gets permanently bound to a network stack instance
> > is a socket and not a process. This translates to the capability of
> > a single process to open multiple sockets in multiple independent
> > stacks.  IMO, one particular strength of such an approach is that
> > it requires absolutely no extensions or modifications to the
> > existing routing socket API.
>
> The existing API should be modified, it is pretty out of date.
>
> > And finally, I'm wondering what structures exactly are you
> > referring to when you say that this approach "duplicates a lot of
> > structures for each stack which don't have to be duplicated"?  I
> > absolutely agree that the virtualization of the network stack
> > should be done as simple and non-intrusive as possible, but my
> > point is that it just cannot be done cleanly / properly without
> > taking some sacrifices in terms of the scope of minimum required
> > modifications.
>
> Multiple interface lists, vm zones, etc. as your FAQ spells out.

Multiple interface lists are a must, whether as a replacement (the way I 
did it) or as a supplement to a global interface list.  They cost 
nothing in terms of memory use, and greatly simplify the code and 
prevent potential performance and cross-stack-boundary-leaking 
pitfails.

For a long time my framework does _not_ use separate VM zones per 
network stack instance for storing PCBs.  True, the FAQ should probably 
be updated, but it already clearly stated my doubts whether separate VM 
zones were really needed, and the later experiments and working code 
proved they indeed weren't.  What still uses multiple VM zones is the 
TCP syncache code, and I agree it could most likely be reworked to use 
only a single global zone.

Any other offending structures? :-)

It looks like we might be converging in terms of what it takes to 
virtualize a network stack :-)

> Again, I think we are talking past each other right now and we have
> different solutions to different problem sets in mind (or already
> coded).  When I have my paper finished my vision and intentions
> should be more clear and then we can have the discussion on the
> merits of each approach and whether parts of each are complementary
> or converse.

OK, looking forward to it...

Cheers,

Marko