running out of mbufs?

[Marko Zec]

On Monday 08 August 2005 18:47, Andre Oppermann wrote:
> Marko Zec wrote:
> > On Monday 08 August 2005 12:32, Andre Oppermann wrote:
> > > Dave+Seddon wrote:
> > > > BTW, I'd be interested to know people's thoughts on multiple IP
> > > > stacks on FreeBSD.  It would be really cool to be able to give
> > > > a jail it's own IP stack bound to a VLAN interface.  It could
> > > > then be like a VRF on Cisco.
> > >
> > > There is a patch doing that for FreeBSD 4.x.  However while
> > > interesting it is not the way to go.  You don't want to have
> > > multiple parallel stacks but just multiple routing tables and
> > > interface groups one per jail. This gives you the same
> > > functionality as Cisco VRF but is far less intrusive to the
> > > kernel.
> >
> > Andre,
> >
> > the stack virtualization framework for 4.x is based precisely on
> > introducing multiple routing tables and interface groups.  In order
> > to cleanly implement support for multiple independent interface
> > groups, one has to touch both the link and network layers, not
> > forgetting the ARP stuff... and in no time you have ended up with a
> > huge and intrusive diff against the original network stack code.
>
> While your stack indexing approach is interesting I don't think it is
> the way we should take for the generic FreeBSD.  There are better
> ways to implement a less limiting superset of the desired
> functionality.

Andre,

there's no doubt almost any idea or particularly software can be 
improved.  Could you provide a more elaborate argumentation to your 
claim the network stack cloning concept is so severely limited that it 
has no place to search for in the future of FreeBSD?  And what exactly 
did you mean by a "stack indexing approach"?

> And the ARP is almost done, I have to review the code 
> and then it goes into -current.

While having a per-interface ARP logic is certainly a nice feature, this 
alone will not solve much with regards to introducing multiple 
independent interface groups.  You will still most probably have to 
revisit the ARP code once you start introducing non-global interface 
lists in the kernel.

> > So I see no point in pretending we could get such a functionality
> > for free, i.e. with only a negligible intrusiveness to the kernel
> > code.  A more appropriate question would be whether the potential
> > benefits of having multiple stack state instances could outweight
> > the trouble and damage associated with the scope of required
> > modifications to the kernel code tree.  Only if we could get an
> > affirmative answer to that question it would make sense to start
> > thinking / debating on the most appropriate methodology to
> > (re)implement the multiple stacks framework.
>
> Having multiple stacks duplicates a lot of structures for each stack
> which don't have to be duplicated.  With your approach you need a new
> jail for every new stack.  In each jail you have to run a new
> instance of a routing daemon (if you do routing).  And it precludes
> having one routing daemon managing multiple routing tables.  While
> removing one limitation you create some new ones in addition to the
> complexity.

Bemusingly, none of the above claims are true.  

A new jail for each network stack instance is NOT required.  Inside the 
kernel what could be considered "per-jail" and per-network stack 
structures are cleanly separated and independent.  In fact, one can run 
multiple jails bound to a single network stack instance, if desired.

Furthermore, a single process can simultaneously attach to multiple 
network stacks, thus potentially allowing a single routing daemon to 
manage multiple separated routing tables and interface groups.  The 
entity that gets permanently bound to a network stack instance is a 
socket and not a process. This translates to the capability of a single 
process to open multiple sockets in multiple independent stacks.  IMO, 
one particular strength of such an approach is that it requires 
absolutely no extensions or modifications to the existing routing 
socket API.

And finally, I'm wondering what structures exactly are you referring to 
when you say that this approach "duplicates a lot of structures for 
each stack which don't have to be duplicated"?  I absolutely agree that 
the virtualization of the network stack should be done as simple and 
non-intrusive as possible, but my point is that it just cannot be done 
cleanly / properly without taking some sacrifices in terms of the scope 
of minimum required modifications.

Cheers,

Marko

> What I have in mind is a redesign of some parts of the stack as such.
> I have got funding to work on this through my fundraise of two weeks
> ago.  Work is about to start later this week.  I will provide a more
> detailed design document for discussion and review by the FreeBSD
> community in a few days.  It will include features such as multiple
> hierarchical routing tables (bound to jails or not), interface
> groups, virtual interfaces belonging to one or more interface groups,
> policy routing (based on source address/interface), multi-path
> routing, a reworked routing-socket and some more stuff.
>
> I recommend waiting for the design document for further discussions
> and jumping to any conclusions. :)