Extending FIBs to support multi-tenancy

Patrick Kelsey pkelsey at freebsd.org
Sat Dec 19 01:25:36 UTC 2015

On Fri, Dec 18, 2015 at 5:26 PM, Ryan Stone <rysto32 at gmail.com> wrote:

> My employer is going through the process of extending our product to
> support multi-tenant networking.  The details of what are product does
> isn't really relevant to the discussion -- it's enough to know that we have
> a number of daemons acting as servers for various network protocols.
> Multi-tenacy, as we've defined the feature, imposes the requirement that
> our network services be able to communicate with clients from completely
> independent networks. This has imposed the following new requirements on
> us:
> - different tenant networks may have different DNS servers
> - they may use different AAA mechanisms (e.g. one tenant uses LDAP, another
> uses Kerberos, and a third also uses LDAP but a different server)
> - they may use independent routing tables
> - different tenant networks may use overlapping IP ranges (so we might see
> two different clients from two different tenant networks with the IP
>, for instance)
> - traffic from different tenant networks is not guaranteed to be segregated
> in any way -- it might all come in the same network interface, without any
> vlan tagging or any other encapsulation that might differentiate tenant
> networks
> - we need to scale to thousands of tenant networks
> - we will impose the requirement that our system can't be assigned the same
> IP address for different tenant networks
> Our intention is to use the destination IP address of incoming packets to
> determine which tenant network the packet came from (hence the requirement
> of not allowing the same IP to be configured for different tenant
> networks)..
> The obvious tool for meeting these requirements in FreeBSD is VIMAGE.
> However, we have prototyped that approach already, and I have been told
> that we discovered that we found this will not scale to thousands of
> networks.  I don't have all of the details as to why, but the root of the
> problem is that any given process can only be associated with a single vnet
> instance.  In our current architecture, we can't have thousands of
> instances of each network service running (and I'm not sure that we really
> could: if we support A services, B tenant networks and C CPU cores, we
> would need a minimum of A * B * C threads to ensure that any given service
> on any single tenant network could fully utilize the system's resources to
> process requests).
> We're instead looking at using FIBs to implement the routing table
> requirement.  To meet our requirements, we're expecting to have to make
> three important chances to how FIBs are managed:
> 1) Allow listening sockets to be wildcarded across FIBs
> 2) Make FIBs a property of an interface address, not the interface
> 3) Allow each thread to set a default FIB that will be applied to newly
> created sockets
> 1)
> We don't really want to change all of our services to instantiate one
> listening socket for every tenant network.  Instead we're looking at
> implementing (and upstreaming) a kernel extension that allows a listening
> socket to be wildcarded across all FIBs (note: yesterday I described this
> feature as allowing us to pick-and-choose FIBs, but people internally have
> convinced me that a wildcard match would make their lives significantly
> easier).  When a new connection attempt to a listening socket in this mode
> is accepted, the socket would not inherit its FIB from the listening
> socket.  Instead, it would be set based on the local IP address of the
> connection.
> 2)
> Currently, FIBs are a property of an interface (struct ifnet).  We aren't
> very enthusiastic about the prospect of having to create thousands of
> interfaces to support thousands of network interfaces.  We would instead
> like to make the FIB a property of the interface address. For backwards
> compatibility reasons I would still let admins set a FIB on an ifnet, but
> instead that would be the default FIB assigned to addresses that aren't
> explicitly assigned a FIB.  That should maintain the current behaviour
> while making it easy to push FIBs down into the address.
> 3)
> The idea of a per-thread FIB has gotten the most pushback so far, and I
> understand the objection.  I'll explain the problem that we're trying to
> solve with this.  When a new request comes in, we may need to perform
> authentication through LDAP or Kerberos.  The problem is that the existing
> open-source implementations that we are using manage sockets directly.  We
> really don't want to have to go through them and make their APIs entirely
> FIB-aware -- that is far too much churn.  By moving awareness of the
> current FIB into the kernel, existing calls to socket() can do the right
> thing transparently.
> We're not entirely happy with the solution, but the "right" way to solve
> the problem involves rototilling a number of libraries.  Even if we could
> convince the upstream projects to take patches, it's far more work than
> we're willing to take on.
Why not address (3) by creating your own wrappers of the socket library
calls whose behavior you want to change (see --wrap in ld(1))?  Your
wrapper(s) could then implement whatever application-specific FIB selection
policy you'd like, for example, checking for a particular thread local
storage key whose value indicates the FIB to use, and if defined setting
the FIB ahead of invoking the original socket library call.

More information about the freebsd-transport mailing list