FreeBSD route tables limited 16?

[Dave Seddon]

Greetings,

Thanks for the quick response.

It sounds like dedicating some space for this in the mbuf would be the
best way forward, but the question is how much.  I'm worried that most
freebsd users won't go for lots of route tables, which is why you went
for 4 bits originally.

Within the network service provider space there is frequently a
requirement for lots of virtual-routing with MPLS.  I imagine there are
others in my situation, including vendors and people working on
equipment like Cisco/Juniper/Lucatel.

Regarding the size to dedicate, the best number might be 12 bits or
4096.  This would allow a route table per VLAN on a 802.1q interface.
(Actually I'm lying a little because the first and last vlan IDs aren't
usable :) ).

Perhaps a separate option for non-common users who want many route
tables would be best.  e.g.

GIANT_ROUTETABLES=12

Seems like there would need to be changes in multiple places although
perhaps this list isn't exhaustive.  So far the files to edit are:
/usr/src/sys/net/route.h
/sys/sys/mbuf.h


Regarding firewalls and these multiple route tables, have you considered
having a separate firewall rule table per route table?


I haven't looked at the vnet jails, yet.  Will do.  Thanks.

Kind regards,
Dave

-----Original Message-----
From: Julian Elischer <julian at elischer.org>
To: dave at seddon.ca
Cc: Andrew Hannam <andrewh at itsallaboutbiz.com>, FreeBSD Net
<net at FreeBSD.org>, Robert Watson <rwatson at FreeBSD.org>
Subject: Re: FreeBSD route tables limited 16?
Date: Mon, 13 Sep 2010 17:56:37 -0700
Mailer: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.9)
Gecko/20100825 Thunderbird/3.1.3

On 9/13/10 5:18 PM, Dave Seddon wrote:
> Greetings Julian,
>
> I've been wondering if it's possible to increase the number of FreeBSD
> route tables to a larger number.  It seems this is currently 4 bits,
> however I was wondering about perhaps 16 bits?


Yes the code is designed to handle many more and if you do
create more then everything SHOULD handle it.
The bottleneck is that we need to store an associated fib with
each outgoing (or for that matter incoming) packet, bit we do not at
this time want to dedicate a whole word in the mbuf to the task.
My "hack" for 8.x (before it was done) was to hide the information
in the flags word of the mbuf.
I only took 4 bits to make sure I didn't trample on other
people's use of bits there. The plan is/was to make a separate
entry in the mbuf some time after 7.x branched (say, "now" for
example :-)  )
you could just steal more bits for now, but if you take 8 bits
there will only be one spare.

(see /sys/sys/mbuf.h)

It may just be time to bite the bullet and steal the entry.

Out of curiosity, why do you need > 16 fibs?

have you considered using vnet jails a well?



>
> /* MRT compile-time constants */
> #ifdef _KERNEL
>   #ifndef ROUTETABLES
>    #define RT_NUMFIBS 1
>    #define RT_MAXFIBS 1
>   #else
>    /* while we use 4 bits in the mbuf flags, we are limited to 16 */
>    #define RT_MAXFIBS 16
>    #if ROUTETABLES>  RT_MAXFIBS
>     #define RT_NUMFIBS RT_MAXFIBS
>     #error "ROUTETABLES defined too big"
>    #else
>     #if ROUTETABLES == 0
>      #define RT_NUMFIBS 1
>     #else
>      #define RT_NUMFIBS ROUTETABLES
>     #endif
>    #endif
>   #endif
> #endif
>
> Really liked your announcement years ago:
> http://lists.freebsd.org/pipermail/freebsd-arch/2007-December/007331.html
>
> Kind regards,
> Dave Seddon
> +61 447 SEDDON
> dave at seddon.ca
>
> -----Original Message-----
> From: Andrew Hannam<andrewh at itsallaboutbiz.com>
> To: dave at seddon.ca
> Subject: RE: FreeBSD route tables - limited to 16 :(
> Date: Mon, 13 Sep 2010 15:55:47 +1000
> Mailer: Microsoft Office Outlook 12.0
>
> I think the gentleman is confusing route-tables with routes.
> 150K routes is easily possible but it is obvious there is currently only support for up to 16 route tables.
>
> I think that you are right and the number of bits will need to be updated.
>
> I don't know the answer to the 'route leaking' question and it has been a long time since I looked at this code.
>
> You really need to speaking the specialist responsible for the multiple route table code. This person should be clearly marked in the code headers.
>
> I'm guessing that no-one has thought about using it the way you are planning to use it.
>
> If I get some time I will have a look - but don't hold your breath.
>
> Regards,
> Andrew.
>
> -----Original Message-----
> From: Dave Seddon [mailto:dave at seddon.ca]
> Sent: Saturday, 11 September 2010 12:52 AM
> To: Aldous, Matthew D
> Cc: dave at seddon.ca; Andrew Hannam; Truman Boyes
> Subject: RE: FreeBSD route tables - limited to 16 :(
>
> Greetings,
>
> I'm guessing we need to adjust the number of bits defined for the route
> table in the mbufs structure definition (where ever that is), then we
> can update the route.h to match.
>
> I guess really we should make the mbufs codes _and_ route.h code pickup
> the KERNCONF definition of the variable ROUTETABLES.
>
> Andrew - thoughts on this?
>
> I'm not sure if the firewall rules allow you to update the route table
> variable in the mbuf, but if it doesn't we should allow this.  This
> would be equivelant to what they call 'route leaking' in MPLS speak,
> when you can pop traffic from one VPN to another (very nasty, but
> sometimes handy).

yes ipfw does allow you to do this but it needs some more work..
It only really works as the naive user may expect on incoming packets.

>
> Regards,
> Dave
>
> On Fri, 2010-09-10 at 19:05 +1000, Aldous, Matthew D wrote:

>> ________________________________
>> From: Dave Seddon [dave at seddon.ca]
>> Sent: Friday, 10 September 2010 6:36 PM
>> To: Andrew Hannam
>> Cc: dave at seddon.ca; Aldous, Matthew D; Truman Boyes
>> Subject: FreeBSD route tables - limited to 16 :(
>>
>> I just tried compiling up FreeBSD 8.1 with 1024 route tables.  It's
>> throwing an error, which is tracked down to the
>> vi /usr/src/sys/net/route.h (line 99ish).  The limit is 16, because as
>> the comments say this is 4 bits.  Need to look into increasing this to
>> say 16 bits :).  Given each mbuf will have this, it could cause a
>> significant increase in memory usage for a system with a large number of
>> packets (although who cares, ram is cheap).
>>
>>
>> /* MRT compile-time constants */
>> #ifdef _KERNEL
>>   #ifndef ROUTETABLES
>>    #define RT_NUMFIBS 1
>>    #define RT_MAXFIBS 1
>>   #else
>>    /* while we use 4 bits in the mbuf flags, we are limited to 16 */
>>    #define RT_MAXFIBS 16
>>    #if ROUTETABLES>  RT_MAXFIBS
>>     #define RT_NUMFIBS RT_MAXFIBS
>>     #error "ROUTETABLES defined too big"
>>    #else
>>     #if ROUTETABLES == 0
>>      #define RT_NUMFIBS 1
>>     #else
>>      #define RT_NUMFIBS ROUTETABLES
>>     #endif
>>    #endif
>>   #endif
>> #endif
>>
>>
>
>
>