Route caching

Alexander V. Chernikov melifaro at
Fri May 30 11:09:52 UTC 2014

On 29.05.2014 16:33, Nadav Har'El wrote:
> Hi,
> I'm working on the OSv project (, a new BSD-licensed
> operating system for virtual machines. OSv's networking code is based
> on that of FreeBSD.
> I recently noticed an inefficiency that I believe exists also in
> FreeBSD's networking code, and I was wondering why this was done,
> and whether FreeBSD can also be improved in the same way by fixing
> this problem.
> My issue is that, for example, when running a UDP server answering
> hundreds of thousands of requests per second, I get the same number of
> calls to the routing table lookup function (rtalloc_ign_fib(), etc.).
> These calls are relatively slow: Each involves several mutex locks and
> unlocks (a rwlock for the radix tree, and a mutex for the individual
> route), which are relatively slow in the uncontended case, but even worse
> when several CPUs start to access the network heavily, and we start to see
> context switches hurting the performance of the server even further.

Yes, that's true.
> Looking at FreeBSD's udp_output(), I see it does the following:
>     error = ip_output(m, inp->inp_options, NULL, ipflags,
>                       inp->inp_moptions, inp)
> Note how NULL is passed as the third parameter. This tells ip_output
> that it can't cache the previously found route, and needs to look for
> it again and again on every packet output - even in the common case
> where a socket will only ever send packets on one interface.
> It seems that this change was done around FreeBSD 5.4. In the original
> UCB code (4.4Lite), I see this:
> 	error = ip_output(m, inp->inp_options, &inp->inp_route,
>                  inp->inp_socket->so_options & (SO_DONTROUTE | SO_BROADCAST),
>                  inp->inp_moptions);
> So the last-found route was cached in inp->inp_route, and possibly
> reused on the next packet to be sent.
> Does anyone have any idea why inp->inp_route was removed in FreeBSD?
> Doesn't this also hurt FreeBSD's network performance?
Well, there are two problems:
First, using cached routes makes it more complex to change routing stack 
to be more efficient.
The second one is basically the fact that using cached routes is simply 
no one protects/notifies you on interface removal. That's why we've 
removed cached route support from various tunneling schemes.

There is some ongoing work to change rte_* api and eliminate the need 
for per-rte mutex (and use different, more efficient lookup mechanisms).
There is also another alternative which you can currently use: flowtable 
(not included in GENERIC).
It has been fixed recently and should work better in your case.

> Thanks,
> Nadav.

More information about the freebsd-net mailing list