Route caching

[Alexander V. Chernikov]

On 29.05.2014 16:33, Nadav Har'El wrote:
> Hi,
Hello!
> I'm working on the OSv project (http://osv.io/), a new BSD-licensed
> operating system for virtual machines. OSv's networking code is based
> on that of FreeBSD.
>
> I recently noticed an inefficiency that I believe exists also in
> FreeBSD's networking code, and I was wondering why this was done,
> and whether FreeBSD can also be improved in the same way by fixing
> this problem.
>
> My issue is that, for example, when running a UDP server answering
> hundreds of thousands of requests per second, I get the same number of
> calls to the routing table lookup function (rtalloc_ign_fib(), etc.).
> These calls are relatively slow: Each involves several mutex locks and
> unlocks (a rwlock for the radix tree, and a mutex for the individual
> route), which are relatively slow in the uncontended case, but even worse
> when several CPUs start to access the network heavily, and we start to see
> context switches hurting the performance of the server even further.

Yes, that's true.
> Looking at FreeBSD's udp_output(), I see it does the following:
>
>     error = ip_output(m, inp->inp_options, NULL, ipflags,
>                       inp->inp_moptions, inp)
>
> Note how NULL is passed as the third parameter. This tells ip_output
> that it can't cache the previously found route, and needs to look for
> it again and again on every packet output - even in the common case
> where a socket will only ever send packets on one interface.
>
> It seems that this change was done around FreeBSD 5.4. In the original
> UCB code (4.4Lite), I see this:
>
> 	error = ip_output(m, inp->inp_options, &inp->inp_route,
>                  inp->inp_socket->so_options & (SO_DONTROUTE | SO_BROADCAST),
>                  inp->inp_moptions);
>
> So the last-found route was cached in inp->inp_route, and possibly
> reused on the next packet to be sent.
>
> Does anyone have any idea why inp->inp_route was removed in FreeBSD?
> Doesn't this also hurt FreeBSD's network performance?
Well, there are two problems:
First, using cached routes makes it more complex to change routing stack 
to be more efficient.
The second one is basically the fact that using cached routes is simply 
incorrect:
no one protects/notifies you on interface removal. That's why we've 
removed cached route support from various tunneling schemes.

There is some ongoing work to change rte_* api and eliminate the need 
for per-rte mutex (and use different, more efficient lookup mechanisms).
There is also another alternative which you can currently use: flowtable 
(not included in GENERIC).
It has been fixed recently and should work better in your case.

> Thanks,
> Nadav.
>
>