Route caching

[Nadav Har'El]

Hi,

I'm working on the OSv project (http://osv.io/), a new BSD-licensed
operating system for virtual machines. OSv's networking code is based
on that of FreeBSD.

I recently noticed an inefficiency that I believe exists also in
FreeBSD's networking code, and I was wondering why this was done,
and whether FreeBSD can also be improved in the same way by fixing
this problem.

My issue is that, for example, when running a UDP server answering
hundreds of thousands of requests per second, I get the same number of
calls to the routing table lookup function (rtalloc_ign_fib(), etc.).
These calls are relatively slow: Each involves several mutex locks and
unlocks (a rwlock for the radix tree, and a mutex for the individual
route), which are relatively slow in the uncontended case, but even worse
when several CPUs start to access the network heavily, and we start to see
context switches hurting the performance of the server even further.

Looking at FreeBSD's udp_output(), I see it does the following:

   error = ip_output(m, inp->inp_options, NULL, ipflags,
                     inp->inp_moptions, inp)

Note how NULL is passed as the third parameter. This tells ip_output
that it can't cache the previously found route, and needs to look for
it again and again on every packet output - even in the common case
where a socket will only ever send packets on one interface.

It seems that this change was done around FreeBSD 5.4. In the original
UCB code (4.4Lite), I see this:

	error = ip_output(m, inp->inp_options, &inp->inp_route,
                inp->inp_socket->so_options & (SO_DONTROUTE | SO_BROADCAST),
                inp->inp_moptions);

So the last-found route was cached in inp->inp_route, and possibly
reused on the next packet to be sent.

Does anyone have any idea why inp->inp_route was removed in FreeBSD?
Doesn't this also hurt FreeBSD's network performance?

Thanks,
Nadav.


-- 
Nadav Har'El                        |     Thursday, May 29 2014, 29 Iyyar 5774
nyh at math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |If glory comes after death, I'm not in a
http://nadav.harel.org.il           |hurry. (Latin proverb)