arp -na performance w/ many permanent entries

[Jeremy Chadwick]

On Sat, Jun 05, 2010 at 09:48:01PM -0400, Nick Rogers wrote:
> On Mon, May 31, 2010 at 10:54 PM, Nick Rogers <ncrogers at gmail.com> wrote:
> 
> >
> > [root@ ~]# time arp -na > /dev/null
> >
> > real 0m12.761s
> > user 0m2.959s
> > sys 0m9.753s
> > [root@ ~]#
> >
> >
> > Notice that "arp -na" takes about 13s to execute even though there is no
> > other load. This can get a lot worse by a few orders of magnitude on a
> > loaded machine in a production environment, and seems to scale up linearly
> > when more aliases are added to the interface (permanent ARP entries
> > created).
> >
> > Is this a reasonable problem that can be fixed/improved, or am I stuck with
> > the slow arp -na output? Any help or comments is greatly appreciated.
> >
> 
> I tried the same scenario on 8.1-BETA1 and it still takes a very long time
> for arp(8) to complete.
> 
> I was able to isolate the performance bottleneck to a small piece of the
> arp(8) code. It seems that looking up the interface for an ARP entry is a
> very heavy operation when that entry corresponds to an alias assigned to the
> interface. Permanent ARP entries that do not correspond with an interface
> alias do not seem to cause arp(8) to puke on the interface lookup.
> 
> The following commands and code diff illustrates how arp(8) can be modified
> to run a lot faster in this scenario, but obviously the associated interface
> is no longer printed for each entry.
> 
> [root@ /usr/src/usr.sbin/arp]# uname -a
> FreeBSD .localdomain 8.1-BETA1 FreeBSD 8.1-BETA1 #0: Thu May 27 15:03:30 UTC
> 2010     root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
> [root@ /usr/src/usr.sbin/arp]# time /usr/sbin/arp -na | wc -l
>     4100
> 
> real 0m14.903s
> user 0m3.133s
> sys 0m11.519s
> [root@ /usr/src/usr.sbin/arp]# pwd
> /usr/src/usr.sbin/arp
> [root@ /usr/src/usr.sbin/arp]# !diff
> diff -ruN arp.c.orig arp.c
> --- arp.c.orig 2010-06-05 18:25:24.000000000 +0000
> +++ arp.c 2010-06-05 18:28:19.000000000 +0000
> @@ -562,7 +562,7 @@
>   const char *host;
>   struct hostent *hp;
>   struct iso88025_sockaddr_dl_data *trld;
> - char ifname[IF_NAMESIZE];
> + //char ifname[IF_NAMESIZE];
>   int seg;
> 
>   if (nflag == 0)
> @@ -591,8 +591,8 @@
>   }
>   } else
>   printf("(incomplete)");
> - if (if_indextoname(sdl->sdl_index, ifname) != NULL)
> - printf(" on %s", ifname);
> + //if (if_indextoname(sdl->sdl_index, ifname) != NULL)
> + //printf(" on %s", ifname);
>   if (rtm->rtm_rmx.rmx_expire == 0)
>   printf(" permanent");
>   else {
> [root@ /usr/src/usr.sbin/arp]# make clean && make
> rm -f arp arp.o arp.4.gz arp.8.gz arp.4.cat.gz arp.8.cat.gz
> Warning: Object directory not changed from original /usr/src/usr.sbin/arp
> cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -c
> arp.c
> cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign
>  -o arp arp.o
> gzip -cn arp.4 > arp.4.gz
> gzip -cn arp.8 > arp.8.gz
> [root@ /usr/src/usr.sbin/arp]# time ./arp -na | wc -l
>     4099
> 
> real 0m0.036s
> user 0m0.015s
> sys 0m0.021s
> [root@ /usr/src/usr.sbin/arp]#
> 
> Notice that 0.036s without the interface lookup is a heck of a lot faster
> than 14.903s when doing the interface lookup.
> 
> Is there something that can be done to speedup the call to if_indextoname(),
> or would it be worthwhile for me to submit a patch that adds the ability to
> skip the interface lookup as an arp(8) option?

This might be a better question for either freebsd-net or
freebsd-hackers.  I should warn you in advance that you might receive a
bit of flack given that you have over 4000 IP aliases assigned to an
interface.  Explaining your setup may also help people understand why it
is you need what you do.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |