arp -na performance w/ many permanent entries

[Garrett Cooper]

On Sat, Jun 5, 2010 at 8:16 PM, Jeremy Chadwick
<freebsd at jdc.parodius.com> wrote:
> On Sat, Jun 05, 2010 at 09:48:01PM -0400, Nick Rogers wrote:
>> On Mon, May 31, 2010 at 10:54 PM, Nick Rogers <ncrogers at gmail.com> wrote:
>>
>> >
>> > [root@ ~]# time arp -na > /dev/null
>> >
>> > real 0m12.761s
>> > user 0m2.959s
>> > sys 0m9.753s
>> > [root@ ~]#
>> >
>> >
>> > Notice that "arp -na" takes about 13s to execute even though there is no
>> > other load. This can get a lot worse by a few orders of magnitude on a
>> > loaded machine in a production environment, and seems to scale up linearly
>> > when more aliases are added to the interface (permanent ARP entries
>> > created).
>> >
>> > Is this a reasonable problem that can be fixed/improved, or am I stuck with
>> > the slow arp -na output? Any help or comments is greatly appreciated.
>> >
>>
>> I tried the same scenario on 8.1-BETA1 and it still takes a very long time
>> for arp(8) to complete.
>>
>> I was able to isolate the performance bottleneck to a small piece of the
>> arp(8) code. It seems that looking up the interface for an ARP entry is a
>> very heavy operation when that entry corresponds to an alias assigned to the
>> interface. Permanent ARP entries that do not correspond with an interface
>> alias do not seem to cause arp(8) to puke on the interface lookup.
>>
>> The following commands and code diff illustrates how arp(8) can be modified
>> to run a lot faster in this scenario, but obviously the associated interface
>> is no longer printed for each entry.
>>
>> [root@ /usr/src/usr.sbin/arp]# uname -a
>> FreeBSD .localdomain 8.1-BETA1 FreeBSD 8.1-BETA1 #0: Thu May 27 15:03:30 UTC
>> 2010     root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>> [root@ /usr/src/usr.sbin/arp]# time /usr/sbin/arp -na | wc -l
>>     4100
>>
>> real 0m14.903s
>> user 0m3.133s
>> sys 0m11.519s
>> [root@ /usr/src/usr.sbin/arp]# pwd
>> /usr/src/usr.sbin/arp
>> [root@ /usr/src/usr.sbin/arp]# !diff
>> diff -ruN arp.c.orig arp.c
>> --- arp.c.orig 2010-06-05 18:25:24.000000000 +0000
>> +++ arp.c 2010-06-05 18:28:19.000000000 +0000
>> @@ -562,7 +562,7 @@
>>   const char *host;
>>   struct hostent *hp;
>>   struct iso88025_sockaddr_dl_data *trld;
>> - char ifname[IF_NAMESIZE];
>> + //char ifname[IF_NAMESIZE];
>>   int seg;
>>
>>   if (nflag == 0)
>> @@ -591,8 +591,8 @@
>>   }
>>   } else
>>   printf("(incomplete)");
>> - if (if_indextoname(sdl->sdl_index, ifname) != NULL)
>> - printf(" on %s", ifname);
>> + //if (if_indextoname(sdl->sdl_index, ifname) != NULL)
>> + //printf(" on %s", ifname);
>>   if (rtm->rtm_rmx.rmx_expire == 0)
>>   printf(" permanent");
>>   else {
>> [root@ /usr/src/usr.sbin/arp]# make clean && make
>> rm -f arp arp.o arp.4.gz arp.8.gz arp.4.cat.gz arp.8.cat.gz
>> Warning: Object directory not changed from original /usr/src/usr.sbin/arp
>> cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
>> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
>> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -c
>> arp.c
>> cc -O2 -pipe  -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall
>> -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes
>> -Wmissing-prototypes -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign
>>  -o arp arp.o
>> gzip -cn arp.4 > arp.4.gz
>> gzip -cn arp.8 > arp.8.gz
>> [root@ /usr/src/usr.sbin/arp]# time ./arp -na | wc -l
>>     4099
>>
>> real 0m0.036s
>> user 0m0.015s
>> sys 0m0.021s
>> [root@ /usr/src/usr.sbin/arp]#
>>
>> Notice that 0.036s without the interface lookup is a heck of a lot faster
>> than 14.903s when doing the interface lookup.
>>
>> Is there something that can be done to speedup the call to if_indextoname(),
>> or would it be worthwhile for me to submit a patch that adds the ability to
>> skip the interface lookup as an arp(8) option?
>
> This might be a better question for either freebsd-net or
> freebsd-hackers.  I should warn you in advance that you might receive a
> bit of flack given that you have over 4000 IP aliases assigned to an
> interface.  Explaining your setup may also help people understand why it
> is you need what you do.

    I agree with Jeremy. I think that the problem that you've
discovered is the fact that it's using stdio-based buffered output
instead of buffering more of the contents in a string and punting it
out in larger chunks.
HTH,
-Garrett