RFC: Updated ARP Queue patch...

[George Neville-Neil]

Howdy,

This is marked as "Updated" because I first proposed this on arch@ but am now sending it to
a wider audience as I'm hoping to commit it in the near future.

Please review the following patch against HEAD:

http://people.freebsd.org/~gnn/head-arpqueue2.diff

This patch makes two changes to the ARP code:

1) It adds a sysctl configurable queue of packets that are held until an ARP reply is received or
timed out.

net.link.ether.inet.maxhold

Having the queue addresses a problem in modern systems where programs that use connectionless 
protocols for communication will suffer from dropping many packets when they start up or when
an ARP entry moves.

2) Makes the time we wait for an arp reply configurable via another sysctl.

net.link.ether.inet.wait

The old, pre 8.0, ARP code would run the timer once per second.  The new
ARP code sets a timeout of 20 seconds on each entry.  Neither value was specified
in RFC 826.  As a matter of fact, RFC 826 had this to say about timeouts:

"It may be desirable to have table aging and/or timeouts.  The
implementation of these is outside the scope of this protocol."

This new code does not change the default value of either the arpqueue (which was
always 1 packet) nor does it change the new value of the ARP down timeout.

I have a different patch for 7, which I will propose after I can get this in to
HEAD and MFC'd to 8.

Best,
George