Implementing SO_RERROR

[Roy Marples]

Hi

Currently in the FreeBSD kernel packets are silently dropped when delivery is 
attempted to a socket who has no space left in it's buffer.

There are comments in the code which imply that someone should be notified of 
this. Here is an example:
https://svnweb.freebsd.org/base/head/sys/netinet6/icmp6.c?revision=362909&view=markup#l1976

I have submitted https://reviews.freebsd.org/D26652 for review to rectify this 
was asked to discuss the change here.

To put it simply, all callers of sbappendaddr() now call soroverflow() when the 
packet cannot be delivered.
If the application enabled SO_RERROR on the socket then it will be woken up and 
the next subsequent read on it will return ENOBUFS. For the curious, this error 
code is allowed to be returned from read(2) by POSIX.

This is a general solution to a very specific case where network aware programs 
rely on the reliable delivery of route(4) messages. If a message is dropped, we 
need to so we can perform a full scan of the host to see what has changed and if 
we need to react to it.

Another use case for this is syslogd can now know if a locally sent message was 
dropped telling the admin to increase the buffer size to avoid this in the 
future. Currently if a locally sent message cannot be delivered to syslogd it is 
also silenty dropped if the syslogd socket buffer is full.

If the view of FreeBSD is that this isn't wanted then please remove comments 
from the codebase implying that it is and ideally replace with a comment 
explaining why.

Thanks for your consideration.

Roy