kern/179901: [patch] Multicast SO_REUSEADDR handled incorrectly
Michael Gmelin
freebsd at grem.de
Mon Jun 24 00:10:01 UTC 2013
>Number: 179901
>Category: kern
>Synopsis: [patch] Multicast SO_REUSEADDR handled incorrectly
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Jun 24 00:10:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator: Michael Gmelin
>Release: FreeBSD 9.1-RELEASE-p2 amd64
>Organization:
Grem Equity GmbH
>Environment:
System: FreeBSD box.grem.de 9.1-RELEASE-p2 FreeBSD 9.1-RELEASE-p2 #5 r249052M: Fri May 31 17:50:16 UTC
>Description:
Traditionally SO_REUSEADDR is used to indicate binding to the same
address and port is permitted in the case of multicast addresses. BSD
introduced the special sockopt SO_REUSEPORT and some point, but
SO_REUSEADDR still implied SO_REUSEPORT automatically, which is
important for portability reasons - client software expects
SO_REUSEADDR to be sufficient and the kernel indicates that this is the
way it should be for multicast addresses.
While debugging an issue causing a unit test of the port devel/ice to
fail, I realized that this is not handled correctly anymore [1].
As part of r227207, which was MFC'd in r227428 [2], handling of
sockopts was slightly changed and setting of the implicit SO_REUSEPORT
flag was moved to ip_output.c, which sets INP_REUSEPORT on
inp->inp_flags2, so that these flags can be checked when binding
additional sockets in in_pcb.c. Unfortunately this approach doesn't
work, since these socket options are set before bind is called, and
therefore checking IN_MULTICAST returns false at this point and the
required INP_REUSEPORT option is never set implicitely.
As a result, attempts to bind additional sockets to the same multicast
address and port fail with EADDRINUSE ("Address already in use"), which
breaks existing software.
As a workaround, applications can be fixed by either using
SO_REUSEPORT, or by calling setsockopt SO_REUSEADDR again after
calling bind (because at that point the address is known and can be
detected as multicast by ip_output.c).
Changing application code is not desireable for obvious reasons.
[1] http://lists.freebsd.org/pipermail/freebsd-ports/2013-June/084480.html
[2] http://svnweb.freebsd.org/base?view=revision&revision=227428
>How-To-Repeat:
Build the following program, which is also available at
http://blog.grem.de/multicast.c
# cc -o multicast -c multicast.c
SNIP
#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <memory.h>
#include <errno.h>
struct testdata {
int port;
int flag;
int settwice;
int expectOk;
};
#define TDSIZE 20
#define BASEPORT 5555
const struct testdata td[TDSIZE] = {
{0, SO_REUSEADDR, 0, 1}, {0, SO_REUSEADDR, 0, 1}, {0, SO_REUSEPORT, 0, 1},
{1, SO_REUSEPORT, 0, 1}, {1, SO_REUSEPORT, 0, 1}, {1, SO_REUSEADDR, 0, 1},
{1, SO_REUSEPORT, 0, 1},
{2, SO_REUSEADDR, 1, 1}, {2, SO_REUSEADDR, 1, 1}, {2, SO_REUSEPORT, 0, 1},
{2, SO_REUSEADDR, 0, 1}, {2, SO_REUSEPORT, 0, 1},
{3, 0, 0, 1}, {3, SO_REUSEADDR, 0, 0}, {3, SO_REUSEPORT, 0, 0},
{4, SO_REUSEADDR, 0, 1}, {4, 0, 0, 0},
{5, SO_REUSEPORT, 0, 1}, {5, SO_REUSEPORT, 0, 1}, {5, 0, 0, 0},
};
int main(int argc, char *argv[])
{
int reuse = 1;
struct sockaddr_in localSock;
int sd;
int i;
int lastport = 0;
int port;
memset((char *) &localSock, 0, sizeof(localSock));
localSock.sin_family = AF_INET;
localSock.sin_addr.s_addr = inet_addr("239.1.1.1");
for (i = 0; i < TDSIZE; ++i) {
port = BASEPORT + td[i].port;
localSock.sin_port = htons(port);
sd = socket(AF_INET, SOCK_DGRAM, 0);
if (lastport != port) {
printf("Port %i:\n", port);
lastport = port;
}
if (td[i].flag) {
printf(" Bind using SO_REUSE%s%s...",
td[i].flag == SO_REUSEADDR ? "ADDR" : "PORT",
td[i].settwice?" x 2":"....");
setsockopt(sd, SOL_SOCKET, td[i].flag, (char *)&reuse, sizeof(reuse));
}
else {
printf(" Bind without socketopts.......");
}
if (bind(sd, (struct sockaddr*)&localSock, sizeof(localSock))) {
printf("FAIL (%sexpected): %s\n", td[i].expectOk?"NOT ":"", strerror(errno));
close(sd);
}
else {
printf("OK (%sexpected)\n", td[i].expectOk?"":"NOT ");
}
if (td[i].settwice && td[i].flag)
setsockopt(sd, SOL_SOCKET, td[i].flag, (char *)&reuse, sizeof(reuse));
}
}
SNAP
Running this on 9.1-RELEASE gives the following result:
Port 5555:
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEADDR.......FAIL (NOT expected): Address already in use
Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in use
Port 5556:
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in use
Port 5557:
Bind using SO_REUSEADDR x 2...OK (expected)
Bind using SO_REUSEADDR x 2...OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in use
Port 5558:
Bind without socketopts.......OK (expected)
Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
Port 5559:
Bind using SO_REUSEADDR.......OK (expected)
Bind without socketopts.......FAIL (expected): Address already in use
Port 5560:
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Bind without socketopts.......FAIL (expected): Address already in use
Out of curiosity, I ran the same code on an outdated version of FreeBSD
(7.2-RELEASE):
Port 5555:
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
use
Port 5556:
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
use
Port 5557:
Bind using SO_REUSEADDR x 2...OK (expected)
Bind using SO_REUSEADDR x 2...OK (expected)
Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
use
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
use
Port 5558:
Bind without socketopts.......OK (expected)
Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
Port 5559:
Bind using SO_REUSEADDR.......OK (expected)
Bind without socketopts.......FAIL (expected): Address already in use
Port 5560:
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Bind without socketopts.......FAIL (expected): Address already in use
This shows that there were issues with this code before r227207 as
well - but only when mixing SO_REUSEADDR and SO_REUSEPORT, which hardly
ever happens in practice. Nevertheless, this code didn't behave like it
should either - at least in my understanding.
>Fix:
Disclaimer: The patch needs to be reviewed by somebody who has a better
understanding of the code in question.
The idea of the attached patch is to introduce a new flag called
INP_REUSEADDR in in_pcb.h, which is set on inp->inp_flags2 in
ip_output.c whenever SO_REUSEADDR is set on a socket. This way there
is an explicit record of the setsockopt call which can be utilized,
even though we don't know that the socket will be bound to a multicast
address yet.
The last step is to modify in_pcb.c (line 597), so that the condition
for emitting EADDRINUSE only matches if it's not a multicast address or
if it is a multicast address and neither INP_REUSEADDR nor
INP_REUSEPORT are set.
After applying the patch and rebuilding the kernel, multicast.c runs as
expected:
Port 5555:
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Port 5556:
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Port 5557:
Bind using SO_REUSEADDR x 2...OK (expected)
Bind using SO_REUSEADDR x 2...OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEADDR.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Port 5558:
Bind without socketopts.......OK (expected)
Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
Port 5559:
Bind using SO_REUSEADDR.......OK (expected)
Bind without socketopts.......FAIL (expected): Address already in use
Port 5560:
Bind using SO_REUSEPORT.......OK (expected)
Bind using SO_REUSEPORT.......OK (expected)
Bind without socketopts.......FAIL (expected): Address already in use
The patch is applied using:
# cd /usr/src
# patch < /path/to/freebsd-multicast.patch
--- freebsd-multicast.patch begins here ---
--- sys/netinet/in_pcb.c.orig 2013-06-23 20:52:10.000000000 +0000
+++ sys/netinet/in_pcb.c 2013-06-23 21:14:45.000000000 +0000
@@ -594,7 +594,11 @@
(reuseport & tw->tw_so_options) == 0)
return (EADDRINUSE);
} else if (t && (reuseport == 0 ||
- (t->inp_flags2 & INP_REUSEPORT) == 0)) {
+ (!IN_MULTICAST(ntohl(sin->sin_addr.s_addr)) &&
+ (t->inp_flags2 & INP_REUSEPORT) == 0) ||
+ (IN_MULTICAST(ntohl(sin->sin_addr.s_addr)) &&
+ (t->inp_flags2 &
+ (INP_REUSEADDR | INP_REUSEPORT)) == 0))) {
#ifdef INET6
if (ntohl(sin->sin_addr.s_addr) !=
INADDR_ANY ||
--- sys/netinet/in_pcb.h.orig 2013-06-23 21:02:42.000000000 +0000
+++ sys/netinet/in_pcb.h 2013-06-23 21:03:40.000000000 +0000
@@ -542,6 +542,7 @@
#define INP_RT_VALID 0x00000002 /* cached rtentry is valid */
#define INP_PCBGROUPWILD 0x00000004 /* in pcbgroup wildcard list */
#define INP_REUSEPORT 0x00000008 /* SO_REUSEPORT option is set */
+#define INP_REUSEADDR 0x00000010 /* SO_REUSEADDR option is set */
/*
* Flags passed to in_pcblookup*() functions.
--- sys/netinet/ip_output.c.orig 2013-06-23 21:03:51.000000000 +0000
+++ sys/netinet/ip_output.c 2013-06-23 21:04:58.000000000 +0000
@@ -915,6 +915,10 @@
else
inp->inp_flags2 &= ~INP_REUSEPORT;
}
+ if ((so->so_options & SO_REUSEADDR) != 0)
+ inp->inp_flags2 |= INP_REUSEADDR;
+ else
+ inp->inp_flags2 &= ~INP_REUSEADDR;
INP_WUNLOCK(inp);
error = 0;
break;
--- freebsd-multicast.patch ends here ---
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list