libc_r: threaded application could stuck in accept(2)

[Ruslan Ermilov]

Hi!

We had a bug in our threaded application that would mistakenly close
the descriptor 0, and this triggers a bug in libc_r which I will try
to describe below.

The bug (in libc_r only, libpthread^Wlibkse is unaffected) causes a
threaded application to stuck in accept(2).  libc_r makes every new
descriptor non-blocking, and uses poll(2) and thread context switching
to emulate a blocking behavior.  The bug somehow causes the descriptor
to be left in a blocking mode.

Attached is the test case that demonstrates this bug (this is the
same on 4.8-STABLE and 5.1-BETA).  The utility runs two threads:
the first thread prints the "alive" message every 3 seconds, and
the second thread emulates what our unfortunate application did,
i.e., open() a TCP socket, listen(), accept(), then close() it,
then mistakenly close() descriptor 0.  (Closing the descriptor
0 causes the next socket() call to return 0.)

Some important notes: this bug is only applicable to descriptors
0 - 2 (stdio set), and might have something to do with the code
in uthread_fd.c.  If you remove two lines that free the descriptor
0 in the attached test case, the bug won't manifest itself.

Attached also is the patch for the threaded close() function
that avoids the bug by disallowing the close() call on a
non-active descriptor.

The patch should be committed, but I'm now more interested in
what's going on inside libc_r that causes the descriptor 0 to
be left in the blocking mode.  IOW, I wonder if the attached
patch is the real fix.  ;)

Ah yes, when you'll run the application, it will report
which TCP port it listens to, you then connect to this port,
the application closes the connection, and on the next loop
the application gets stuck in accept().


Cheers,
-- 
Ruslan Ermilov		Sysadmin and DBA,
ru at sunbay.com		Sunbay Software AG,
ru at FreeBSD.org		FreeBSD committer.
-------------- next part --------------
#include <sys/param.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include <err.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static void *thread1(void *);
static void *thread2(void *);

int
main(void)
{
	pthread_t thr1, thr2;
	int r;

	if ((r = pthread_create(&thr1, NULL, thread1, NULL)) != 0)
		errc(1, r, "pthread_create");
	if ((r = pthread_create(&thr2, NULL, thread2, NULL)) != 0)
		errc(1, r, "pthread_create");
	if ((r = pthread_join(thr1, NULL)) != 0)
		errc(1, r, "pthread_join");
	if ((r = pthread_join(thr2, NULL)) != 0)
		errc(1, r, "pthread_join");

	exit(0);
}

static void *
thread1(void *arg __unused)
{

	for (;;) {
		printf("thread 1: alive\n");
		sleep(3);
	}
}

static void *
thread2(void *arg __unused)
{
	int s, s2;
	struct sockaddr_in addr;
	socklen_t addrlen;

	printf("thread 2: freeing descriptor 0 (closing)\n");
	close(0);

	for (;;) {
		if ((s = socket(AF_INET, SOCK_STREAM, 0)) == -1)
			warn("thread 2: socket");
		else {
			printf("thread 2: socket() returned %d\n", s);
			if (listen(s, 1) == -1)
				warn("thread 2: listen");
			else {
				addrlen = sizeof(addr);
				if (getsockname(s, (struct sockaddr *)&addr, &addrlen) == -1)
					warn("thread 2: getsockname");
				else
					printf("thread 2: listening on port %d\n",
					    ntohs(addr.sin_port));
				printf("thread 2: calling accept() on %d\n", s);
				s2 = accept(s, (struct sockaddr *)&addr, &addrlen);
				if (s2 == -1)
					warn("thread 2: accept");
				else {
					printf("thread 2: accept() returned %d, closing\n", s2);
					close(s2);
				}
			}
			printf("thread 2: closing descriptor %d (1st)\n", s);
			close(s);
			printf("thread 2: closing descriptor %d (2nd)\n", s);
			close(s);
		}
		sleep(1);
	}
}
-------------- next part --------------
Index: uthread_close.c
===================================================================
RCS file: /home/ncvs/src/lib/libc_r/uthread/uthread_close.c,v
retrieving revision 1.10.2.3
diff -u -p -r1.10.2.3 uthread_close.c
--- uthread_close.c	22 Oct 2002 14:44:02 -0000	1.10.2.3
+++ uthread_close.c	30 May 2003 14:13:54 -0000
@@ -47,7 +47,8 @@ _close(int fd)
 	struct stat	sb;
 	struct fd_table_entry	*entry;
 
-	if ((fd == _thread_kern_pipe[0]) || (fd == _thread_kern_pipe[1])) {
+	if ((fd == _thread_kern_pipe[0]) || (fd == _thread_kern_pipe[1]) ||
+	    (_thread_fd_table[fd] == NULL)) {
 		/*
 		 * Don't allow silly programs to close the kernel pipe.
 		 */
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20030530/a1491814/attachment.bin