standards/124860: flockfile(3) doesn't work when the memory has been exhausted.

[Tomohisa Tanaka]

>Number:         124860
>Category:       standards
>Synopsis:       flockfile(3) doesn't work when the memory has been exhausted.
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-standards
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Jun 21 23:00:13 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Tomohisa Tanaka
>Release:        FreeBSD 6.3-RELEASE i386
>Organization:
>Environment:
System: FreeBSD freebsd.localdomain 6.3-RELEASE FreeBSD 6.3-RELEASE #0: Wed Jan 16 04:18:52 UTC 2008 root at dessler.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386

>Description:
	When the flockfile(3) acquires a lock, and the specified
	stream has never locked, and the memory has been exhausted,
	flockfile(3) returns without the stream locked.

	The flockfile(3) calls _pthread_mutex_lock() according to the
	file /usr/src/lib/libc/stdio/_flock_stub.c, as follows.

/usr/src/lib/libc/stdio/_flock_stub.c:
     69 #define _lock _extra
     ...
     83                 _pthread_mutex_lock(&fp->_lock->fl_mutex);

        The _pthread_mutex_lock() does never fail if
        fp->_extra->fl_mutex has already been dynamically initialized.
        However, if fp->_extra->fl_mutex has been statically
        initialized, the _pthread_mutex_lock() tries to perform the
        dynamic initialization of it by calling the
        pthread_mutex_init(3), as follows.

/usr/src/lib/libpthread/thread/thr_mutex.c:
    302 static int
    303 init_static_private(struct pthread *thread, pthread_mutex_t *mutex)
    304 {
        ...
    310                 ret = pthread_mutex_init(mutex, &static_mattr);
        ...
    317 }
        ...
    849 int
    850 _pthread_mutex_lock(pthread_mutex_t *m)
    851 {
        ...
    866         else if ((*m != NULL) ||
    867             ((ret = init_static_private(curthread, m)) == 0))
    868                 ret = mutex_lock_common(curthread, m, NULL);
        ...
    871 }

    	When the memory has been exhausted, the pthread_mutex_init(3)
    	returns ENOMEM.  All mutexes of the stdio stream are
    	statically initialized, according to the file
    	/usr/src/lib/libc/stdio/findfp.c.

>How-To-Repeat:
% cat test.c
#include <stdio.h>
#include <stdlib.h>
#include <err.h>
#include <errno.h>
#include <pthread.h>

static pthread_barrier_t barrier;
static FILE *fp;

static void *
run(void *arg)
{
  pthread_barrier_wait(&barrier);
  flockfile(fp); /* [1] */
  return NULL;
}

int
main(void)
{
  pthread_t thread;
  pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
  void *p;
  size_t k;

  if (pthread_barrier_init(&barrier, NULL, 2) != 0)
    err(1, "pthread_barrier_init");
  if (pthread_create(&thread, NULL, run, NULL) != 0)
    err(1, "pthread_create");
  if ((fp = fopen("log", "w")) == NULL)
    err(1, "log");

#if 1
  for (k = 1; k < 64; ++k) {
    while ((p = malloc(k)) != NULL)
      ;
  }
  /*
   * Now, we are in a special situation where the memory has been
   * extremely exhausted.  Here, 'mutex' has not initialized with
   * pthread_mutex_init(3), so pthread_mutex_lock(3) as follows will
   * try to initialize "mutex" using pthread_mutex_init(3) internally.
   * However, malloc(3) within the pthread_mutex_init(3) returns NULL,
   * and then pthread_mutex_lock(3) fails and returns ENOMEM.
   */
  if ((errno = pthread_mutex_lock(&mutex)) != 0) {
    warn("pthread_mutex_lock");
  }
#endif

  flockfile(fp); /* [2] */
  /*
   * Here, this main thread does NOT have the lock of "fp", because
   * pthread_mutex_lock(3) within flockfile(3) at [2] has failed.  We
   * must remember "fp->_extra->fl_mutex" has not been initialized by
   * pthread_mutex_init(3) as well as "mutex".
   */
  pthread_barrier_wait(&barrier);
  /*
   * Another thread will call flockfile(3) at [1], but it returns
   * immediately.  No thread can get the lock of 'fp' in this
   * situation.  If we comment out from "#if 1" to "#endif", a
   * deadlock occurs.
   */
  pthread_join(thread, NULL);
  return 0;
}

% cat test.sh
#!/bin/sh
ulimit -v 10000
./a.out

% gcc -pthread test.c 
% sh test.sh  
a.out: pthread_mutex_lock: Cannot allocate memory 
% 

>Fix:
	I think that the function __sfp() of
	/usr/src/lib/libc/stdio/findfp.c must perform the dynamic
	initialization of fp->_extra->fl_mutex, if it has been only
	statically initialized.  And if it failed, __sfp() must return
	NULL.
>Release-Note:
>Audit-Trail:
>Unformatted: