misc/148581: fopen fails with EMFILE if there are more than SHORT_MAX fds open

[Manish Vachharajani]

>Number:         148581
>Category:       misc
>Synopsis:       fopen fails with EMFILE if there are more than SHORT_MAX fds open
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jul 13 21:30:07 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Manish Vachharajani
>Release:        7.3-RELEASE
>Organization:
LineRate Systems
>Environment:
FreeBSD wks1.int.lineratesystems.com 7.3-RELEASE FreeBSD 7.3-RELEASE #0: Fri Jul  2 12:56:26 MDT 2010     root at wks1.int.lineratesystems.com:/usr/obj/usr/src/sys/DEV73PMC  amd64

>Description:
fopen will fail with EMFILE if there are more than SHORT_MAX file descriptors open in the process.  It does not matter that these fds were not created by fopen.  

To make matters worse gcc's libstdc++ uses fopen and friends to implement ofstream so those function mysteriously fail if there are more than 32k fds open in the process.





>How-To-Repeat:
To reproduce the problem compile and run:

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

#define BIGNUM 50000
int main(int argc, char *argv[]) {
  int i;
  int fds[BIGNUM];
  for(i=0; i < BIGNUM; ++i) {
    fds[i] = open("/dev/null", O_RDONLY);
  }

  FILE *fil = fopen("/dev/null", "r");
  if(fil == NULL) {
    fprintf(stderr, "Could not open /dev/null: %s\n" , strerror(errno));
  }

  for(i=0; i < BIGNUM; ++i) {
    close(fds[i]);
  }

  return 0;
}
>Fix:
The simple fix is to make _file from struct sFILE { ... } FILE; in stdio.h an int instead of a short.  However, this will break binary compatibility with anyone compiled with an old libc.

A very dirty fix that would not break binary compatibility is, for each architecture, use the open space from the padding and alignment requirements of FILE to stash the other bits of _file and make all users of FILE use an accessor macro that pulls out the right bits.

A quick fix to double the threshold at which the problem occurs would be to make _file an unsigned short and use the all 1's value to indicate that this is not a file resource.  Not sure if this will work either, though.

>Release-Note:
>Audit-Trail:
>Unformatted: