FreeBSD 7.2 + NFS + nullfs + unlink + fstat = Stale NFS File Handle

[Linda Messerschmidt]

We have encountered a problem with a weird behavior when NFS and
nullfs are combined and a program creates, unlinks, and then fstats a
file in the resulting directory.

After encountering this problem in the wild, I wrote a quick little C
program.  It creates a file, unlinks the file, and then fstat's the
open file descriptor.  The results:

UFS: OK
NFS: OK
UFS+NULLFS: OK
NFS+NULLFS: fstat returns ESTALE

The UFS test is just run in /tmp.  All others are run in /mnt (with
umounts in between).

The NFS setup looks like this:

client# mount -o tcp server:/export/example /mnt

The UFS+NULLFS setup looks like this:

client# mount -t nullfs /tmp /mnt

The NFS+NULLFS mount setup looks like this:

client# mount -o tcp server:/export /export
client# mount -t nullfs /export/example /mnt

As far as I understand, this behavior should be supported, and the
file should be "finally" deleted when the descriptor is closed.

Even so, this is obscure enough that the response would be "don't do
that" except that the open-unlink-fstat behavior was encountered with
/usr/bin/vi, which does exactly that on its temporary files.  So, we
either fix it or retool everything not to use nullfs.

Does anyone know what the likely source of this different behavior is,
and whether it is feasible to address?  Or is NFS+NULLFS just pushing
the envelope a little too far?

This is on 7.2-RELEASE-p1 and 7.2-STABLE.

Thanks for any advice!

Test program source below:

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>

int main() {
	int fd;
	struct stat st;

	fd = open("testfile", O_RDWR | O_CREAT | O_TRUNC, 0666);
	if (!fd) {
		fprintf(stderr, "open failed: %s\n",strerror(errno));
		return 10;
	}
	if (unlink("testfile")) {
		fprintf(stderr, "unlink failed: %s\n",strerror(errno));
		return 20;
	}
	if (fstat(fd, &st)) {
		fprintf(stderr, "fstat failed: %s\n",strerror(errno));
		return 30;
	}
	close(fd);
	return 0;
}