Problem with FUSE + fts

[Rick Macklem]

Hiroshi Nishida wrote:
> Hi,
> 
> I found a weird error with FUSE + fts_read().
> Every time a command like find, rm -r that calls fts_read() is used for a
> FUSE mounted filesystem, it outputs "XXX: No such file or directory" errors
> for some (not all) files/directories.
> 
I think I see the same thing when doing an "rm -r" on a fuse/GlusterFS volume.
To be honest, I just add a "-f" to the command to shut it up and then it deleted
the tree.

I think, in general, what readdir() returns after an entry is unlink'd is undefined
behaviour. As such, the safe way to delete all of a directory is something like:
- in a loop until readdir() returns EOF
  - opendir()
  - readdir() the first entry
  - unlink() that entry
  - closedir()
--> So that all you ever do is readdir() the first entry after an opendir().

However few, if any, apps do this and loop on readdir(), unlink() instead.

>From my limited experience with fuse, the directory offset is an index (which
isn't monotonically increasing for GlusterFS) and those indexes only work while
the directory is open.

I think that in Linux, an opendir(), readdir() reads an entire directory into
storage maintained by the libc functions and then readdir() returns subsequent
entries without doing a kernel syscall. As such, deletion of entries that, in turn,
change directory offsets in the file system, don't affect this.
In FreeBSD, this (reading the entire directory when opened) is done for "unionfs",
but not otherwise.
--> This causes grief for NFS, due to directory offsets (called cookies in NFS)
    change when entries are deleted.
--> It is also the case that many (including Linux, I think) return these "cookies"
    in their "struct dirent", but FreeBSD does not.

I have been tempted to enable "read the entire directory on opendir()" for other
file system types, but the downside is that apps. will use memory for this (possibly
lots for large directories). Since most use UFS or ZFS and don't see problems, I
haven't tried to make a case for this change. (I may try if I find that this
change fixes fuse.)

If you don't mind rebuilding libc from sources, take a look at opendir.c and
readdir.c (I can't remember exactly how it is done, but you should find where it
chooses to read the entire directory upon opendir or the first readdir) and you
could try enabling that for fuse.

The directory offset problem is a thorny one, but will at least be partially
fixed by a new "struct dirent" with a d_off field in it. This has to be done
someday to make ino_t 64bits anyhow.

*** If you see this for cases other than "rm -r", such as "ls -lR", then all
    of the above is bogus.

Another issue (which I doubt is causing this) is that FreeBSD has a 32bit ino_t
but fuse uses a 64bit inode#, so the FreeBSD fuse interface ends up truncating
the high order 32bits off. (GlusterFS uses more than 32bits, but the high order
bits are invariant, so it doesn't break things.)

Hope this somehow helps, rick

> In /usr/src/lib/libc/gen/fts.c, there is fts_safe_changedir(FTS *, FTSENT *,
> int, char *) and the error seems to occur there in the following way:
> 
> FTS *sp;
> FTSENT *p = sp->fts_cur; // Current node
> DIR *dirp = opendir2(p->fts_accpath, oflag); // Open dir
> int fd = _dirfd(dirp); // File descriptor of dirp
> struct stat sb; _fstat(fd, &sb); // fstat current node through fd
> 
> p->fts_ino != sb.st_ino // This happens for some reason...... and sets errno
> = ENOENT
> 
> When the error happens, p->fts_ino always has a small number and sb.st_ino
> has a great number like:
> p->fts_ino = 13, sb.st_ino = 54136
> 
> So, a new inode number seems to be allocated to sb.st_ino though the node
> already has an inode number.
> 
> I would appreciate hearing any feedback on this, though I already posted
> fuse-devel ML and haven't received any helpful responses yet.
> The problem seems to be particular to FreeBSD because I don't get any errors
> with Ubuntu.
> 
> The sample FUSE program is located at
> https://github.com/scopedog/FUSE-Test
> 
> and fts.c is also located under freebsd and ubuntu dirs (interestingly, fts.c
> of FreeBSD and Ubuntu are almost identical).
> 
> Thank you.
> 
> --
> Hiroshi Nishida
> nishida at asusa.net
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>