Re: native inotify implementation

On Fri, May 16, 2025 at 11:02:33AM -0500, Jake Freeland wrote:
> On Mon May 12, 2025 at 3:58 PM CDT, Mark Johnston wrote:
> > For the past while I've been hacking on a native implementation of
> > Linux's inotify.  Functionality-wise, this is similar to but not quite
> > equivalent to the EVFILT_VNODE kqueue filter.  While we already have a
> > userspace implementation of inotify built on top of kqueue, it shares
> > the limitations of EVFILT_VNODE, and my version can also be used in the
> > Linuxulator.  (Please let me know if you're interested in working on
> > that and testing it out.)
> >
> > The WIP implementation is here: https://reviews.freebsd.org/D50315
> > There are some loose ends to tie up there, but I wanted to solicit
> > feedback before I keep spending time on it.  I also wonder how this
> > feature should be handled in the ports tree where libinotify is used
> > today: if src starts installing /usr/include/sys/inotify.h, will ports
> > start using the native implementation automatically?  Do we need to have
> > some kind of flag day?
> 
> Seems like libinotify might cause some trouble.
> 
> See https://gcc.gnu.org/onlinedocs/cpp/Invocation.html:
> 
> "You can use -I to override a system header file, substituting your own
> version, since these directories are searched before the standard system
> header file directories."
> 
> Looks like this is true in practice for clang:
> 
> [root@freebsd ~]$ cpp -v -I/usr/local/include /dev/null -o /dev/null
> ...
> clang -cc1 version 19.1.7 based upon LLVM 19.1.7 default target x86_64-unknown-freebsd15.0
> #include "..." search starts here:
> #include <...> search starts here:
>  /usr/local/include
>  /usr/lib/clang/19/include
>  /usr/include
> End of search list.
> 
> Projects that use libinotify likely already specify -I/usr/local/include
> so the C preprocessor can find /usr/local/include/sys/inotify.h. I
> imagine they'd need to remove that include or uninstall the package to
> pick up your new system header.
>
> > This work was largely motivated by a race condition in EVFILT_VNODE: in
> > order to get events for a particular file, you first have to open it, by
> > which point you may have missed the event(s) you care about.  For
> > instance, if some upload service adds files to a directory, and you want
> > to know when a new file has finished uploading, you'd have to watch the
> > directory to get new file events, scan the directory to actually find
> > the new file(s), open them, and then wait for NOTE_CLOSE (which might
> > never arrive if the upload had already finished).  Aside from that, the
> > need to hold each monitored file open is also a problem for large
> > directory hierarchies as it's easy to exhaust file descriptor limits.
> >
> > My initial solution was a new kqueue filter, EVFILT_FSWATCH, which lets
> > one watch for all file events under a mountpoint.  The consumer would
> > allocate a ring buffer with space to store paths and event metadata,
> > register that with the kernel, and the kernel would write entries to the
> > buffer, using reverse lookups to find a path for each event vnode.  This
> > prototype worked, but got somewhat hairy and I decided it would be
> > better to simply implement an existing interface: inotify already exists
> > and is commonly used, and has a somewhat simpler model, as it merely
> > watches for events within a particular directory.
> 
> I've found that more and more developers are blindly using Linux-specific
> interfaces these days, so +1 for natively supporting another one.
> 
> The more support we have for these, the easier porting/Linux emulation is.
> I think the benefits of this far outweighs the cost of maintaining the
> code.

I think so too.  My perspective is that we should implement widely used
Linux interfaces as part of the larger goal of making existing software
usable on FreeBSD.  This is more important than the purity of the
kernel's interfaces or architecture, at least up to a certain point.

The whole purpose of an OS is to let users run the programs they want to
run, without getting in the way (too much).

> Let me know if you need any help testing this.

Thanks.  I'll follow up on in this thread once the patch is a bit closer
to being ready.  Right now, Gleb is kindly helping run various test
suites and comparing the native inotify implementation with libinotify.