Re: native inotify implementation

On Mon May 12, 2025 at 3:58 PM CDT, Mark Johnston wrote:
> For the past while I've been hacking on a native implementation of
> Linux's inotify.  Functionality-wise, this is similar to but not quite
> equivalent to the EVFILT_VNODE kqueue filter.  While we already have a
> userspace implementation of inotify built on top of kqueue, it shares
> the limitations of EVFILT_VNODE, and my version can also be used in the
> Linuxulator.  (Please let me know if you're interested in working on
> that and testing it out.)
>
> The WIP implementation is here: https://reviews.freebsd.org/D50315
> There are some loose ends to tie up there, but I wanted to solicit
> feedback before I keep spending time on it.  I also wonder how this
> feature should be handled in the ports tree where libinotify is used
> today: if src starts installing /usr/include/sys/inotify.h, will ports
> start using the native implementation automatically?  Do we need to have
> some kind of flag day?

Seems like libinotify might cause some trouble.

See https://gcc.gnu.org/onlinedocs/cpp/Invocation.html:

"You can use -I to override a system header file, substituting your own
version, since these directories are searched before the standard system
header file directories."

Looks like this is true in practice for clang:

[root@freebsd ~]$ cpp -v -I/usr/local/include /dev/null -o /dev/null
...
clang -cc1 version 19.1.7 based upon LLVM 19.1.7 default target x86_64-unknown-freebsd15.0
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /usr/lib/clang/19/include
 /usr/include
End of search list.

Projects that use libinotify likely already specify -I/usr/local/include
so the C preprocessor can find /usr/local/include/sys/inotify.h. I
imagine they'd need to remove that include or uninstall the package to
pick up your new system header.

> This work was largely motivated by a race condition in EVFILT_VNODE: in
> order to get events for a particular file, you first have to open it, by
> which point you may have missed the event(s) you care about.  For
> instance, if some upload service adds files to a directory, and you want
> to know when a new file has finished uploading, you'd have to watch the
> directory to get new file events, scan the directory to actually find
> the new file(s), open them, and then wait for NOTE_CLOSE (which might
> never arrive if the upload had already finished).  Aside from that, the
> need to hold each monitored file open is also a problem for large
> directory hierarchies as it's easy to exhaust file descriptor limits.
>
> My initial solution was a new kqueue filter, EVFILT_FSWATCH, which lets
> one watch for all file events under a mountpoint.  The consumer would
> allocate a ring buffer with space to store paths and event metadata,
> register that with the kernel, and the kernel would write entries to the
> buffer, using reverse lookups to find a path for each event vnode.  This
> prototype worked, but got somewhat hairy and I decided it would be
> better to simply implement an existing interface: inotify already exists
> and is commonly used, and has a somewhat simpler model, as it merely
> watches for events within a particular directory.

I've found that more and more developers are blindly using Linux-specific
interfaces these days, so +1 for natively supporting another one.

The more support we have for these, the easier porting/Linux emulation is.
I think the benefits of this far outweighs the cost of maintaining the
code.

Let me know if you need any help testing this.

Thanks,
Jake Freeland

>
> Many thanks to Klara for sponsoring this work.