Monitoring FS changes

[Jordan Hubbard]

> On Jan 3, 2016, at 1:08 PM, Mark Felder <feld at FreeBSD.org> wrote:
> 
> If someone, anyone out there is capable of bringing us something that
> does scale it would be greatly appreciated. Lots of nice Linux software
> uses this, but when they do port to FreeBSD we have to do full
> filesystem scans. It's such a waste.

I’ve been pondering this for awhile since a lot of interesting enterprise features require a working filesystem change notification mechanism that scales to thousands or even millions of files (how did we bump into the 32 bit NFS file handle problem at iXsystems?  Somebody tried to share more than 4 billion files over NFS - Enterprise folks do some crazy s**t!).

The big question is less whether it’s possible and more what kind of mechanism people will find palatable.  The OS X FSEvents mechanism works reasonably well and is used constantly to trigger things like spotlight search indexing and such, and I was by no means involved in its creation at Apple so I can only speak peripherally to the implementation, but it seems like it took a fairly long time for it to become “light weight” enough to use without the overhead being punitive.  Any similar mechanism in FreeBSD would also have to go through some evolutionary performance iterations - do people want it badly enough to invest in it long-term?  I don’t know, but I do know that a long-term investment would be necessary to really make it work well and provide all of the appropriate APIs for talking to it.

I think we can probably all agree that Linux inotify wouldn’t be worth the trouble.  From the wikipedia page:

	• Inotify does not support recursively watching directories, meaning that a separate inotify watch must be created for every subdirectory.[4]
	• Inotify does report some but not all events in sysfs and procfs.
	• Notification via inotify requires the kernel to be aware of all relevant filesystem events, which is not always possible for networked filesystems such as NFS where changes made by one client are not immediately broadcast to other clients.
	• Rename events are not handled directly; i.e., inotify issues two separate events that must be examined and matched in a context of potential race conditions.

I think the first issue alone is a deal killer.  Having to walk the filesystem tree posting notifications on every [new] directory just to watch a filesystem in its entirety would be pretty onerous and failure-prone to boot.  By contrast:  https://en.wikipedia.org/wiki/FSEvents

This is also not to say that I would expect anything in FreeBSD to be API-compatible (though the upstream clients would probably grumble at yet another notification mechanism API to #ifdef into their code), simply that there are only so many design patterns to follow.  A filesystem change is a filesystem change.  Everything beyond that is just a glorified pub/sub mechanism.

Assuming there’s interest, I could potentially see throwing some engineering effort into this.

- Jordan