Impact of having a large number of open file descriptors

[Giorgos Keramidas]

On Mon, 2 Jun 2008 21:07:15 -0400, Garance A Drosihn <drosih at rpi.edu> wrote:
> At 12:33 AM +0200 6/3/08, Kris Kennaway wrote:
>>Ivan Voras wrote:
>>>Suleiman Souhlal wrote:
>>>
>>>> I have an old patch that makes kqueue monitor every file write on
>>>> the system and return the inode number in the knote's data field:
>>>> http://people.freebsd.org/~ssouhlal/testing/kqueue-anyvnode-20050503.diff
>>>> .
>>>>
>>>>I'd think it shouldn't be too hard to make it per-mountpoint..
>>
>> FWIW, I would love to use this.  I have situations where I have huge
>> numbers of files and need to cheaply detect changes so I can
>> resynchronize them to remote machines.
>
> I remember a discussion of changes to MacOS10 in Leopard which made it
> easier to implement features such as Spotlight and TimeMachine. The
> description starts here, I think:
>
> http://arstechnica.com/reviews/os/mac-os-x-10-5.ars/7
>
> the section on file-system events.
>
> The idea I thought was interesting was to save the metadata on a
> directory basis, instead of saving it on the file.  So, if file
> /some/dir/fname was changed, then they'd record that *some* file under
> /some/dir has changed.
>
> So when your userland process comes along later on, it still has to
> scan all files in that directory to see which file(s) actually
> changed.  But that's a lot less work than scanning all files in the
> filesystem, and it also means there is much less data that has to be
> kept track of.
>
> I have no idea how easy it would be to implement something similar on
> FreeBSD, but the strategy seemed like a pretty neat idea.

It sounds like a useful compromise between the number of tracked entries
and scanning the entire fs :)