EVFILT_VNODE doesn't scale to large directory trees?

[Kenton Varda]

On Mon, Oct 25, 2010 at 1:54 AM, Igor V. Ruzanov <igorr at canmos.ru> wrote:

> I thought so too but was not sure about his problem. With
> kqueue()/kevent() we can monitor every open file descriptor. So how many
> files are laid down in the directory tree, 100, 1000, 15000? In every
> such way its possible to write or find already written daemon that
> monitors every file in selected folder doing several jobs in separate
> threads. There is might be little problem with receiving of *lots* events,
> so we could create several pipes (for example) for setting up of
> communication between deamon and system.
> Note that we must remember to set proper meaning of kern.maxfiles sysctl
> variable.
>

I worry that simply increasing the FD limits to meet my needs would have
some negative effects, otherwise the limits would be much higher in the
first place.  How much kernel memory does each open FD consume?  Probably
most of that is wasted space, since I'm opening these FDs for no other
purpose than to pass them to kqueue -- I never read or write them.  But it
sounds like you're saying that there is no alternative (other than polling,
which would obviously be a lot worse), so I guess I'll live with it.

Well, one other idea:  Is there a way to simply monitor *all* I/O by all
processes owned by the current user?  I could then filter the events down to
the directory I'm interested in.  Not the ideal solution, but it would scale
to a source tree of infinite size (since the machine can only be accessing a
finite number of those files at once).  It seems likely that this has been
implemented somewhere due to the obvious system monitoring applications, but
I'm not quite sure where to start looking.