EVFILT_VNODE doesn't scale to large directory trees?

Igor V. Ruzanov igorr at canmos.ru
Tue Oct 26 07:34:03 UTC 2010


On Mon, 25 Oct 2010, Kenton Varda wrote:

|On Mon, Oct 25, 2010 at 1:54 AM, Igor V. Ruzanov <igorr at canmos.ru> wrote:
|
|> I thought so too but was not sure about his problem. With
|> kqueue()/kevent() we can monitor every open file descriptor. So how many
|> files are laid down in the directory tree, 100, 1000, 15000? In every
|> such way its possible to write or find already written daemon that
|> monitors every file in selected folder doing several jobs in separate
|> threads. There is might be little problem with receiving of *lots* events,
|> so we could create several pipes (for example) for setting up of
|> communication between deamon and system.
|> Note that we must remember to set proper meaning of kern.maxfiles sysctl
|> variable.
|>
|
|I worry that simply increasing the FD limits to meet my needs would have
|some negative effects, otherwise the limits would be much higher in the
|first place.  How much kernel memory does each open FD consume?  Probably
|most of that is wasted space, since I'm opening these FDs for no other
|purpose than to pass them to kqueue -- I never read or write them.  But it
|sounds like you're saying that there is no alternative (other than polling,
|which would obviously be a lot worse), so I guess I'll live with it.
|
|Well, one other idea:  Is there a way to simply monitor *all* I/O by all
|processes owned by the current user?  I could then filter the events down to
|the directory I'm interested in.  Not the ideal solution, but it would scale
|to a source tree of infinite size (since the machine can only be accessing a
|finite number of those files at once).  It seems likely that this has been
|implemented somewhere due to the obvious system monitoring applications, but
|I'm not quite sure where to start looking.
|
As a weak solution - is to install `lsof' on your system and do
grepping of the command output periodically to see what the files 
are opened by process(es) with a certain their owner. But this method not 
very good since we must collect any system event in real time way that is 
implemented in kqueue. Another way is to poll events with FAM mechanism 
that comes from SGI IRIX. Try to research if any solutions to use FAM 
under FreeBSD and what the methods of events monitoring could be used in 
FAM together with your project.


+-------------------------------------------+
! CANMOS ISP Network                        !
+-------------------------------------------+
! Best regards                              !
! Igor V. Ruzanov, network operational staff!
! e-Mail: igorr at canmos.ru                   !
+-------------------------------------------+


More information about the freebsd-questions mailing list