KQueue dropping events?

[Brandon Erhart]

Hi,

I am writing a web sucker downer (mirror) for a project on indexing the web 
(got myself a 1TB raid, just gonna d/l text ..). I am using the KQueue API 
in FreeBSD 4.9-REL to take care of watching over my sockets. I seem to be 
running into a nasty problem, however.

Here's a scenario. I set the outgoing connections to, say, 5000. The 
problem is, the amount of connections my program shows as being connected 
is roughly 1/5 to sometimes even 1/8th of what is actually connected. I see 
what is "actually connected" by doing a netstat. The program would say 
750/5000 connections, while a netstat would show 4500 connections in the 
ESTABLISHED state.

You may be saying, "it must be a bug in your connection tracking logic". I 
honestly don't think that's it. I have only TWO places in my code where I 
check if the connection was successful by calling getpeername(), if it 
wasnt, i return back to the main loops, otherwise I know it connected (and 
therefore increasing the global connection counter) -- in the callback 
function for data read() from KQueue-monitored fds (all of the sockets), 
and then also in my main loop (I check for read, write and connect timeouts 
there, right after my call to kevent()).

Basically the main loop looks like, in psuedo-code of course:

while (there_is_still_events)
{
	if (kevent())   <-- i pull 16 events from the kqueue
	{
		execute_the_callback_function;
	}

	check_for_read()_write()_and_connect()_timeouts;

	check_for_"client_descriptors(basically just a structure that holds info 
on the kqueue event)"_that_need_to_be_connected_to_the_next_server_and_do_so;
}

It's pretty straight forward. I have no idea why my program would be 
reporting a smaller amount. Is it possible that my program is not getting 
ALL the information it needs from kevent()? Perhaps the KQueue is becoming 
"full"? Is this possible? Should I be pulling more than the 16 events off 
the kqueue at a time?

I have been up ALL NIGHT trying to debug this, and cannot figure it out.

Any and all help is appreciated!

Thanks,

Brandon