Network Packet drops in FreeBSD 5.2.1

[Anand Subramanian]

Hi All,
I am using an Intel Celeron box (single CPU,1.7GHz, 495MB real and 472 MB
avail memory, FreeBSD 5.2.1 #15 release), to run a daemon process which
shares a circular queue/buffer with the kernel. The daemon drains objects
off the front of the queue while the queue objects are populated by the
protocol processing function say, XXX_input() in the kernel, called by
ip_input(). If the front and rear indices of the shared buffer are equal
then the protocol stack drops the packet alright.

This model works fine for packet (60-64 bytes in size) input rates upto
11500 packets/sec, after which packets are lost. The machine running the
daemon uses the SiS 900 NIC, 10/100Mbps.

The packet loss was detected using the "netstat -I sis0" command, run both
on starting the daemon and upon shutting the daemon down. The Ipkts field
in the netstat output should indicate the number of packets received by
the interface in question..

With an appropriate chosen value for the shared buffer length, no packets
are dropped because the shared queue is full. Hence packets seem to be
dropped at the adapter level. The surprising part seems to be that though
packets are being dropped/lost, top shows a ~70% idle system with peak
interrupt time of ~25%.

The daemon uses a "hacked" version of the select() call with a timeout
value. The XXX_input() protocol processing function signals the thread/KSE
waiting on the hacked select() call as soon as it sees that there are
packets in the shared buffer(shared between the daemon and the kernel).

Question is :

1. Is top really accurate in reporting all stats at such workloads, or
input pkt rates? Can the %Idle time reported by top be trusted?

2. At increasing network input loads (12000 pkts/sec), much of the system
time maybe spent in the hardware interrupt handler, ip processing
functions. With the user daemon calling select(), any time spent in the
select() call would be charged to the daemon's timeslice. Would it be
fairly scheduled to run. It should be(of course depending on the RPLs),
but wanted to confirm this.

3. When ip_drain() is called and it calls the DEQUEUE macro, it acquires
Giant. DOes this mean other netisr's and handlers are disabled so that the
queue gets emptied in a sort of batch-mode behavior?

4. I am trying different clock speeds by changing kern.hz in loader.conf.
Doesn't seem to help but I am still looking into this.

When network packets are being dropped at the interface level, is it
really necessary for the system to be ~0% idle???


Any other input is greatly appreciated.

Best,
Anand