Network Packet drops in FreeBSD 5.2.1

[Matt Freitag]

Anand Subramanian wrote:

> Hi All,
> I am using an Intel Celeron box (single CPU,1.7GHz, 495MB real and 472 MB
> avail memory, FreeBSD 5.2.1 #15 release), to run a daemon process which
> shares a circular queue/buffer with the kernel. The daemon drains objects
> off the front of the queue while the queue objects are populated by the
> protocol processing function say, XXX_input() in the kernel, called by
> ip_input(). If the front and rear indices of the shared buffer are equal
> then the protocol stack drops the packet alright.
>
> This model works fine for packet (60-64 bytes in size) input rates upto
> 11500 packets/sec, after which packets are lost. The machine running the
> daemon uses the SiS 900 NIC, 10/100Mbps.
>
>  
>
Try a nic that has hardware support for checksum and fragmentation 
offloading.  If I'm not mistaken that sis900 is an integrated nic? It 
was somewhat flaky for me.. If you're operating at 100Mbps, your line 
rate doesn't become saturated at 12kpps. In fact, 100Mbps isn't 
saturated until around ~140kpps from my own lab results.

> The packet loss was detected using the "netstat -I sis0" command, run 
> both
> on starting the daemon and upon shutting the daemon down. The Ipkts field
> in the netstat output should indicate the number of packets received by
> the interface in question..
>
> With an appropriate chosen value for the shared buffer length, no packets
> are dropped because the shared queue is full. Hence packets seem to be
> dropped at the adapter level. The surprising part seems to be that though
> packets are being dropped/lost, top shows a ~70% idle system with peak
> interrupt time of ~25%.
>
> The daemon uses a "hacked" version of the select() call with a timeout
> value. The XXX_input() protocol processing function signals the 
> thread/KSE
> waiting on the hacked select() call as soon as it sees that there are
> packets in the shared buffer(shared between the daemon and the kernel).
>
> Question is :
>
> 1. Is top really accurate in reporting all stats at such workloads, or
> input pkt rates? Can the %Idle time reported by top be trusted?
>  
>
I've seen some exceptions to the rule, yet overall it seems accurate - 
but I've never read the inter workings of top, so I'm no expert.

> 2. At increasing network input loads (12000 pkts/sec), much of the system
> time maybe spent in the hardware interrupt handler, ip processing
> functions. With the user daemon calling select(), any time spent in the
> select() call would be charged to the daemon's timeslice. Would it be
> fairly scheduled to run. It should be(of course depending on the RPLs),
> but wanted to confirm this.
>  
>
I'm not sure if it's just the NIC you're using, but you should probably 
enable device polling, perhaps even enable fastforwarding.

> 3. When ip_drain() is called and it calls the DEQUEUE macro, it acquires
> Giant. DOes this mean other netisr's and handlers are disabled so that 
> the
> queue gets emptied in a sort of batch-mode behavior?
>
> 4. I am trying different clock speeds by changing kern.hz in loader.conf.
> Doesn't seem to help but I am still looking into this.
>
> When network packets are being dropped at the interface level, is it
> really necessary for the system to be ~0% idle???
>
>  
>
I'm sure it's all interrupts, again a better NIC could help :>

> Any other input is greatly appreciated.
>
> Best,
> Anand
>
>
>  
>
HTH,
-mpf

> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to 
> "freebsd-hackers-unsubscribe at freebsd.org"
>
>  
>