sys/dev/em/if_em.c

[Murat Balaban]

Hello hackers,

I have a special-purpose setting where I have a ng_hub like kernel module (ng_lb)
which I've been coding. The box I'm using has two em(4) adapters, and I've
hooked em0's lower with my ng_lb's link0, and em1's lower with ng_lb's link1.

Situation looks like this:

        lower            link0                 link1              lower
em0 ---------------> -------------> ng_lb --------------> ---------------> em1

Every packet that is received by em0 is handed over to my netgraph module and
after very little modification in the packet ethernet header (changing destination
mac addresss) I NG_FWD_ITEM() the packet to em1.

I'm generating traffic with a packet generator, and em0 seems to be ok with around
910 Mbit/s traffic.

However if I write the packets into em1, em1 seems to drop 40-60 Mbit/s (of 910 Mbit/s)
data. I digged the problem a bit, and found out that, IFQ_HANDOFF, called deep inside
from NG_FWD_ITEM was returning ENOBUFS.

A little more investigation proved me that the source of ENOBUFS error was that
the em1 was running out of Tx descriptors. The relative logic in dev/em/if_em.c
(em_encap) was that if # of Tx descriptors falls below a threshold, the driver
tries to clean transmit interrupts once. # of available Tx desc. is again checked
and if the number is still not incresed ENOBUFS error is returned.

What I'd like to ask is, instead of cleaning the transmit interrupts only once,
why not do it many times till the number of available tx descriptors increases 
to a moderate level?

The following patch solved my problem, though I wanted to get your opinions about
it.

Cheers, 
Murat
http://www.enderunix.org/murat/

PS: Both cards are plugged into a 64-bit 66 Mhz PCI-X bus. I've polling enabled
in both interfaces, and HZ set to 10000.


--- if_em_murat.c       Wed Jul 26 13:59:22 2006
+++ if_em.c     Wed Jul 26 14:01:11 2006
@@ -1177,11 +1177,9 @@
          * available hits the threshold
          */
         if (adapter->num_tx_desc_avail <= EM_TX_CLEANUP_THRESHOLD) {
-                em_clean_transmit_interrupts(adapter);
-                if (adapter->num_tx_desc_avail <= EM_TX_CLEANUP_THRESHOLD) {
-                        adapter->no_tx_desc_avail1++;
-                        return(ENOBUFS);
-                }
+               do {
+                       em_clean_transmit_interrupts(adapter);
+               while (adapter->num_tx_desc_avail <= EM_TX_CLEANUP_THRESHOLD);
         }

         /*