dummynet dropping too many packets

[Eugene Grosbein]

On Tue, Oct 06, 2009 at 06:14:58PM +0500, rihad wrote:

> >No, generally handles much more. Please show your ipfw rule(s)
> >containing 'tablearg'.
> 
> 01031         x            x allow ip from any to any
> 01040         x            x skipto 1100 ip from table(127) to any out 
> recv bce0 xmit bce1
> 01060         x            x pipe tablearg ip from any to table(0) out 
> recv bce0 xmit bce1
> 01070         x            x allow ip from any to table(0) out recv bce0 
> xmit bce1
> 01100         x            x pipe tablearg ip from any to table(2) out
> 65535         x            x allow ip from any to any
> 
> table(127) contains country-wide ISPs' netblocks (under 100 entries).
> table(0) and table(2) contain same user IP addresses, but different pipe 
> IDs - normally around 3-4k entries each.
> 
> Now please pay special attention to rule 1031. I've added it to bypass 
> dummynet and stop packets from being dropped for now. Normally the rule 
> isn't there.
> 
> As I found out today after rebooting, drops only start occurring when 
> the number of entries in table(0) exceeds 2000 or so (please see my 
> previous email). Maybe it's a coincidence - I don't know. Global traffic 
> load doesn't matter - it was approximately the same before and after the 
> drops (around 450 mbit/s).

It's possible that pipe lookup by its number is inefficient
and firewall keeps its lock for too much time while searching the pipe,
just a guess. And packets start to drop, eh?

Try setting net.isr.direct to 0 and make large net.inet.ip.intr_queue_maxlen.
This way, one of your cores may run bce's thread, enqueue incoming
packets and return to work immediately. The rest of processing may be
performed by another kernel thread, hopefully using another core.
Just to see if this changes anything. top -S should help here too.

Eugene