ipfw (dummynet) adds delay, but not configured to do so

[Ian Smith]

On Thu, 5 Mar 2009, Sebastian Mellmann wrote:

 > >  > If I configure another rule (or like 10 more rules) that matches the
 > >  > packet, I can see the delay increasing.
 > >  > For example a delay of ~20ms, when I configure 10 pipes.
 > >  > Am I doing something wrong?
 > >
 > > Configuring more pipes shouldn't make any difference unless packets are
 > > made to traverse each of the pipes in turn.  That would imply having set
 > > net.inet.ip.fw.one_pass=0 (or having run 'ipfw disable one_pass') so
 > > that each packet is reinjected into the firewall at the following rule,
 > > after traversing each pipe; is that what you're doing?
 > 
 > Yes, I've set net.inet.ip.fw.one_pass=0 so packets are reinjected into the
 > firewall after passing a pipe.

Good; your results would have been pretty weird otherwise ..

 > > Also, without using a separate pipe for either traffic direction, you're
 > > using 'half-duplex' mode, as well described in ipfw(8) TRAFFIC SHAPING.

Paired pipes will speed things up.  Maybe not noticeably for pings (call 
and response work half-duplex) but for esp TCP it could be considerable.

 > > Output of 'sysctl net.inet.ip.fw.one_pass' and 'ipfw show' with your
 > > example of using multiple pipes?
 > 
 > [root@ ~/ipfw]# sysctl net.inet.ip.fw.one_pass
 > net.inet.ip.fw.one_pass: 0
 > 
 > [root@ ~/ipfw]# ipfw show
 > 00010        0          0 allow ip from any to any via lo0
 > 10000      122      11832 allow ip from any to any via em2
 > 10100        0          0 pipe 100 ip from 192.168.5.0/26 to 192.168.7.0/24 in via em0
 > 10200        0          0 pipe 200 ip from 192.168.7.0/24 to 192.168.5.0/26 out via em0
 > 10300      342      28728 pipe 500 ip from any to any via em0
 > 10400      359      36512 pipe 510 ip from any to any via em1
 > 10500        0          0 pipe 300 udp from 80.80.80.1 to 60.60.60.1 src-port 4000 dst-port 4000 via em1
 > 10600        0          0 pipe 305 udp from 60.60.60.1 to 80.80.80.1 src-port 4000 dst-port 4000 via em0
 > 10700        0          0 pipe 310 udp from 80.80.80.1 to 60.60.60.1 src-port 4001 dst-port 4001 via em1
 > 10800        0          0 pipe 315 udp from 60.60.60.1 to 80.80.80.1 src-port 4001 dst-port 4001 via em0
 > 65535 14144748 9784372451 allow ip from any to any

A note of caution: using 'via' unqualified can be tricky; 'via em0' on 
the inbound pass is the same as 'in recv em0', but 'via em0' on the 
outbound pass includes packets that came IN on em0 but are going out on 
any interface, as well as those that came in on any interface that are 
going OUT on em0.  I prefer specifying 'in recv' and 'out xmit' where 
there might be any ambiguity; this usually works naturally with pipes, 
where you'd normally have one pipe per flow direction anyway.

Hopefully increasing kern.hz solves your main issue, and worth trying 
the new! net.inet.ip.dummynet.io_fast too.  Let us know your results?

cheers, Ian