dummynet configuration for automated tests

Hi,

I’ve been trying (and failing) to write a few basic test cases for 
dummynet (with ipfw for now).

The full test script can be found here: 
https://people.freebsd.org/~kp/dummynet.sh but the relevant bit is this:

	queue_v6_body()
	{
	        fw=$1
	        firewall_init $fw
	        dummynet_init $fw

	        epair=$(vnet_mkepair)
	        epair_link=$(vnet_mkepair)
	        vnet_mkjail alcatraz ${epair}b ${epair_link}a
	        vnet_mkjail srv ${epair_link}b

	set -x

	        ifconfig ${epair}a inet6 2001:db8:42::1/64 no_dad up
	        route add -6 2001:db8:43::/64 2001:db8:42::2

	        jexec alcatraz ifconfig ${epair}b inet6 2001:db8:42::2 no_dad 
up
	        jexec alcatraz ifconfig ${epair_link}a inet6 2001:db8:43::2 
no_dad up
	        jexec alcatraz sysctl net.inet6.ip6.forwarding=1

	        jexec srv ifconfig ${epair_link}b inet6 2001:db8:43::1 no_dad 
up
	        jexec srv route add -6 default 2001:db8:43::2
	        jexec srv /usr/sbin/inetd -p inetd-alcatraz.pid \
	            $(atf_get_srcdir)/../pf/echo_inetd.conf

	        # Sanity check
	        atf_check -s exit:0 -o ignore ping6 -i .1 -c 3 -s 1200 
2001:db8:42::2
	        atf_check -s exit:0 -o ignore ping6 -i .1 -c 3 -s 1200 
2001:db8:43::2
	        atf_check -s exit:0 -o ignore ping6 -i .1 -c 3 -s 1200 
2001:db8:43::1

	        reply=$(echo "foo" | nc -w 5 -N 2001:db8:43::1 7)
	        if [ "$reply" != "foo" ];
	        then
	                atf_fail "Echo sanity check failed"
	        fi

	        jexec alcatraz dnctl pipe 1 config bw 300Byte/s queue 5 mask 
proto 0xff
	        jexec alcatraz dnctl sched 1 config pipe 1 type wf2q+ mask 
proto 0xff
	        jexec alcatraz dnctl queue 1 config sched 1 weight 99 queue 5 
mask proto 0xff
	        jexec alcatraz dnctl queue 2 config sched 1 weight 1 queue 5 
mask proto 0xff

	        firewall_config alcatraz ${fw} \
	                "ipfw"  \
	                        "ipfw add queue 2 ipv6-icmp from any to any 
icmp6types 128,129" \
	                        "ipfw add queue 1 tcp from any to any"

	        # Single ping succeeds
	        atf_check -s exit:0 -o ignore ping6 -c 3 2001:db8:43::1
	        # Unsaturated TCP succeeds
	        reply=$(echo "foo" | nc -w 5 -N 2001:db8:43::1 7)
	        if [ "$reply" != "foo" ];
	        then
	                atf_fail "Unsaturated echo failed"
	        fi

	        # Saturate the link
	        ping6 -i .01 -s 1200 2001:db8:43::1 &

	        # Give that a chance to fill the queue & pipe
	        sleep 1

	        jexec alcatraz ipfw show

	        # We should now be hitting the limits and get this packet 
dropped.
	        atf_check -s exit:2 -o ignore ping6 -c 1 -W 1 -s 1200 
2001:db8:43::1

	        # TCP should still just pass
	        for i in `seq 0 4`
	        do
	                reply=$(echo "foo $i" | nc -w 10 -N 2001:db8:43::1 7)
	                if [ "$reply" != "foo $i" ];
	                then
	                        atf_fail "Failed to prioritise traffic on 
interation $i"
	                fi
	                sleep 1
	        done

	        jexec alcatraz ipfw flush
	        # This will fail if we don't differentiate the traffic
	        firewall_config alcatraz ${fw} \
	                "ipfw"  \
	                        "ipfw add queue 1 ipv6-icmp from any to any 
icmp6types 128,129" \
	                        "ipfw add queue 2 tcp from any to any"

	        # Carry over state?
	        killall ping6
	        ping6 -i .01 -s 1200 2001:db8:43::1 &
	        sleep 1

	        reply=$(echo "baz" | nc -w 10 -N 2001:db8:43::1 7)
	        if [ "$reply" == "baz" ];
	        then
	                jexec alcatraz ipfw show
	                atf_fail "TCP still made it through, even when not 
prioritised"
	        fi
	}

The idea is to set up a very slow link (using a pipe), and then to send 
both ICMP echo and TCP traffic through it. There’s vastly more ICMP 
traffic than TCP, and the expectation is that without prioritisation the 
ICMP traffic will drown out TCP and cause the connection to fail.
We then try to use dummynet to give TCP priority over ICMP, so that the 
TCP connections do succeed.

However, I simply cannot get it to behave in any sort of predictable or 
consistent way. Sometimes the TCP connection succeeds, despite attempts 
to prioritise ICMP, or vice versa.

Clearly I’m misconfiguring something, but at this point I do not 
understand what. Does anyone see my mistake, or have any relevant 
configuration examples to share?

Thanks,
Kristof