Pipes and wrong IPs

[Alexey Kouznetsov]

Hello!

Time to time a see at FreeBSD router some troubles with pipes

We have rooles:
...
20000 pipe tablearg ip from any to table(30) xmit em1 out
20000 pipe tablearg ip from table(31) to any recv em1 in
...

no more rooles where we use pipe or queue

in table 30 we have list of IPs, have to be shaped for incoming and in table 31 IPs we have to shape for outgoing. With parameters - the PIPE numbers

Also we have lot of pipes:
/sbin/ipfw pipe  122 config mask dst-ip 0xffffffff bw 32768Byte/s queue 32 
gred 0.002/8/16/0.1
/sbin/ipfw pipe 1122 config mask src-ip 0xffffffff bw 32768Byte/s queue 32 
gred 0.002/8/16/0.1

Each pair of pipes for their speeds

so, for example if we want to shape IP 10.10.10.10 to 32 MB for both directipons we will have 
ipfw table 30 add 10.10.10.10 122
ipfw table 31 add 10.10.10.10 1122

And.. time to time I see at output for 
ipfw pipe XXX show
wrong IPs, whch should not be at this pipe. Often I see IPs which should not be at any pape and which are not listen in 30/31 table at all

Just for now: (after 1 hour after reboot and i'm 100% sure such IP was not in tables 30/31 for this 1 hour)

$ sudo ipfw table 31 list | egrep 10.10.101.120
$ sudo ipfw table 30 list | egrep 10.10.101.120
$ sudo ipfw pipe list | egrep 10.10.101.120
12652 ip           0.0.0.0/0       10.10.101.120/0     8463  3827439  0    0  0
$ sudo ipfw pipe list | egrep 10.10.101.120
12652 ip           0.0.0.0/0       10.10.101.120/0     8478  3831640  0    0  0

So, traffic come to pipe (here, in example, this is pipe 101, but possible to be any pipe used in system). There no other calls for pipes in firewall rooles. Here are other tables used in firewall, where possible we have such IP, but other tables not checked here.

And, if we checks speed from IP 10.10.101.120, we wioll see this is really shaped to speed of pipe 101, so this is not print error, this is actually shaped by this pipe.

Some details about system:

FreeBSD xxx.xxx.xxx 7.2-STABLE FreeBSD 7.2-STABLE #0: Wed Aug 26 14:56:23 
MSD 2009     root at xxx.xxx.xxx:/usr/obj/usr/src/sys/xxxxxxxxx amd64
System was rebuild today, CVSUPed yesterday.after I found this problem.
First time I saw such problem 2 years (or about) ago on 7.0 amd64. but before I fixed it by reboot and problem come again not so often.... but now it come again and again. I never saw it at 6.x or 5.x.

also, 2 years ago here was not tablearg at my firewall set. here was lot of rooles like 
..
20000 pipe 122 ip from any to table(122) xmit em1 out
20000 pipe 1122 ip from table(122) to any recv em1 in
20000 pipe 123 ip from any to table(123) xmit em1 out
20000 pipe 1123 ip from table(123) to any recv em1 in
..

Also, here are some small tuning of pipes:

net.inet.ip.dummynet.hash_size=16384
net.inet.ip.dummynet.expire=0

ipfw and dumminet complied in the kernel.

sudo ipfw l | wc -l
ipfw: DEPRECATED: 'l' matched 'list' as a sub-string
      69

Any ideas?

sorry for my english. :<

With best regards
/Alexey