kern/148260: pf rdr incompatible with dummynet

[Alexey Guskov]

>Number:         148260
>Category:       kern
>Synopsis:       pf rdr incompatible with dummynet
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jun 30 12:20:07 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Alexey Guskov
>Release:        7.2-RELEASE
>Organization:
Areal Company
>Environment:
FreeBSD test.local 7.2-RELEASE FreeBSD 7.2-RELEASE #1: Tue Jun 29 09:04:57 UTC 2010     root at tamaez.office1.a-real.ru:/usr/src/sys/i386/compile/GATE  i386
>Description:
The problem is the same as described here: http://lists.freebsd.org/pipermail/freebsd-net/2006-July/011076.html

I'm using two firewalls: ipfw for dummynet traffic shaping and pf for rules, forwards etc. When a dummynet pipe is used and rdr rule exists pf (i.e. for transparent proxy) tcp connections doesn't get established.

For example: we have a pipe for traffic going to user, and rdr rule for trasparent proxy. Ipfw processes incoming packets before pf. Here's how traffic goes thru kernel:

SYN packet:
[network interface] -> ... -> pfil_run_hooks() -> ipfw_check_in() -> pf_check_in() /the packet is being redirected and pf state entry is created/ -> ... -> [squid]

The packet is successfully delivered.

ACK packet:
[squid] -> ... -> ip_output() -> ... -> pfil_run_hooks() -> pf_check_out() /the packet is being unredirected using state table entry/ -> ipfw_check_out() -> ip_dn_io_ptr()

NB: ipfw stops processing the packet after it bin injected into dummynet. Heres what happens next:

ip_dn_ip_ptr() -> /some pipe vodoo magic/ -> dummynet_send() -> ip_output()

TADA! After coming out from pipe the packet is being re-injected into main packet-processing routine! Let's see what's gonna happen:

ip_output() -> ... -> pfil_run_hooks() -> pf_check_out() /pf cannot find entry in state table because the packet has already been unredirected and violently kills it/ -> OH SHI--

So, tcp connection never gets established.
>How-To-Repeat:
pf.conf
--
rdr on le1 proto tcp from { 192.168.1.0/24 } to any port { 80 } -> (le1) port 3128 
--

ipfw.conf
--
ipfw pipe 1 config bw 16KByte/s
ipfw 100 add pipe 1 all from any to 192.168.1.101
--

make ipfw process incoming packets before pf 
--- 
# ipfw disable firewall
# ipfw enable firewall 
--
>Fix:
When ipfw send a packet to dummynet it tags it with PACKET_TAG_DUMMYNET and saves rule number that triggered this action. When the packet comes to ipfw for the second time, ipfw can see that it already has been checked and continues processing from the next rule.

The problem is pf doesn't know that the packet already has been processed by him. So we'll teach him:

--- sys/contrib/pf/net/pf_ioctl.c	2010-06-14 06:09:06.000000000 +0400
+++ nsys/contrib/pf/net/pf_ioctl.c	2010-06-30 14:20:55.000000000 +0400
@@ -3636,6 +3637,7 @@
 	 */
 	struct ip *h = NULL;
 	int chk;
+  struct m_tag *dn_tag;
 
 	if ((*m)->m_pkthdr.len >= (int)sizeof(struct ip)) {
 		/* if m_pkthdr.len is less than ip header, pf will handle. */
@@ -3643,7 +3645,13 @@
 	        HTONS(h->ip_len);
 	        HTONS(h->ip_off);
 	}
-	chk = pf_test(PF_IN, ifp, m, NULL, inp);
+	
+	dn_tag = m_tag_find(*m, PACKET_TAG_DUMMYNET, NULL);
+	if (dn_tag == NULL)
+    chk = pf_test(PF_IN, ifp, m, NULL, inp);
+	else
+    chk = PF_PASS;
+
 	if (chk && *m) {
 		m_freem(*m);
 		*m = NULL;
@@ -3671,6 +3679,7 @@
 	 */
 	struct ip *h = NULL;
 	int chk;
+	struct m_tag *dn_tag;
 
 	/* We need a proper CSUM befor we start (s. OpenBSD ip_output) */
 	if ((*m)->m_pkthdr.csum_flags & CSUM_DELAY_DATA) {
@@ -3683,7 +3692,13 @@
 	        HTONS(h->ip_len);
 	        HTONS(h->ip_off);
 	}
-	chk = pf_test(PF_OUT, ifp, m, NULL, inp);
+	
+	dn_tag = m_tag_find(*m, PACKET_TAG_DUMMYNET, NULL);
+	if (dn_tag == NULL)
+	  chk = pf_test(PF_OUT, ifp, m, NULL, inp);
+	else
+    chk = PF_PASS;
+	
 	if (chk && *m) {
 		m_freem(*m);
 		*m = NULL;


>Release-Note:
>Audit-Trail:
>Unformatted: