layer2 ipfw 'fwd' support
Eduardo Meyer
dudu.meyer at gmail.com
Fri Oct 8 15:55:56 UTC 2010
On Thu, Oct 7, 2010 at 10:23 PM, Eduardo Meyer <dudu.meyer at gmail.com> wrote:
> On Thu, Oct 7, 2010 at 12:19 AM, Julian Elischer <julian at freebsd.org> wrote:
>> On 10/6/10 12:06 PM, Eduardo Meyer wrote:
>>>
>>> On Tue, Oct 5, 2010 at 5:31 PM, Julian Elischer<julian at freebsd.org>
>>> wrote:
>>>>
>>>> On 10/5/10 12:56 PM, Eduardo Meyer wrote:
>>>>>
>>>>> On Mon, Oct 4, 2010 at 6:23 PM, Julian Elischer<julian at freebsd.org>
>>>>> wrote:
>>>>>>
>>>>>> On 10/4/10 12:18 PM, Eduardo Meyer wrote:
>>>>>>>
>>>>>>> On Mon, Oct 4, 2010 at 3:35 PM, Julian Elischer<julian at freebsd.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On 10/4/10 10:16 AM, Eduardo Meyer wrote:
>>>>>>>>>
>>>>>>>>> On Mon, Oct 4, 2010 at 2:02 PM, Brandon Gooch
>>>>>>>>> <jamesbrandongooch at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 4, 2010 at 9:44 AM, Eduardo Meyer<dudu.meyer at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> In the past I have used this patch by Luigi Rizzo, which helped me
>>>>>>>>>>> well.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://lists.freebsd.org/pipermail/freebsd-ipfw/2003-September/000526.html
>>>>>>>>>>>
>>>>>>>>>>> I tried with a friend to port it to -STABLE, but we were not able
>>>>>>>>>>> to
>>>>>>>>>>> find out what has replaced mt_tag. Also on ip_input.c we dirty
>>>>>>>>>>> hacked
>>>>>>>>>>> to following piece of code:
>>>>>>>>>>>
>>>>>>>>>>> #ifdef IPFIREWALL_FORWARD
>>>>>>>>>>> if (m->m_flags& M_FASTFWD_OURS) {
>>>>>>>>>>> m->m_flags&= ~M_FASTFWD_OURS;
>>>>>>>>>>> goto pass; /* XXX was 'ours' - SHOULD WE MODIFY IT
>>>>>>>>>>> HERE
>>>>>>>>>>> */
>>>>>>>>>>> }
>>>>>>>>>>> if ((dchg = (m_tag_find(m, PACKET_TAG_IPFORWARD, NULL) !=
>>>>>>>>>>> NULL))
>>>>>>>>>>> != 0) {
>>>>>>>>>>> /*
>>>>>>>>>>> * Directly ship the packet on. This allows
>>>>>>>>>>> forwarding
>>>>>>>>>>> * packets originally destined to us to some other
>>>>>>>>>>> directly
>>>>>>>>>>> * connected host.
>>>>>>>>>>> */
>>>>>>>>>>> ip_forward(m, dchg);
>>>>>>>>>>> return;
>>>>>>>>>>> }
>>>>>>>>>>> #endif /* IPFIREWALL_FORWARD */
>>>>>>>>>>>
>>>>>>>>>>> And this is something we are not sure if its correct.
>>>>>>>>>>>
>>>>>>>>>>> So my very obvious question is:
>>>>>>>>>>>
>>>>>>>>>>> Does anyone has a recent version of this patch to share?
>>>>>>>>>>>
>>>>>>>>>>> Can anyone familiar with ipfw source code help me with that?
>>>>>>>>>>>
>>>>>>>>>> I'm certainly not an expert, but I wonder if the patch your
>>>>>>>>>> referring
>>>>>>>>>> to is still required? Can you provide more detail about your
>>>>>>>>>> particular application?
>>>>>>>>>>
>>>>>>>>>> -Brandon
>>>>>>>>>
>>>>>>>>> Yes, its still required since ipfw fwd ignores layer2 frames.
>>>>>>>>>
>>>>>>>>> The application is the very same: squid. I mean, Lusca in fact
>>>>>>>>> (squid
>>>>>>>>> fork).
>>>>>>>>>
>>>>>>>>> Thank you for your interest.
>>>>>>>>
>>>>>>>> Cisco/Ironport have a patch that does this..
>>>>>>>> I had permission to bring it back when I worked there but never got
>>>>>>>> it
>>>>>>>> committed.
>>>>>>>>
>>>>>>>> Adrian, was it part of the set I gave you?
>>>>>>>
>>>>>>> Hello Elischer,
>>>>>>>
>>>>>>> Was this made public?
>>>>>>>
>>>>>>> I hope Chadd has some good news. In fact I tent to use with Lusca in
>>>>>>> tproxy mode. I bet this is the only missing piece of software.
>>>>>>>
>>>>>> I just dug up my old changes.
>>>>>> do you want to fwd from a bridge? or what?
>>>>>> (it makes a difference what patches are needed)
>>>>>>
>>>>>> If you want to fwd from a bridge to make a transparent layer 2 proxy,
>>>>>> this
>>>>>> may help..
>>>>>>
>>>>>>
>>>>>> Here are parts of it that may be relevent:
>>>>>> these are old (2007 I think) but may be of use still.
>>>>>>
>>>>>> adrian had the full set at
>>>>>>
>>>>>> ==quote adrian=====
>>>>>> The stuff is in p4 now, but I haven't tested it out at all.
>>>>>>
>>>>>> //depo/projects/adrian_spoof_clientip/ I -think-.
>>>>>> == end quote===
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Index: net/if_bridge.c
>>>>>> ===================================================================
>>>>>> RCS file: /usr/local/cvsroot/freebsd/src/sys/net/if_bridge.c,v
>>>>>> retrieving revision 1.107
>>>>>> diff -u -r1.107 if_bridge.c
>>>>>> --- net/if_bridge.c 6 Nov 2007 23:01:42 -0000 1.107
>>>>>> +++ net/if_bridge.c 28 Nov 2007 06:59:10 -0000
>>>>>> @@ -2908,6 +2908,11 @@
>>>>>> struct ip *ip;
>>>>>> struct llc llc1;
>>>>>> u_int16_t ether_type;
>>>>>> + int is_ip = 0;
>>>>>> +#ifdef IPFIREWALL_FORWARD
>>>>>> + struct m_tag *fwd_tag;
>>>>>> +#endif
>>>>>> +
>>>>>>
>>>>>> snap = 0;
>>>>>> error = -1; /* Default error if not error == 0 */
>>>>>> @@ -2967,6 +2972,7 @@
>>>>>> #ifdef INET6
>>>>>> case ETHERTYPE_IPV6:
>>>>>> #endif /* INET6 */
>>>>>> + is_ip = 1;
>>>>>> break;
>>>>>> default:
>>>>>> /*
>>>>>> @@ -3024,6 +3030,30 @@
>>>>>>
>>>>>> if (*mp == NULL)
>>>>>> return (error);
>>>>>> +
>>>>>> +#ifdef IPFIREWALL_FORWARD
>>>>>> + /*
>>>>>> + * Did the firewall want to forward it somewhere?
>>>>>> + * If so, let the ip stack handle it.
>>>>>> + */
>>>>>> + if (i == 0&& args.next_hop != NULL&&
>>>>>> + is_ip /*&& src != NULL */) {
>>>>>> +
>>>>>> + fwd_tag = m_tag_get(PACKET_TAG_IPFORWARD,
>>>>>> + sizeof(struct sockaddr_in),
>>>>>> M_NOWAIT);
>>>>>> + if (fwd_tag == NULL)
>>>>>> + goto drop;
>>>>>> + bcopy(args.next_hop, (fwd_tag+1),
>>>>>> + sizeof(struct sockaddr_in));
>>>>>> + m_tag_prepend(*mp, fwd_tag);
>>>>>> +
>>>>>> + if (in_localip(args.next_hop->sin_addr))
>>>>>> + (*mp)->m_flags |= M_FASTFWD_OURS;
>>>>>> + ether_demux(src, *mp);
>>>>>> + return (NULL);
>>>>>> + }
>>>>>> +#endif
>>>>>> +
>>>>>>
>>>>>> if (DUMMYNET_LOADED&& (i == IP_FW_DUMMYNET)) {
>>>>>>
>>>>>> ==================
>>>>>> Index: netinet/ip_fw2.c
>>>>>> ===================================================================
>>>>>> RCS file: /usr/local/cvsroot/freebsd/src/sys/netinet/ip_fw2.c,v
>>>>>> retrieving revision 1.178
>>>>>> diff -u -r1.178 ip_fw2.c
>>>>>> --- netinet/ip_fw2.c 28 Oct 2007 17:12:47 -0000 1.178
>>>>>> +++ netinet/ip_fw2.c 28 Nov 2007 06:59:10 -0000
>>>>>>
>>>>>> @@ -3446,8 +3507,10 @@
>>>>>> case O_FORWARD_IP: {
>>>>>> struct sockaddr_in *sa;
>>>>>> sa =&(((ipfw_insn_sa *)cmd)->sa);
>>>>>> +#if 0
>>>>>> if (args->eh) /* not valid on layer2
>>>>>> pkts
>>>>>> */
>>>>>> break;
>>>>>> +#endif
>>>>>> if (!q || dyn_dir == MATCH_FORWARD) {
>>>>>> if (sa->sin_addr.s_addr ==
>>>>>> INADDR_ANY) {
>>>>>>
>>>>>> bcopy(sa,&args->hopstore,
>>>>>>
>>>>>> =============================================
>>>>>> Index: netinet/ip_output.c
>>>>>
>>>>> Dear Julian,
>>>>>
>>>>> Is anything missing from the above code? Say, like ip_output stuff?
>>>>>
>>>>> I have tried what you sent me, compiled fine but did not work.
>>>>>
>>>>> Here is my only rule (I have tried both with and without layer2 on the
>>>>> rule):
>>>>>
>>>>> 00001 36 4338 fwd 127.0.0.1,80 tcp from any to not me
>>>>> dst-port 80 layer2
>>>>> 65535 32842101 2107060460 allow ip from any to any
>>>>>
>>>>> Here are the sysctl tunables:
>>>>>
>>>>> net.link.bridge.ipfw: 1
>>>>> net.link.bridge.inherit_mac: 0
>>>>> net.link.bridge.log_stp: 0
>>>>> net.link.bridge.pfil_local_phys: 0
>>>>> net.link.bridge.pfil_member: 0
>>>>> net.link.bridge.pfil_bridge: 1
>>>>> net.link.bridge.ipfw_arp: 0
>>>>> net.link.bridge.pfil_onlyip: 0
>>>>> net.link.ether.inet.log_arp_permanent_modify: 1
>>>>> net.link.ether.inet.log_arp_movements: 1
>>>>> net.link.ether.inet.log_arp_wrong_iface: 1
>>>>> net.link.ether.inet.proxyall: 0
>>>>> net.link.ether.inet.useloopback: 1
>>>>> net.link.ether.inet.maxtries: 5
>>>>> net.link.ether.inet.max_age: 1200
>>>>> net.link.ether.ipfw: 1
>>>>>
>>>>> And my bridge:
>>>>>
>>>>> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>>>>> mtu
>>>>> 1500
>>>>> ether 16:52:8e:91:2f:45
>>>>> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>>>>> maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
>>>>> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>>>>> member: vr0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>>>>> ifmaxaddr 0 port 5 priority 128 path cost 200000
>>>>> member: sis0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>>>>> ifmaxaddr 0 port 1 priority 128 path cost 200000
>>>>>
>>>>> The ipfw counter gets increased by nothing hits by Apache. Instead I
>>>>> go to Internet directly.
>>>>>
>>>>> sis0 is on internet, vr0 is cross-over to the laptop (customer).
>>>>>
>>>>> How should I debug it?
>>>>>
>>>>>
>>>> basically I woud suggest code inspection for a start..
>>>>
>>>> look at where ipfw is called (just before where the patch went in) and
>>>> follow the packet up into ipfw
>>>> and back, and read what it would do..
>>>>
>>>> It's actually not a very hard path to follow.
>>>>
>>>> I'll try look at it after work..
>>>
>>> Hello Julian / Adrian.
>>>
>>> Thank you for your attention. A friend added some log entries so we
>>> could try to find out what gets run and what doesnt.
>>>
>>> Here is my current patch against RELENG_8:
>>>
>>> --- if_bridge.c.orig 2010-09-11 22:02:36.000000000 +0000
>>> +++ if_bridge.c 2010-10-05 17:59:13.000000000 +0000
>>> @@ -2957,6 +2957,13 @@
>>> struct ip *ip;
>>> struct llc llc1;
>>> u_int16_t ether_type;
>>> + int is_ip = 0;
>>> +#ifdef IPFIREWALL_FORWARD
>>> + struct m_tag *fwd_tag;
>>> +#endif
>>> +
>>> +
>>> +
>>>
>>> snap = 0;
>>> error = -1; /* Default error if not error == 0 */
>>> @@ -3016,6 +3023,8 @@
>>> #ifdef INET6
>>> case ETHERTYPE_IPV6:
>>> #endif /* INET6 */
>>> + is_ip=1;
>>> + log(LOG_NOTICE, "Entered 0: is_ip=%i\n",is_ip);
>>> break;
>>> default:
>>> /*
>>> @@ -3091,6 +3100,32 @@
>>> if (*mp == NULL)
>>> return (error);
>>>
>>> +#ifdef IPFIREWALL_FORWARD
>>> + /*
>>> + * Did the firewall want to forward it somewhere?
>>> + * If so, let the ip stack handle it.
>>> + */
>>> + log(LOG_NOTICE, "Entered 1");
>>> + if (i == 0&& args.next_hop != NULL&&
>>> + is_ip /*&& src != NULL */) {
>>> + log(LOG_NOTICE, "Entered 2");
>>> +
>>> + fwd_tag = m_tag_get(PACKET_TAG_IPFORWARD,
>>> + sizeof(struct sockaddr_in),
>>> M_NOWAIT);
>>> + if (fwd_tag == NULL)
>>> + goto drop;
>>> + bcopy(args.next_hop, (fwd_tag+1),
>>> + sizeof(struct sockaddr_in));
>>> + m_tag_prepend(*mp, fwd_tag);
>>> +
>>> + if (in_localip(args.next_hop->sin_addr))
>>> + (*mp)->m_flags |= M_FASTFWD_OURS;
>>> + ether_demux(src, *mp);
>>> + return (NULL);
>>> + }
>>> +#endif
>>> +
>>> +
>>> if (ip_dn_io_ptr&& (i == IP_FW_DUMMYNET)) {
>>>
>>> /* put the Ethernet header back on */
>>> --- ../netinet/ipfw/ip_fw2.c.orig 2010-09-16 15:11:17.000000000
>>> +0000
>>> +++ ../netinet/ipfw/ip_fw2.c 2010-10-06 12:17:12.000000000 +0000
>>> @@ -2059,8 +2059,14 @@
>>> break;
>>>
>>> case O_FORWARD_IP:
>>> - if (args->eh) /* not valid on layer2
>>> pkts */
>>> +#if 0
>>> + /* not valid on layer2 pkts */
>>> + if (args->eh) {
>>> + log(LOG_NOTICE, "ip_fw2.c Entered
>>> 1");
>>> break;
>>> + }
>>> + log(LOG_NOTICE, "ip_fw2.c Entered 2");
>>
>> these will never happen as they are in the #if 0 section.
>>
>> the #if 0 is to REMOVE that code from being compiled.
>>
>>
>>> +#endif
>
> Hello Julian,
>
> Thank you again for your feedback. I appreciate it very much.
>
> On my understanding this "if 0" was to really ignore this portion of
> code, because as I understand what is does is to break (leave the
> loop) if the packet is on layer2, and this is something we would not
> want, but I guess I am wrong.
>
> I tested now with your suggestion, and what we get is:
>
> Oct 7 15:45:16 phoenix kernel: Entered 0: is_ip=1
> Oct 7 15:45:16 phoenix kernel: ip_fw2.c Entered 1
> Oct 7 15:45:16 phoenix kernel: Entered 0: is_ip=1
> Oct 7 15:45:50 phoenix last message repeated 29 times
> Oct 7 15:47:53 phoenix last message repeated 237 times
> Oct 7 15:57:56 phoenix last message repeated 1029 times
> Oct 7 16:02:51 phoenix last message repeated 655 times
> Oct 7 16:02:51 phoenix kernel: ip_fw2.c Entered 1
> Oct 7 16:02:51 phoenix kernel: Entered 0: is_ip=1
> Oct 7 16:03:23 phoenix last message repeated 54 times
> Oct 7 16:05:24 phoenix last message repeated 345 times
> Oct 7 16:15:26 phoenix last message repeated 1135 times
> Oct 7 16:15:33 phoenix last message repeated 8 times
>
> So yes, we entered on ipfw code now, and executed only the instruction
> before we "break".
>
> The curious thing is that the counter did not count now with both:
>
> 00001 0 0 fwd 127.0.0.1,80 tcp from any to not me dst-port 80 layer2
> 00001 0 0 fwd 127.0.0.1,80 tcp from any to not me dst-port 80
>
> How can I move forth?
Hello,
I am glad to tell you that some helped me out and we made it work. In
fact two friends called Luiz Otavio (he has helped on IP_BINDANY
before, on lusca's tproxy support) and Patrick Tracanelli sorted out
the missing piece of code and shown up with this patch:
Index: netinet/ipfw/ip_fw2.c
===================================================================
--- netinet/ipfw/ip_fw2.c (revision 213573)
+++ netinet/ipfw/ip_fw2.c (working copy)
@@ -2059,8 +2059,10 @@
break;
case O_FORWARD_IP:
+#if 0
if (args->eh) /* not valid on layer2 pkts */
break;
+#endif
if (!q || dyn_dir == MATCH_FORWARD) {
struct sockaddr_in *sa;
sa = &(((ipfw_insn_sa *)cmd)->sa);
Index: net/if_bridge.c
===================================================================
--- net/if_bridge.c (revision 213573)
+++ net/if_bridge.c (working copy)
@@ -79,6 +79,7 @@
#include "opt_inet.h"
#include "opt_inet6.h"
+#include "opt_ipfw.h"
#include <sys/param.h>
#include <sys/mbuf.h>
@@ -2951,14 +2952,18 @@
static int
bridge_pfil(struct mbuf **mp, struct ifnet *bifp, struct ifnet *ifp, int dir)
{
- int snap, error, i, hlen;
+ int snap, error, i, is_ip, hlen;
struct ether_header *eh1, eh2;
struct ip_fw_args args;
struct ip *ip;
struct llc llc1;
u_int16_t ether_type;
+#ifdef IPFIREWALL_FORWARD
+ struct m_tag *fwd_tag;
+#endif
snap = 0;
+ is_ip = 0;
error = -1; /* Default error if not error == 0 */
#if 0
@@ -3016,6 +3021,7 @@
#ifdef INET6
case ETHERTYPE_IPV6:
#endif /* INET6 */
+ is_ip = 1;
break;
default:
/*
@@ -3091,6 +3097,46 @@
if (*mp == NULL)
return (error);
+#ifdef IPFIREWALL_FORWARD
+ /*
+ * Did the firewall want to forward it somewhere?
+ * If so, let the ip stack handle it.
+ */
+ if (i == 0 && args.next_hop != NULL && is_ip) {
+
+ fwd_tag = m_tag_get(PACKET_TAG_IPFORWARD,
+ sizeof(struct sockaddr_in), M_NOWAIT);
+ if (fwd_tag == NULL)
+ return (error);
+ bcopy(args.next_hop, (fwd_tag+1),
+ sizeof(struct sockaddr_in));
+ m_tag_prepend(*mp, fwd_tag);
+
+ if (in_localip(args.next_hop->sin_addr))
+ (*mp)->m_flags |= M_FASTFWD_OURS;
+
+ /*
+ * Put everything back the way it was and reinject the
+ * packet.
+ */
+ if (snap) {
+ M_PREPEND(*mp, sizeof(struct llc), M_DONTWAIT);
+ if (*mp == NULL)
+ return (error);
+ bcopy(&llc1, mtod(*mp, caddr_t),
+ sizeof(struct llc));
+ }
+
+ M_PREPEND(*mp, ETHER_HDR_LEN, M_DONTWAIT);
+ if (*mp == NULL)
+ return (error);
+ bcopy(&eh2, mtod(*mp, caddr_t), ETHER_HDR_LEN);
+
+ ether_demux(ifp, *mp);
+ return (error);
+ }
+#endif
+
if (ip_dn_io_ptr && (i == IP_FW_DUMMYNET)) {
/* put the Ethernet header back on */
Luiz has added it to: http://loos.no-ip.org:280/lusca_bridge.diff
I have tested and it works pretty well.
I hope someone can add it to -HEAD, so we won't loose it again. With
time, ipfw code changes and such great patches like Rizzo's and
Julian's stop working one day. It's bad we miss such great
functionality.
Thank you again everyone envolved.
Adrian / Luiz / Julian,
With this patch fwd does it's job on L2, ordinary proxy works like a
charm. But TPROXY won't work. It would be perfect to have both
features together. If you can suggest any further tests or changes I
will be pleased to test.
Thanks.
--
===========
Eduardo Meyer
pessoal: dudu.meyer at gmail.com
profissional: ddm.farmaciap at saude.gov.br
More information about the freebsd-ipfw
mailing list