From nobody Fri Dec 16 00:10:53 2022 X-Original-To: freebsd-cloud@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NY8dk2CMGz4jZwM for ; Fri, 16 Dec 2022 00:11:06 +0000 (UTC) (envelope-from robballantyne3@gmail.com) Received: from mail-vs1-xe29.google.com (mail-vs1-xe29.google.com [IPv6:2607:f8b0:4864:20::e29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NY8dj5XCxz4X0w for ; Fri, 16 Dec 2022 00:11:05 +0000 (UTC) (envelope-from robballantyne3@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-vs1-xe29.google.com with SMTP id a66so855958vsa.6 for ; Thu, 15 Dec 2022 16:11:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=YD2tUvFCkegdykr91Qi6Exh0FiHXzH4PIzAcIXo8ckE=; b=e+ibecgphUOAEsefyRW037jTy5aL4OkcEAqmnhUpUfXuQ1XXR6ScZu+fNm6UR0uyBa Tu4TJR/g5hEMTCxYwylJG46Srw3cZhQiiWsBb8xm8QsI6MYtlpqJWC8IZ0jVEeMfUNTQ lt3Skexd1Mzo12GKk1RbGi4QqeTHn84FfJASrFaTTDg0Cjqob3+rRXKjRdjjZcZT1mBP xBNGd/6h0vX4jAOIIBHDzCX+v6SkfnVxngWIvmYvQL9RPKzC7iE4MqlsbYfwLexY73lZ 17fMD/nO5zIpbHmxYdWUZD6eKb4FVjA33HWyawcD5JPvWyAfukI9jruoWZ+SFF4RAFW2 gFww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YD2tUvFCkegdykr91Qi6Exh0FiHXzH4PIzAcIXo8ckE=; b=RdielJp3qsFzL/oGi7GMz3vc7nV7UtMJIF57hHP/R3qWLgOeuHBbs8Qq3jRlX3azaH sWJNfXyLX44gdb1NdObNYLtxc3sym2kqltPXtjE81LKoe62bUk50wp0DemcuvfU6z23q ecvUGI47wmfdlWx8A+qbKAKJAMjDCP9tod4cnPb6IKC4Cpv2kBojWJA3GDpGHeFKhb9q lfpGpqSPMNKkgEcjxDfChJZj6aaiUve2QSmiE6webkSVqPfquhezR8EGAbqXsaMt5fOC E6q21GBNrRms62cTTzn0/L9AnxovH3N931li4Z50w4gs40xqKzpw7es5Fm+sdQ6huZsu gaFA== X-Gm-Message-State: ANoB5pnnQj4gBiWuURe6EQOvSyh054GHgUjOJqsC0fJUsQbQpz8rEzc9 G06hmE7d/VzhJ6dwkDeGcovBD6M+xKTc8/MSD6lUCi5j X-Google-Smtp-Source: AA0mqf5SDCAjQU8xhqHKPPLbYD19keA6ATkTiJyyt7rgRFjqgX+REHvrzaedDN/2jixX5U7/QoQtd4cEX4LAYNTDYDU= X-Received: by 2002:a05:6102:807:b0:3b0:cc5c:a3f7 with SMTP id g7-20020a056102080700b003b0cc5ca3f7mr26410867vsb.1.1671149464890; Thu, 15 Dec 2022 16:11:04 -0800 (PST) List-Id: FreeBSD on cloud platforms (EC2, GCE, Azure, etc.) List-Archive: https://lists.freebsd.org/archives/freebsd-cloud List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-cloud@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rob Ballantyne Date: Thu, 15 Dec 2022 16:10:53 -0800 Message-ID: Subject: Re: What is a VPC (google's specifically but it could be more general) really? To: leif@ofwilsoncreek.com Cc: freebsd-cloud@freebsd.org Content-Type: multipart/alternative; boundary="00000000000038b1af05efe6ce04" X-Rspamd-Queue-Id: 4NY8dj5XCxz4X0w X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000038b1af05efe6ce04 Content-Type: text/plain; charset="UTF-8" Thank you Leif, I probably should have mentioned that I got the OpenVPN tunnel working as well. I was confused as to what was going on until I looked carefully at what Google had installed in the routing table and saw what ought to be a link local route (which would normally just be directed at a link#k entry in the Gateway Field) was actually directed to what I believe is the VPC router interface in the subnet (10.1.1.1 above). It's working now but I've got an uneasy feeling I haven't done it 'right.' If this were ordinary VLAN/Ethernet stuff, it would work like this too (I think) but it would be incurring an extra L3 hop through the router when it could have gone over the VLAN/Ethernet fabric direct. Thanks again! Rob On Thu, Dec 15, 2022 at 2:27 PM Leif Pedersen wrote: > Hi, > > I don't have a direct answer, but as a user I can confirm that OpenVPN in > layer 3 mode works for me. I simply haven't tried it in layer 2 mode with > GCE (because I've no need for layer 2 and it incurs the extra overhead of > broadcast packets). Layer 2 mode probably won't work anyway because the MTU > has to be reduced to 1460, unless you do that on all participating hosts. > Point is, if that's an option for you it might be worth exploring. > > As a side note, I configure the tun devices with the same IP address at > the vtnet device. That actually works perfectly, even though the two > endpoints are on wildly different networks, and avoids maintaining DNS > entries and routes for the point to point network. > > For example: > vtnet0: flags=8943 metric > 0 mtu 1460 > inet 10.1.2.3 netmask 0xffffffff broadcast 10.1.2.3 > inet 130.x.x.x netmask 0xffffffff broadcast 130.x.x.x > tun5000: flags=8051 metric 0 mtu 1500 > inet 10.1.2.3 --> 10.16.0.1 netmask 0xffffffff > > (The internal IP on my home router's ethernet interface is 10.16.0.1/16, > overlapping with its tun interface.) > > -Leif > > > > > > On Thu, Dec 15, 2022 at 4:03 PM Rob Ballantyne > wrote: > >> Hello, >> >> I have a question about what the internal structure and forwarding is >> within Google's VPCs. >> >> I started into a project using OpenVPN to bind my home network to an >> isolated VPC in Google's Cloud when I discovered the routing didn't work >> quite the way I thought. I had assumed that VPCs would look like a private >> VLAN (Layer2) into which Google's infrastructure would inject L3 router >> interfaces and/or ip/ethernet filters. >> >> I set up a private VPC and two test FreeBSD boxes to test and see >> exactly how VPC configures routing. >> >> First, I just used a standard install of 13.1 and the routing table >> after everything is up and configured looks like: >> >> ---- >> Internet: >> Destination Gateway Flags Netif Expire >> default 10.1.1.1 UGS vtnet0 >> 10.1.1.1 link#1 UHS vtnet0 >> 10.1.1.20 link#1 UH lo0 >> 127.0.0.1 link#2 UH lo0 >> ---- >> >> This looked a little unusual to me so (there was no link local route >> for all the addresses in the VPC), I commented out the rc.conf entry >> 'google_network_daemon_enable=YES' and setup the vtnet0 interface up >> manually with: 'ifconfig_vtnet0="inet 10.1.1.20 netmask 255.255.255.0"' >> The resulting routing table: >> >> ---- >> Internet: >> Destination Gateway Flags Netif Expire >> 10.1.1.0/24 link#1 U vtnet0 >> 10.1.1.20 link#1 UHS lo0 >> 127.0.0.1 link#2 UH lo0 >> ---- >> >> This configuration wasn't able to communicate. The latter routing table >> looks more usual though, with a 10.1.1.0/24 route to the local link. >> >> So, it appears to me that VPCs are really configured to be a >> point-to-point (star really) network where the Google router interface >> (10.1.1.1 in this case) has to handle all forwarding between nodes of a >> network. >> >> I've searched around the web to try and confirm this but there is scant >> detail on how exactly forwarding works within a single VPC. >> >> My VPN project involved using a bastion VPN host that would have >> terminated the VPN/SSL tunnel and routed traffic between my home network >> and the isolated network behind the bastion. >> >> Before I make final decisions on configuration, I wanted to know if my >> understanding is correct and whether there is any documentation on this >> that I've somehow missed. >> >> FreeBSD is, of course, the host of choice for this operation! >> >> If anyone does know any details, any info would be greatly appreciated. >> >> Many Thanks, >> Rob Ballantyne >> > --00000000000038b1af05efe6ce04 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thank you Leif,

=C2=A0 I probably should have mentione= d that I got the OpenVPN tunnel working as well.=C2=A0 I was confused as to= what was going on until I looked carefully at what Google had installed in= the routing table and saw what ought to be a link local route (which would= normally just be directed at a link#k entry in the Gateway Field) was actu= ally directed to what I believe is the VPC router interface in the subnet (= 10.1.1.1 above).

=C2=A0 It's working now but I've got an une= asy feeling I haven't done it 'right.'=C2=A0 If this were ordin= ary VLAN/Ethernet stuff, it would work like this too (I think) but it would= be incurring an extra L3 hop through the router when it could have gone ov= er the VLAN/Ethernet fabric direct.

=C2=A0 Thanks again!
Rob
<= /div>
O= n Thu, Dec 15, 2022 at 2:27 PM Leif Pedersen <leif@ofwilsoncreek.com> wrote:
Hi,

I don't have a direct answer, but as a user I can confirm that OpenV= PN in layer 3 mode works for me. I simply haven't tried it in layer 2 m= ode with GCE (because I've no need for layer 2 and it incurs the extra = overhead of broadcast packets). Layer 2 mode probably won't work anyway= because the MTU has to be reduced to 1460, unless you do that on all parti= cipating hosts. Point is, if that's an option for you it might be worth= exploring.

As a side note, I configure the tun de= vices with the same IP address at the vtnet device. That actually works per= fectly, even though the two endpoints are on wildly different networks, and= avoids maintaining DNS entries and routes for the point to point network.<= /div>

For example:
vtnet0: flags=3D8943<UP,= BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1460
=
=C2=A0 =C2=A0 inet 10.1.2.3 netmask 0xffffffff broadcast 10.1.2.3
= =C2=A0 =C2=A0 inet 130.x.x.x netmask 0xffffffff broadcast 130.x.x.x
tun5= 000: flags=3D8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500=
=C2=A0 =C2=A0 inet 10.1.2.3 --> 10.16.0.1 netmask 0xffffffff

(The internal IP on my home router's ethernet int= erface is 10.16.0.1/16, overlapping with its tun interface.)


On Thu, Dec 15= , 2022 at 4:03 PM Rob Ballantyne <robballantyne3@gmail.com> wrote:
Hello,

= =C2=A0 I have a question about what the internal structure and forwarding i= s within Google's VPCs.

=C2=A0 I started into a project using = OpenVPN to bind my home network to an isolated VPC in Google's Cloud wh= en I discovered the routing didn't work quite the way I thought.=C2=A0 = I had assumed that VPCs would look like a private VLAN (Layer2) into which = Google's infrastructure would inject L3 router interfaces and/or ip/eth= ernet filters.

=C2=A0 I set up a private VPC and two test FreeBSD bo= xes to test and see exactly how VPC configures routing.=C2=A0=C2=A0

= =C2=A0 First, I just used a standard install of 13.1 and the routing table = after everything is up and configured looks like:

----
Internet:Destination =C2=A0 =C2=A0 =C2=A0 =C2=A0Gateway =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0Flags =C2=A0 =C2=A0 Netif Expire
default =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A010.1.1.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UGS =C2= =A0 =C2=A0 =C2=A0vtnet0
10.1.1.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 link= #1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UHS =C2=A0 =C2=A0 =C2=A0vtnet0=
10.1.1.20 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#1 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 UH =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0lo0
127.0.0.1= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 UH =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0lo0
----

=C2=A0 This looked a little unusual to me so (th= ere was no link local route for all the addresses in the VPC), I commented = out the rc.conf entry 'google_network_daemon_enable=3DYES' and setu= p the vtnet0 interface up manually with: 'ifconfig_vtnet0=3D"inet = 10.1.1.20 netmask 255.255.255.0"'=C2=A0 The resulting routing tabl= e:

----
Internet:
Destination =C2=A0 =C2=A0 =C2=A0 =C2=A0Gateway= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Flags =C2=A0 =C2=A0 Netif Expire<= br>10.1.1.0/24 =C2=A0 = =C2=A0 =C2=A0 =C2=A0link#1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 U =C2= =A0 =C2=A0 =C2=A0 =C2=A0vtnet0
10.1.1.20 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0link#1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UHS =C2=A0 =C2=A0 =C2= =A0 =C2=A0 lo0
127.0.0.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#2 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UH =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0lo= 0
----

=C2=A0 This configuration wasn't able to communicate= . The latter routing table looks more usual though, with a 10.1.1.0/24 route to the local link.
=
=C2=A0 So, it appears to me that VPCs=C2=A0are really configured to b= e a point-to-point (star really) network where the Google router interface = (10.1.1.1 in this case) has to handle all forwarding between nodes of a net= work.

=C2=A0 I've searched around the web to try and confirm thi= s but there is scant detail on how exactly forwarding works within a single= VPC.

=C2=A0 My VPN project involved using a bastion VPN host that w= ould have terminated the VPN/SSL tunnel and routed traffic between my home = network and the isolated network behind the bastion.

=C2=A0 Before = I make final decisions on configuration, I wanted=C2=A0to know if my unders= tanding is correct and whether there is any documentation on this that I= 9;ve somehow missed.

=C2=A0 FreeBSD is, of course, the host of choic= e for this operation!

=C2=A0 If anyone does know any details, any in= fo would be greatly appreciated.

Many Thanks,
Rob Ballantyne
<= /div>
--00000000000038b1af05efe6ce04--