From nobody Thu Dec 15 22:27:06 2022 X-Original-To: freebsd-cloud@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NY6LT5C0hz4gd8h for ; Thu, 15 Dec 2022 22:27:45 +0000 (UTC) (envelope-from bilbo@hobbiton.org) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NY6LT3JPNz4Q0x for ; Thu, 15 Dec 2022 22:27:43 +0000 (UTC) (envelope-from bilbo@hobbiton.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x633.google.com with SMTP id vv4so2053122ejc.2 for ; Thu, 15 Dec 2022 14:27:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ofwilsoncreek-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=lJo9wZe2pXKdwomLd1sLMj8x934VtEq+oZ7v2bmsUgk=; b=CaJ5v8EBVE6MmuoK349WEUz84eeIJPviyC79pngaKa10ppCF28KfyGUhG8XLVF74Ho 93svjpaVRulu/ohEBM+tVwDQqut+QHiXvb5BBbS6IJtzY2PJWeoNqBELNG3PNhrea527 BTa8Je1KWq5lyitQYOTJaKuc5+Nf9FWSRC4jQTUkU4z/Eoq2qKaimYmKfmp4EVcbcDA/ NqLtWyoLu7RRcs57PAuAWisxy/1theze88VLACs5Py8jp89Sdf0IlVtHjGy2l9yn2FEB ufRdk2lJD8Y97ME/cRv62jtajH7fpZzsxrk9VEtVClUDregMLYsiW5b0v7NPWiy1CBom qVpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lJo9wZe2pXKdwomLd1sLMj8x934VtEq+oZ7v2bmsUgk=; b=ZUAhB9pd62D1hGyWYeaxAw57Cz4Q+k1acdkIIICQUHxZFtC9goSZIuYt5YNYZoDhKq AOlOvRWBfqsihkt3Tdd8LjmS8hnhJ3RjCAfJVMVyPFmm/2hBehs7al8LfNXXpBmzoLnG XPivMnHdI17Bkz/u5q2WfFZtQViU5SjN/uuj+ySRfyULuoQy38sBw8xa/WacNNZAoSDY PfUYEQCUKflxNrLqn2xXyxdmyvYvWzY4Wzpw1aO9fgnQeR6r7jMaDtDAZkflNdl9fuN5 ILe9arbqDK8mO4vz0Js1ykZXfFqdUzY/0U+VRFFmDCU0K6yD8IO7LRC9lmkcTOQvVU18 ObXw== X-Gm-Message-State: ANoB5plak0O/Tnu7IOAv4vnGQnzcCCotnWaHNRDuVP1WUK4bmfMz4E9s hVrVK289caSUhYpYVqJ3LyIO4d5b9YHs2qTNlVFgnA== X-Google-Smtp-Source: AA0mqf4gOPI/3kMS3H8xasXVkndc75m0GBsbHXjP/4n9jhLiX+J3kCgo/Gj/c0rY1EjN4aaub0Y8lyQ1O/OCZD3uqz8= X-Received: by 2002:a17:906:2854:b0:7ae:3684:84b0 with SMTP id s20-20020a170906285400b007ae368484b0mr73066163ejc.622.1671143262705; Thu, 15 Dec 2022 14:27:42 -0800 (PST) List-Id: FreeBSD on cloud platforms (EC2, GCE, Azure, etc.) List-Archive: https://lists.freebsd.org/archives/freebsd-cloud List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-cloud@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Leif Pedersen Date: Thu, 15 Dec 2022 16:27:06 -0600 Message-ID: Subject: Re: What is a VPC (google's specifically but it could be more general) really? To: Rob Ballantyne Cc: freebsd-cloud@freebsd.org Content-Type: multipart/alternative; boundary="0000000000008aeee105efe55c4d" X-Rspamd-Queue-Id: 4NY6LT3JPNz4Q0x X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000008aeee105efe55c4d Content-Type: text/plain; charset="UTF-8" Hi, I don't have a direct answer, but as a user I can confirm that OpenVPN in layer 3 mode works for me. I simply haven't tried it in layer 2 mode with GCE (because I've no need for layer 2 and it incurs the extra overhead of broadcast packets). Layer 2 mode probably won't work anyway because the MTU has to be reduced to 1460, unless you do that on all participating hosts. Point is, if that's an option for you it might be worth exploring. As a side note, I configure the tun devices with the same IP address at the vtnet device. That actually works perfectly, even though the two endpoints are on wildly different networks, and avoids maintaining DNS entries and routes for the point to point network. For example: vtnet0: flags=8943 metric 0 mtu 1460 inet 10.1.2.3 netmask 0xffffffff broadcast 10.1.2.3 inet 130.x.x.x netmask 0xffffffff broadcast 130.x.x.x tun5000: flags=8051 metric 0 mtu 1500 inet 10.1.2.3 --> 10.16.0.1 netmask 0xffffffff (The internal IP on my home router's ethernet interface is 10.16.0.1/16, overlapping with its tun interface.) -Leif On Thu, Dec 15, 2022 at 4:03 PM Rob Ballantyne wrote: > Hello, > > I have a question about what the internal structure and forwarding is > within Google's VPCs. > > I started into a project using OpenVPN to bind my home network to an > isolated VPC in Google's Cloud when I discovered the routing didn't work > quite the way I thought. I had assumed that VPCs would look like a private > VLAN (Layer2) into which Google's infrastructure would inject L3 router > interfaces and/or ip/ethernet filters. > > I set up a private VPC and two test FreeBSD boxes to test and see > exactly how VPC configures routing. > > First, I just used a standard install of 13.1 and the routing table > after everything is up and configured looks like: > > ---- > Internet: > Destination Gateway Flags Netif Expire > default 10.1.1.1 UGS vtnet0 > 10.1.1.1 link#1 UHS vtnet0 > 10.1.1.20 link#1 UH lo0 > 127.0.0.1 link#2 UH lo0 > ---- > > This looked a little unusual to me so (there was no link local route for > all the addresses in the VPC), I commented out the rc.conf entry > 'google_network_daemon_enable=YES' and setup the vtnet0 interface up > manually with: 'ifconfig_vtnet0="inet 10.1.1.20 netmask 255.255.255.0"' > The resulting routing table: > > ---- > Internet: > Destination Gateway Flags Netif Expire > 10.1.1.0/24 link#1 U vtnet0 > 10.1.1.20 link#1 UHS lo0 > 127.0.0.1 link#2 UH lo0 > ---- > > This configuration wasn't able to communicate. The latter routing table > looks more usual though, with a 10.1.1.0/24 route to the local link. > > So, it appears to me that VPCs are really configured to be a > point-to-point (star really) network where the Google router interface > (10.1.1.1 in this case) has to handle all forwarding between nodes of a > network. > > I've searched around the web to try and confirm this but there is scant > detail on how exactly forwarding works within a single VPC. > > My VPN project involved using a bastion VPN host that would have > terminated the VPN/SSL tunnel and routed traffic between my home network > and the isolated network behind the bastion. > > Before I make final decisions on configuration, I wanted to know if my > understanding is correct and whether there is any documentation on this > that I've somehow missed. > > FreeBSD is, of course, the host of choice for this operation! > > If anyone does know any details, any info would be greatly appreciated. > > Many Thanks, > Rob Ballantyne > --0000000000008aeee105efe55c4d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I don't have a direct answer, b= ut as a user I can confirm that OpenVPN in layer 3 mode works for me. I sim= ply haven't tried it in layer 2 mode with GCE (because I've no need= for layer 2 and it incurs the extra overhead of broadcast packets). Layer = 2 mode probably won't work anyway because the MTU has to be reduced to = 1460, unless you do that on all participating hosts. Point is, if that'= s an option for you it might be worth exploring.

A= s a side note, I configure the tun devices with the same IP address at the = vtnet device. That actually works perfectly, even though the two endpoints = are on wildly different networks, and avoids maintaining DNS entries and ro= utes for the point to point network.

For example:<= /div>
vtnet0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULT= ICAST> metric 0 mtu 1460
=C2=A0 =C2=A0 inet 10.1.2.3 netma= sk 0xffffffff broadcast 10.1.2.3
=C2=A0 =C2=A0 inet 130.x.x.x netmask 0x= ffffffff broadcast 130.x.x.x
tun5000: flags=3D8051<UP,POINTOPOINT,RUN= NING,MULTICAST> metric 0 mtu 1500
=C2=A0 =C2=A0 inet 10.1.2.3 --> = 10.16.0.1 netmask 0xffffffff

(The internal IP = on my home router's ethernet interface is 10.16.0.1/16, overlapping with its tun interface.)

-Leif





On Thu, Dec 15, 2022 at 4:03 PM Rob Ballantyne <robballantyne3@gmail.com> = wrote:
Hello,

=C2=A0 I have a question about what the internal structu= re and forwarding is within Google's VPCs.

=C2=A0 I started into= a project using OpenVPN to bind my home network to an isolated VPC in Goog= le's Cloud when I discovered the routing didn't work quite the way = I thought.=C2=A0 I had assumed that VPCs would look like a private VLAN (La= yer2) into which Google's infrastructure would inject L3 router interfa= ces and/or ip/ethernet filters.

=C2=A0 I set up a private VPC and tw= o test FreeBSD boxes to test and see exactly how VPC configures routing.=C2= =A0=C2=A0

=C2=A0 First, I just used a standard install of 13.1 and t= he routing table after everything is up and configured looks like:

-= ---
Internet:
Destination =C2=A0 =C2=A0 =C2=A0 =C2=A0Gateway =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Flags =C2=A0 =C2=A0 Netif Expire
default = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A010.1.1.1 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 UGS =C2=A0 =C2=A0 =C2=A0vtnet0
10.1.1.1 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 link#1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UHS =C2=A0 = =C2=A0 =C2=A0vtnet0
10.1.1.20 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#1 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UH =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0lo0
127.0.0.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#2 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UH =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0lo0
----

=C2=A0 This looked a little= unusual to me so (there was no link local route for all the addresses in t= he VPC), I commented out the rc.conf entry 'google_network_daemon_enabl= e=3DYES' and setup the vtnet0 interface up manually with: 'ifconfig= _vtnet0=3D"inet 10.1.1.20 netmask 255.255.255.0"'=C2=A0 The r= esulting routing table:

----
Internet:
Destination =C2=A0 =C2=A0= =C2=A0 =C2=A0Gateway =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Flags =C2=A0= =C2=A0 Netif Expire
10= .1.1.0/24 =C2=A0 =C2=A0 =C2=A0 =C2=A0link#1 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 U =C2=A0 =C2=A0 =C2=A0 =C2=A0vtnet0
10.1.1.20 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0link#1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UH= S =C2=A0 =C2=A0 =C2=A0 =C2=A0 lo0
127.0.0.1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0link#2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 UH =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0lo0
----

=C2=A0 This configuration wasn't = able to communicate. The latter routing table looks more usual though, with= a 10.1.1.0/24 route t= o the local link.

=C2=A0 So, it appears to me that VPCs=C2=A0are rea= lly configured to be a point-to-point (star really) network where the Googl= e router interface (10.1.1.1 in this case) has to handle all forwarding bet= ween nodes of a network.

=C2=A0 I've searched around the web to = try and confirm this but there is scant detail on how exactly forwarding wo= rks within a single VPC.

=C2=A0 My VPN project involved using a bast= ion VPN host that would have terminated the VPN/SSL tunnel and routed traff= ic between my home network and the isolated network behind the bastion.
=C2=A0 Before I make final decisions on configuration, I wanted=C2=A0to= know if my understanding is correct and whether there is any documentation= on this that I've somehow missed.

=C2=A0 FreeBSD is, of course,= the host of choice for this operation!

=C2=A0 If anyone does know a= ny details, any info would be greatly appreciated.

Many Thanks,
Rob= Ballantyne
--0000000000008aeee105efe55c4d--