What is a VPC (google's specifically but it could be more general) really?

From: Rob Ballantyne <robballantyne3_at_gmail.com>
Date: Thu, 15 Dec 2022 22:03:32 UTC
Hello,

  I have a question about what the internal structure and forwarding is
within Google's VPCs.

  I started into a project using OpenVPN to bind my home network to an
isolated VPC in Google's Cloud when I discovered the routing didn't work
quite the way I thought.  I had assumed that VPCs would look like a private
VLAN (Layer2) into which Google's infrastructure would inject L3 router
interfaces and/or ip/ethernet filters.

  I set up a private VPC and two test FreeBSD boxes to test and see exactly
how VPC configures routing.

  First, I just used a standard install of 13.1 and the routing table after
everything is up and configured looks like:

----
Internet:
Destination        Gateway            Flags     Netif Expire
default            10.1.1.1           UGS      vtnet0
10.1.1.1           link#1             UHS      vtnet0
10.1.1.20          link#1             UH          lo0
127.0.0.1          link#2             UH          lo0
----

  This looked a little unusual to me so (there was no link local route for
all the addresses in the VPC), I commented out the rc.conf entry
'google_network_daemon_enable=YES' and setup the vtnet0 interface up
manually with: 'ifconfig_vtnet0="inet 10.1.1.20 netmask 255.255.255.0"'
The resulting routing table:

----
Internet:
Destination        Gateway            Flags     Netif Expire
10.1.1.0/24        link#1             U        vtnet0
10.1.1.20          link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0
----

  This configuration wasn't able to communicate. The latter routing table
looks more usual though, with a 10.1.1.0/24 route to the local link.

  So, it appears to me that VPCs are really configured to be a
point-to-point (star really) network where the Google router interface
(10.1.1.1 in this case) has to handle all forwarding between nodes of a
network.

  I've searched around the web to try and confirm this but there is scant
detail on how exactly forwarding works within a single VPC.

  My VPN project involved using a bastion VPN host that would have
terminated the VPN/SSL tunnel and routed traffic between my home network
and the isolated network behind the bastion.

  Before I make final decisions on configuration, I wanted to know if my
understanding is correct and whether there is any documentation on this
that I've somehow missed.

  FreeBSD is, of course, the host of choice for this operation!

  If anyone does know any details, any info would be greatly appreciated.

Many Thanks,
Rob Ballantyne