Creating a Shared VPC with Deployment Manager (tutorial)

This tutorial guides you through the process of setting up Shared VPC using Deployment Manager, which provides a way to enforce strict network security rules across your organization for access to GCP resources.

Introduction

In large organizations, you may need to put different departments or different applications into different projects to separate budgeting, access control, and so on. With Shared VPC, Organization administrators can give multiple projects permission to use a single, shared VPC network and corresponding networking resources.

With Shared VPC, as an Organization administrator, you can allow the network and security admins of your organization to manage a VPC network of RFC 1918 IP spaces (and related features such as VPNs or firewall rules) that associated projects can use. Administrators in associated projects can create virtual machine (VM) instances in the shared VPC network space. You can apply and enforce consistent policies across an organization.

Because Shared VPC is often used in large organizations, or in organizations with strict security rules, being able to easily reproduce a Shared VPC setup is important. You can use Deployment Manager, an Infrastructure as Code (IaC) tool, to achieve this.

Note: Shared VPC was previously known as Cross-Project Networking (XPN).

About running the tutorial

This tutorial assumes that you are familiar with Organization resources in Google Cloud Platform (GCP) and that you are the administrator of an Organization resource. Understanding Shared VPCs and Deployment Manager will help you follow this tutorial. Throughout this tutorial, the word deployment refers to a deployment from Deployment Manager.

For simplicity, this tutorial uses a single IAM user—your existing identity as an Organization resource administrator. (If you don't already have an Organization resource, you can find instructions in the Before you begin section for creating one.) In a company, three different people are usually involved in setting up the scenario illustrated by this tutorial: an Organization resource administrator, a network administrator (who manages the Shared VPC), and a user of the Shared VPC.

Deployment architecture

The following diagram shows the architecture of this solution:

In this tutorial, you create three of the projects from this diagram:

• The Deployment Manager home project is managed by an Organization resource administrator (in this tutorial we assume that's you) and is used to create the other projects and the Shared VPC setup.
• The host project is managed by network administrators and hosts the Shared VPC. All the networking configuration will be done in this project.
• The service project is managed by users of the Shared VPC. In this project, resources can be created in the Shared VPC from the host project.

Objectives

• Create and configure the Deployment Manager home project.
• Create the host and service projects with Deployment Manager.
• Configure the Shared VPC feature.
• Configure the VPC and some subnetworks in the host project.
• Verify that the VPC can be used in the service project.

Costs

This tutorial uses billable components of GCP, including:

• Compute Engine

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

Before you begin

This tutorial is set in the context of a GCP Organization. If you do not have one, create one. For details, see Creating and Managing Organizations.

Later in this tutorial, you create two GCP projects with a deployment (that is, a Deployment Manager deployment). You create this deployment in a dedicated project with a specific configuration. The following steps guide you through the initial setup of this project and its configuration.

Note: Some of the steps of this tutorial can be done in the Cloud Console, but the most important ones can't be. Therefore, we recommend that you use the gcloud command-line tool throughout this tutorial.

Creating the home project for Deployment Manager

1. Open Cloud Shell.

   Open Cloud Shell

2. If you do not already have one, create a billing account. For details, see Create a Billing Account.

3. Create a new project in your Organization and set a billing account for it. You must choose a unique ID for this project. For instance, you can use a name like [YOUR_NAME]-[DATE]-dm-home.

   Warning: This project is going to create and own the Deployment Manager deployments. Because of the permissions granted in later steps, this project should not be used for any purpose other than creating other projects.

   gcloud

   1. Display the organization list and make a note of the organization ID:

      gcloud organizations list

   2. Set environment variables for values you will use repeatedly in this tutorial. Replace [ORG_ID] with the value you copied in the previous step, and replace [DM_HOME_PROJECT_ID] with the ID of your home project.

      export ORG_ID=[ORG_ID]
      export DM_HOME_PROJECT=[DM_HOME_PROJECT_ID]

   3. Create the new project:

      gcloud projects create $DM_HOME_PROJECT \
          --organization=$ORG_ID

   4. Get a list of billing IDs for the project and make a note of the billing ID:

      gcloud beta billing accounts list

   5. If you have access to several billing accounts, review your internal policies for each of them and choose the appropriate one.

   6. Set an environment variable to the ID of the billing account that you got from the previous step. Replace [BILLING_ACCOUNT_ID] with the billing ID you got from the previous step.

      export BILLING_ACCOUNT_ID=[BILLING_ACCOUNT_ID]

   7. Enable billing for the new project:

      gcloud beta billing projects link $DM_HOME_PROJECT \
          --billing-account=$BILLING_ACCOUNT_ID

   Console

   1. In the Cloud Console, open the Manage resources page.

      [Go to the Manage Resources Page](https://console.cloud.google.com/cloud-resource-manager){: target="console" class="button button-primary" track-type="tutorial" track-name="consoleLink" track-metadata-position="body" }

   2. In the Organization drop-down list at upper left, select your organization.

   3. Click Create Project, and then enter the ID you chose as the project name.

   4. If you have access to several billing accounts, select the one you want to use for this tutorial (review your internal policies for each of them) in the Billing Account section.

   5. In the Organization section, confirm that the organization is selected.

   6. Click Create.

4. Enable the APIs needed for the rest of the tutorial.

   gcloud

   gcloud services enable cloudbilling.googleapis.com \
       deploymentmanager.googleapis.com \
       cloudresourcemanager.googleapis.com \
       iam.googleapis.com \
       servicemanagement.googleapis.com \
       --project $DM_HOME_PROJECT

   Console

   1. Make sure that you have selected your new project in the Cloud Console. If necessary, switch to the new project in the Manage resources page.

      [Go to the Manage Resources Page](https://console.cloud.google.com/cloud-resource-manager){: target="console" class="button button-primary" track-type="tutorial" track-name="consoleLink" track-metadata-position="body" }

   2. Go to the API Library page.

      [Go to the API Library page](https://console.cloud.google.com/apis/library){: target="console" class="button button-primary" track-type="tutorial" track-name="consoleLink" track-metadata-position="body" }

   3. Enable the following APIs:

      • Google Cloud Billing API
      • Google Identity and Access Management (IAM) API
      • Google Service Management API
      • Google Cloud Deployment Manager V2 API
      • Google Cloud Resource Manager API

Configuring the Organization resource policies

Shared VPC is an Organization-level feature. As such, it requires some Organization-level policies to be configured—the service account used by Deployment Manager needs specific roles at the Organization level.

gcloud

1. Set environment variables for the project number and service account name. The following commands read the values from your project and then use the values to set the environment variables.

   export PROJECT_NUMBER=$(gcloud projects describe $DM_HOME_PROJECT \
       --format="value(projectNumber)")
   export DM_SA=${PROJECT_NUMBER}@cloudservices.gserviceaccount.com

2. Add IAM policies:

   gcloud organizations add-iam-policy-binding $ORG_ID \
       --role=roles/resourcemanager.projectCreator \
       --member="serviceAccount:${DM_SA}"
   gcloud organizations add-iam-policy-binding $ORG_ID \
       --role=roles/billing.user \
       --member="serviceAccount:${DM_SA}"
   gcloud organizations add-iam-policy-binding $ORG_ID \
       --role=roles/compute.xpnAdmin \
       --member="serviceAccount:${DM_SA}"

console

1. In your project, go to IAM & admin > Settings.

2. Note the project number.

3. Go to IAM & admin > IAM.

4. Switch to your organization in the top project-selection menu.

5. Click Add at the top of the window.

6. Add the following roles to the Deployment Manager service account. This service account name is [PROJECT_NUMBER]@cloudservices.gserviceaccount.com, where [PROJECT_NUMBER] is the value you noted earlier.

   • Resource Manager > Project Creator
   • Billing > Billing Account User
   • Compute Engine > Compute Shared VPC Admin

Note: In a production environment, you should protect the Shared VPC host project against deletion. In order to do that, run the following command:

gcloud beta resource-manager org-policies enable-enforce 

    --organization $ORG_ID compute.restrictXpnProjectLienRemoval

Creating the projects and Shared VPC configuration

You now are ready to create the Host and Service project with Deployment Manager. If you have been using the Cloud Console, from here on, you must switch to Cloud Shell.

1. If your session has timed out, or if you have closed it, open a new Cloud Shell session.

   OPEN CLOUD SHELL

2. In Cloud Shell, clone the following Git repository:

   git clone https://github.com/GoogleCloudPlatform/deploymentmanager-samples.git

3. Navigate to the project_creation sample in the repository you cloned:

   cd $HOME/deploymentmanager-samples/examples/v2/project_creation

4. Open the config_shared_vpc.yaml file:

   nano config_shared_vpc.yaml

5. Set the values listed in the following table.

   Setting	Description
   HOST_PROJECT	A unique name for your host project. The host project is where the VPC will be created and managed. Note that you must also change this value in the last line of the file, for a total of two replacements. Example: [YOUR_NAME]-[DATE]-host
   SERVICE_PROJECT	A unique name for your service project. The service project is where the VPC from the host project will be used. Example: [YOUR_NAME]-[DATE]-svc
   ORG_ID	Your Organization ID. You should have this value from earlier steps. If not, you can get it by running the following command: gcloud organizations list Use quotation marks around the Organization ID so that it's not considered a numeric value in the YAML file.
   BILLING_ACCOUNT_ID	The ID of your billing account. You should have this value from earlier steps. If not, you can get it by running the following command: gcloud beta billing accounts list
   EMAIL	Your email address. Note that there are a total of five places where you need to set the email address.

   Note: There are two resources of type project.py in the config_shared_vpc.yaml file. The first is the host project for the Shared VPC, and the second is the service project. You need to modify both with your own values.

6. Save the file and exit nano with the following key sequence:

   ^O <Enter> ^X

7. If you used the Cloud Console to create the Deployment Manager home project, set an environment variable to the ID of your home project:

   export DM_HOME_PROJECT=[DM_HOME_PROJECT]

8. Create a preview of the deployment:

   gcloud deployment-manager deployments \
       create shared-vpc-projects \
       --config config_shared_vpc.yaml --preview \
       --project $DM_HOME_PROJECT

   Creating a preview of a Deployment Manager deployment allows you to see what resources will be created, updated, or deleted.

9. Apply the preview of the deployment:

   gcloud deployment-manager deployments \
       update shared-vpc-projects \
       --project $DM_HOME_PROJECT

10. View your deployment in the Cloud Console. If needed, select your Deployment Manager home project in the top project selection menu.

    GO TO DEPLOYMENT MANAGER

After a few minutes, the deployment is complete and your two new projects are created. Although the service project is already linked to the host project, you have not yet created the VPC that is going to be shared.

Troubleshooting

If problems occurred while the preview was being created, you can delete the deployment (no resources have been created yet) and retry the process.

However, if the preview was created but the deployment failed, you might not be able to re-create that deployment. Projects cannot be deleted and re-created immediately; they are marked for deletion for a safety period of 30 days. If you do need to re-create the deployment, change the values of the HOST_PROJECT and SERVICE_PROJECT settings in the config_shared_vpc.yaml file before you try again.

Creating the VPC

You now are going to use another Deployment Manager template to create the VPC in the host project. This is typically an operation that would be done by a network administrator. Because the Shared VPC configuration has already been set up, the VPC is going to be available for use in the service project immediately.

Deploying your networking configuration

1. Set environment variables to the names of the host project and service project names that you set in the config_shared_vpc.yaml file:

   export HOST_PROJECT=[HOST_PROJECT]
   export SERVICE_PROJECT=[SERVICE_PROJECT]

2. Navigate to the network sample in the Git repository:

   cd $HOME/deploymentmanager-samples/community/network/

   In the config.yaml file, you can see one resource of type network.py with three subnetworks with their own CIDRs. You can experiment with changing or duplicating this resource to suit your needs.

3. Create a preview of the deployment. This deployment is created in the host project.

   gcloud deployment-manager deployments create host-network \
       --config config.yaml --preview \
       --project $HOST_PROJECT

   Note: Because it creates the VPC and other networking resources, this deployment is normally managed by a network administrator.

4. Apply the preview of the deployment.

   gcloud deployment-manager deployments update host-network \
       --project $HOST_PROJECT

Verification

You can now verify access to the host project's network from the service project. You can access this network because you have the role roles/compute.networkUser in the host project. For a production deployment, you need to assign this role to every user who is going to use Shared VPC.

1. Create a test instance in the service project using a subnetwork from the host project.

   gcloud

   gcloud compute instances create instance-test \
       --zone us-east1-b \
       --subnet \
       "projects/$HOST_PROJECT/regions/us-east1/subnetworks/host-network-us-east1" \
       --project $SERVICE_PROJECT

   Console

   
   

   1. Select your service project.
      
      [GO TO THE MANAGE RESOURCES PAGE](https://console.cloud.google.com/cloud-resource-manager){: target="console" class="button button-primary" track-type="tutorial" track-name="consoleLink" track-metadata-position="body" }
   2. Go to the VM Instances page.
      
      [GO TO THE VM INSTANCES PAGE](https://console.cloud.google.com/compute/instances){: target="console" class="button button-primary" track-type="tutorial" track-name="consoleLink" track-metadata-position="body" }
   3. Click Create.
   4. In the Name section, type `instance-test`.
   5. Make sure that `us-east1-b` is selected in the Zone section.
   6. Click Management, security, disks, networking, sole tenancy.
   7. Click the Networking tab.
   8. Click the existing network interface.
   9. Choose the Networks shared with me option.
   {: class="screenshot" width="50%"}
   10. Click Done at the bottom of the Network interface section.
   11. Click Create.

   Allow a minute or two for the instance to start up. When the instance is ready, it is listed on the VM Instances page with a green status icon.

2. Verify that your instance is using the Shared VPC.

   gcloud

   gcloud compute instances describe instance-test \
       --zone us-east1-b --project $SERVICE_PROJECT \
       --format="value(networkInterfaces[0].network)"

   The result of this command is the link to the VPC used by the instance you created. You can see from this link that the VPC is from the host project, even if the instance was created in the service project (a result of using the --project flag in the command).

   Console

   
   

   1. Select your instance in the instance list.
   2. Verify that it is using the network from the host project.
      
      {: class="screenshot"}

Cleaning up

After you've finished the current tutorial, you can clean up the resources that you created on Google Cloud so they won't take up quota and you won't be billed for them in the future. The following sections describe how to delete or turn off these resources.

1. In Cloud Shell, delete the test instance:

   gcloud compute instances delete instance-test \
       --project $SERVICE_PROJECT \
       --zone us-east1-b

2. If you set up project removal protection, remove it:

   gcloud beta resource-manager org-policies disable-enforce \
       --organization $ORG_ID compute.restrictXpnProjectLienRemoval

3. Delete the host-network deployment:

   gcloud deployment-manager deployments delete host-network \
       --project $HOST_PROJECT

4. Delete the shared-vpc-projects deployment:

   gcloud deployment-manager deployments \
       delete shared-vpc-projects \
       --project $DM_HOME_PROJECT

5. Delete the home project:

   gcloud projects delete $DM_HOME_PROJECT

What's next

• Learn about the Private Google Access feature of VPC
• Learn to share images across projects to be used with Deployment Manager
• Learn how to design GCP policies
• Try out other Google Cloud features for yourself. Have a look at our tutorials.