Direct VPC egress with a VPC network

Preview — Direct VPC egress

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

You can enable your Cloud Run service or job to send traffic to a VPC network by using Direct VPC egress with no Serverless VPC Access connector required.

Before you begin

• If you don't already have a VPC network in your project, create one.

• If you use Shared VPC, see Connecting to a Shared VPC network.

• Make sure your subnet has at least a few hundred IP addresses available. If the total number of instances across all your Cloud Run services and jobs grows beyond 100, we recommend having at least 4 times (4X) the total number available.

Limitations

• Direct VPC egress supports a maximum of 100 instances (--max-instances=100) for Cloud Run services.

• To ensure proper job executions, only use Direct VPC egress for jobs that don't require more than 8 concurrent instances and be sure to reserve a minimum of 1,024 IP addresses.

The following regions are supported by Direct VPC egress:

• us-central1
• us-east1
• europe-west1
• europe-west3
• asia-northeast1

The following items are not supported by Direct VPC egress:

• Cloud NAT
• Firewall Rules Logging
• VPC Flow Logs
• Packet Mirroring
• Network Intelligence Center
• IPv6 traffic
• Using network tags
in ingress firewall rules applied to the destination resource.
• Using service identity as the source service account in ingress firewall rules applied to the destination resource.
• Firewall rules cannot use Resource Manager tags attached to Cloud Run workloads.
• Cloud Run jobs that run for more than 1 hour might experience connection breaks during maintenance events. The container receives a SIGTSTP signal 10 seconds before the event and a SIGCONT signal after the event. After the container receives the SIGCONT signal, retry the connection.

IP address allocation

To place your Cloud Run service or job on a VPC network, you must specify a network and subnet. Cloud Run allocates IP addresses from your subnet.

IP addresses are ephemeral, so do not create policies based on individual IPs. If you need to create a policy based on IPs, such as in firewall rules, you must use the IP address range of the entire subnet.

To change the network or subnet that your service or job uses, deploy a new service revision or execute a new job task that uses the new network and subnet values.

Scale up

To enable fast scale up in the event of a traffic surge, Cloud Run allocates IP addresses before they are needed.

At any given time, there are likely more allocated IP addresses than the number of instances that currently exist. To ensure that Cloud Run can obtain enough IP addresses, make sure your subnet has at least a few hundred IP addresses available. If the subnet's total number of instances across all Cloud Run services and jobs grows beyond 100, we recommend having at least 4 times (4X) the total number available. If Cloud Run cannot allocate more IP addresses, it cannot start any more service instances or job tasks until more IP addresses become available. If your IP address space is limited, see Supported IP ranges for more options. For IP allocation efficiency and ease of management, place multiple services or jobs on the same subnet.

Scale down

Even after all services or jobs scale down to zero, Cloud Run reserves some IP addresses from the subnet in case services or jobs must quickly scale up again.

To delete the subnet, you must first delete or redeploy your Cloud Run services or jobs to stop using the subnet, and then wait 1-2 hours.

Supported IP ranges

Cloud Run supports the following IPv4 ranges for your subnet:

• RFC 1918 (recommended)
  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.168.0.0/16
• RFC 6598
  • 100.64.0.0/10
• Class E (not recommended with on-premises setups)
  • 240.0.0.0/4

Note: Test non-RFC 1918 setups to identify possible compatibility issues with non-Google Cloud resources or hardware. Learn more about creating subnets with non-RFC 1918 IP ranges.

Set up IAM permissions

Ensure that Cloud Run has access to the VPC network by using one of the following methods:

• Cloud Run Service Agent role: By default, the Cloud Run service agent has the Cloud Run Service Agent role (roles/run.serviceAgent) that contains the necessary permissions.

• Custom permissions: For more granular control, grant the Cloud Run service agent with the following additional permissions on the project:

  • compute.networks.use
  • compute.subnetworks.use on the project or the specific subnet
  • compute.subnetworks.list
  • compute.addresses.get
  • compute.addresses.list
  • compute.addresses.createInternal
  • compute.addresses.deleteInternal

• Compute Network User role: If you don't use the default Cloud Run Service Agent role or the custom permissions, grant the Compute Network User role (roles/compute.networkUser) on the Cloud Run Service Agent service account by running the following command:

  gcloud projects add-iam-policy-binding PROJECT_ID \
  --member "serviceAccount:[e-mail]" \
  --role "roles/compute.networkUser"
  

  Replace the following:

  • PROJECT_ID: the ID of your project.
  • PROJECT_NUMBER: the project number where you deploy your Cloud Run service or job.

Deploy a service

Direct VPC egress allows your Cloud Run service to send traffic to a VPC network without a Serverless VPC Access connector. Network costs scale to zero just like the service itself. You can also use network tags directly on Cloud Run service revisions for more granular network security.

You can configure Direct VPC egress with a service by using the Google Cloud console, Google Cloud CLI, or YAML.

Console

1. Go to Cloud Run

2. Click Create Service if you are configuring a new service you are deploying to. If you are configuring and deploying an existing service, click on the service, then click Edit and deploy new revision.

3. If you are configuring a new service, fill out the initial service settings page as desired, then click Container(s), volumes, networking, security to expand the service configuration page.

4. Click the Networking tab.

5. Click Connect to a VPC for outbound traffic.

6. Click Send traffic directly to a VPC.

7. In the Network field, select the VPC network that you want to send traffic to.

8. In the Subnet field, select the subnet where your service receives IP addresses from.

9. Optional: Enter the names of the network tags that you want to associate with your service or services. Network tags are specified at the revision-level. Each service revision can have different network tags, such as network-tag-2.

10. For Traffic routing, select one of the following:

    • Route only requests to private IPs to the VPC to send only traffic to internal addresses through the VPC network.
    • Route all traffic to the VPC to send all outbound traffic through the VPC network.

11. Click Create or Deploy.

12. To verify that your service is on your VPC network, click on the service, then click the Networking tab. The network and subnet are listed in the VPC card.

    You can now send requests from your Cloud Run service to any resource on the VPC network, as allowed by your firewall rules.

gcloud

To deploy a Cloud Run service without a connector from the Google Cloud CLI:

1. Ensure that the Compute Engine API is enabled for your project:

   gcloud services enable compute.googleapis.com
   

2. Deploy your Cloud Run service with the following command:

   gcloud beta run deploy SERVICE_NAME \
   --image=IMAGE_URL \
   --network=NETWORK \
   --subnet=SUBNET \
   --network-tags=NETWORK_TAG_NAMES \
   --vpc-egress=EGRESS_SETTING \
   --region=REGION
   

   Replace:

   • SERVICE_NAME with the name of your Cloud Run service.
   • IMAGE_URL with a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL has the shape REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG
   • NETWORK with the name of your VPC network.
   • SUBNET with the name of your subnet. The subnet must be /26 or larger. Direct VPC egress supports IPv4 ranges RFC 1918, RFC 6598, and Class E. You can deploy or execute multiple services or jobs on the same subnet.
   • Optional: NETWORK_TAG_NAMES with the comma-separated names of the network tags you want to associate with a service. For services, network tags are specified at the revision-level. Each service revision can have different network tags, such as network-tag-2.
   • EGRESS_SETTING with an egress setting value:
     • all-traffic: Sends all outbound traffic through the VPC network.
     • private-ranges-only: Sends only traffic to internal addresses through the VPC network.
   • REGION with a region for your service.

3. To verify that your service is on your VPC network, run the following command:

   gcloud beta run services describe SERVICE_NAME \
   --region=REGION
   

   Replace:

   • SERVICE_NAME with the name of your service.
   • REGION with the region for your service that you specified in the previous step.

   The output should contain the name of your network, subnet, and egress setting, for example:

   VPC access:
     Network:       default
     Subnet:        subnet
     Egress:        private-ranges-only
   

You can now send requests from your Cloud Run service to any resource on the VPC network, as allowed by your firewall rules.

YAML

You can store your service specification in a YAML file and then deploy it by using the gcloud CLI.

1. Create a new service.yaml file with the following content:

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: SERVICE_NAME
     annotations:
       run.googleapis.com/launch-stage: BETA
     labels:
       cloud.googleapis.com/location: REGION
   spec:
     template:
       metadata:
         annotations:
           run.googleapis.com/network-interfaces: '[{"network":"NETWORK","subnetwork":"SUBNET","tags":"NETWORK_TAG_NAMES"}]'
           run.googleapis.com/vpc-access-egress: EGRESS_SETTING
       spec:
         containers:
         - image: IMAGE

   Replace:

   • SERVICE_NAME with the name of your Cloud Run service. Service names must be 49 characters or less and must be unique per region and project.
   • REGION with the region for your Cloud Run service, which must match the region of your subnet.
   • NETWORK with the name of your VPC network.
   • SUBNET with the name of your subnet.
   • Optional: NETWORK_TAG_NAMES with the names of the network tags you want to associate with a service. For services, network tags are specified at the revision-level. Each service revision can have different network tags, such as network-tag-2.
   • EGRESS_SETTING with an egress setting value:
     • all-traffic: Sends all outbound traffic through the VPC network.
     • private-ranges-only: Sends only traffic to internal addresses through the VPC network.
   • IMAGE with the URL of your service container image.

   You can also specify more configuration, such as environment variables or memory limits.

2. Deploy the new service by running the following command:

   gcloud run services replace service.yaml

3. Optionally, make your service public if you want to allow unauthenticated access to the service.

Caution: If you are creating a service revision, the following instructions replace your existing service configuration with the one specified in the YAML file. If you use YAML to make revision changes, avoid also using the Google Cloud console or gcloud CLI to make configuration changes because those can be overwritten when you use YAML.

If you need to download or view the configuration of an existing service, use the following command to save results to a YAML file:

gcloud run services describe SERVICE --format export > service.yaml

From a service configuration YAML file, modify any spec.template child attributes as desired to update revision settings, and then deploy the new revision:

gcloud run services replace service.yaml

Note: Administrators can restrict the egress settings that developers can select by setting the run.allowedVPCEgress organization policy.

Create a job

Direct VPC egress allows your Cloud Run job to send traffic to a VPC network without a Serverless VPC Access connector.

You can configure Direct VPC egress with a job by using the Google Cloud console, Google Cloud CLI, or YAML.

Console

1. Go to Cloud Run

2. If you are configuring a new job, click the Jobs tab and fill out the initial job settings page as desired. If you are configuring an existing job, click the job, then click Edit.

3. Click Container, Variables & Secrets, Connections, Security to expand the job properties page.

4. Click the Connections tab.

5. Click Connect to a VPC for outbound traffic.

6. Click Send traffic directly to a VPC.

7. In the Network field, select the VPC network that you want to send traffic to.

8. In the Subnet field, select the subnet where your job receives IP addresses from.

9. For Traffic routing, select one of the following:

   • Route only requests to private IPs to the VPC to send only traffic to internal addresses through the VPC network.
   • Route all traffic to the VPC to send all outbound traffic through the VPC network.

10. Optional: Enter the names of the network tags that you want to associate with your service or services. Network tags are specified at the revision-level. Each service revision can have different network tags, such as network-tag-2.

11. Optional: Enter the names of the network tags that you want to associate with your job or jobs. For jobs, network tags are specified at the execution level. Each job execution can have different network tags, such as network-tag-2.

12. Click Create or Update.

13. To verify that your job is on your VPC network, click on the job, then click the Configuration tab. The network and subnet are listed in the VPC card.

    You can now execute your Cloud Run job and send requests from the job to any resource on the VPC network, as allowed by your firewall rules.

gcloud

To create a Cloud Run job without a connector from the Google Cloud CLI:

1. Ensure that the Compute Engine API is enabled for your project:

   gcloud services enable compute.googleapis.com
   

2. Create a Cloud Run job with the following command:

   gcloud run jobs create JOB_NAME \
   --image=IMAGE_URL \
   --network=NETWORK \
   --subnet=SUBNET \
   --network-tags=NETWORK_TAG_NAMES \
   --vpc-egress=EGRESS_SETTING \
   --region=REGION
   

   Replace:

   • JOB_NAME with the name of your Cloud Run job.
   • IMAGE_URL with a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/job:latest
   • NETWORK with the name of your VPC network.
   • SUBNET with the name of your subnet. The subnet must be /26 or larger. Direct VPC egress supports IPv4 ranges RFC 1918, RFC 6598, and Class E. You can deploy or execute multiple services or jobs on the same subnet.
   • Optional: NETWORK_TAG_NAMES with the names of the network tags you want to associate with a job. For jobs, network tags are specified at the execution-level. Each job execution can have different network tags, such as network-tag-2.
   • EGRESS_SETTING with an egress setting value:
     • all-traffic: Sends all outbound traffic through the VPC network.
     • private-ranges-only: Sends only traffic to internal addresses through the VPC network.
   • REGION with a region for your job.

3. To verify that the job is on your VPC network, run the following command:

   gcloud run jobs describe JOB_NAME \
     --region=REGION
     

   Replace:

   • JOB_NAME with the name of your job.
   • REGION with the region for your job that you specified in the previous step.

   The output should contain the name of your network and subnet, for example:

   VPC network:
     Network:       default
     Subnet:        default
   

You can now execute your Cloud Run job and send requests from the job to any resource on the VPC network, as allowed by your firewall rules.

YAML

You can store your job specification in a YAML file and then deploy it by using the gcloud CLI.

1. Create a new job.yaml file with the following content:

   apiVersion: run.googleapis.com/v1
   kind: Job
   metadata:
     name: JOB_NAME
     annotations:
       run.googleapis.com/launch-stage: BETA
     labels:
       cloud.googleapis.com/location: REGION
   spec:
     template:
       metadata:
         annotations:
           run.googleapis.com/network-interfaces: '[{"network":"NETWORK","subnetwork":"SUBNET","tags":"NETWORK_TAG_NAMES"}]'
           run.googleapis.com/vpc-access-egress: EGRESS_SETTING
       spec:
         containers:
         - image: IMAGE

   Replace:

   • JOB_NAME with the name of your Cloud Run job. Job names must be 49 characters or less and must be unique per region and project.
   • REGION with the region for your Cloud Run job, which must match the region of your subnet.
   • NETWORK with the name of your VPC network.
   • SUBNET with the name of your subnet.
   • Optional: NETWORK_TAG_NAMES with the names of the network tags you want to associate with a job. For jobs, network tags are specified at the execution-level. Each job execution can have different network tags, such as network-tag-2.
   • EGRESS_SETTING with an egress setting value:
     • all-traffic: Sends all outbound traffic through the VPC network.
     • private-ranges-only: Sends only traffic to internal addresses through the VPC network.
   • IMAGE with the URL of your job container image.

2. Deploy the new job by running the following command:

   gcloud run jobs replace job.yaml

Caution: The following instructions replace your existing job configuration with the one specified in the YAML file. If you use YAML to make changes, avoid also using the Google Cloud console or gcloud CLI to make configuration changes because those can be overwritten when you use YAML.

If you need to download or view the configuration of an existing job, use the following command to save results to a YAML file:

gcloud run jobs describe JOB --format export > job.yaml

From a job configuration YAML file, modify any spec.template child attributes as desired to update revision settings, and then redeploy:

gcloud run jobs replace job.yaml

Note: Administrators can restrict the egress settings that developers can select by setting the run.allowedVPCEgress organization policy.

Restrict access with firewall rules

Restrict access to resources in a VPC network by using VPC firewall rules. Add these restrictions by using one of the following strategies:

• Create an ingress firewall rule that refers to your service or job by using the subnet's IP range.

• Create an egress firewall rule that refers to your service or job.

  In the egress firewall rule, refer to your service or job by using the linked service account service identity, the subnet's IP range, or the associated network tags.

Network tags for egress

Add an additional layer of network security by using network tags in egress firewall rules.

Console

To associate network tags with a service or job:

1. In the Google Cloud console, go to the Cloud Run page.

   Go to Cloud Run

2. Click the service or job you want to associate network tags with, then click Edit and deploy new revision for services or Edit for jobs.

3. Click the Networking tab for services, or the Connections tab for jobs.

4. Ensure that you have selected Connect to a VPC for outbound traffic and Send traffic directly to a VPC.

5. In the Network tags field, enter the names of the network tags that you want to associate with your service or job.

6. Click Deploy or Update.

For services, every service revision can have a different set of network tags because network tags are specified at the revision level. For jobs, a job execution has the same network tags that the job had when the execution of the job was created.

gcloud

To associate network tags with a service or job, use the gcloud beta run deploy command:

gcloud beta run deploy SERVICE_JOB_NAME \
    --image=IMAGE_URL \
    --network=NETWORK \
    --subnet=SUBNET \
    --network-tags=NETWORK_TAG_NAMES \
    --region=REGION

Replace the following:

• SERVICE_JOB_NAME: the name of your service or job.
• IMAGE_URL: the image URL of the service or job.
• NETWORK: the name of your VPC network.
• SUBNET: the name of your subnet.
• NETWORK_TAG_NAMES: the name of your network tag or a comma-separated list of network tags.
• REGION: the name of your region.

For services, every service revision can have a different set of network tags because network tags are specified at the revision level. For jobs, a job execution has the same network tags that the job had when the execution of the job was created.

Disconnect a service

Console

• To remove your service from the VPC network:

  1. Go to Cloud Run

  2. Click the service you want to remove, then click Edit and deploy new revision.

  3. Click the Networking tab.

  4. Clear Connect to a VPC for outbound traffic.

  5. Click Deploy.

  6. To verify that your service is no longer on your VPC network, click the Networking tab. The network and subnet are no longer listed in the VPC card.

• To remove only the network tags while keeping the service connected to the VPC network:

  1. Click the service that contains the network tags you want to remove, then click Edit and deploy new revision.

  2. Click the Networking tab.

  3. Clear the names of the network tags that you no longer want to associate with your service.

  4. Click Deploy.

gcloud

• To remove your service from the VPC network, run the following command:

  gcloud beta run services update SERVICE_NAME --region=REGION \
  --clear-network
  

• To remove only the network tags while keeping the service connected to the VPC network, run the following command:

  gcloud beta run services update SERVICE_NAME --region=REGION \
  --clear-network-tags
  

  Replace the following:

  • SERVICE_NAME: the name of your Cloud Run service.
  • REGION: the region for your Cloud Run service.

YAML

• To remove your service from the VPC network:

  1. Remove the following content from your service.yaml file:

     run.googleapis.com/network-interfaces: '[{"network":"NETWORK","subnetwork":"SUBNET","tags":"NETWORK_TAG_NAMES"}]'

     Where

     • NETWORK: the name of your VPC network.
     • SUBNET: the name of your subnet.
     • Optional: NETWORK_TAG_NAMES: the names of the network tags if you had associated them with a service.

  2. Deploy the service revision by running the following command:

     gcloud run services replace service.yaml

• To remove only the network tags while keeping the service connected to the VPC network:

  1. Remove the tags variable from the content in your service.yaml file, leaving the network and subnetwork variables in place, as shown in the following example:

     run.googleapis.com/network-interfaces: '[{"network":"NETWORK","subnetwork":"SUBNET"}]'

     Where

     • NETWORK: the name of your VPC network.
     • SUBNET: the name of your subnet.

  2. Deploy the service revision by running the following command:

     gcloud run services replace service.yaml

Disconnect a job

Console

• To remove your job from the VPC network:

  1. Go to Cloud Run

  2. Click the job you want to remove, then click Edit and deploy new revision.

  3. Click the Connections tab.

  4. Clear Connect to a VPC for outbound traffic.

  5. Click Update.

  6. To verify that your job is no longer on your VPC network, click the Configuration tab. The network and subnet are no longer listed in the VPC card.

• To remove only the network tags while keeping the job connected to the VPC network:

  1. Click the job that contains the network tags you want to remove, then click Edit and deploy new revision.

  2. Click the Connections tab.

  3. Clear the names of the network tags that you no longer want to associate with your job.

  4. Click Update.

gcloud

• To remove your job from the VPC network, run the following command:

  gcloud run jobs update JOB_NAME --region=REGION \
    --clear-network
    

• To remove only the network tags while keeping the job connected to the VPC network, run the following command:

  gcloud run jobs update JOB_NAME --region=REGION \
    --clear-network-tags
    

  Replace the following:

  • JOB_NAME: the name of your Cloud Run job.
  • REGION: the region for your Cloud Run job.

YAML

• To remove your job from the VPC network:

  1. Remove the following content from your job.yaml file:

     run.googleapis.com/network-interfaces: '[{"network":"NETWORK","subnetwork":"SUBNET","tags":"NETWORK_TAG_NAMES"}]'

     Replace the following:

     • NETWORK: the name of your VPC network.
     • SUBNET: the name of your subnet.
     • Optional: NETWORK_TAG_NAMES with the names of the network tags if you had associated them with a job.

  2. Redeploy the job by running the following command:

     gcloud run jobs replace job.yaml

• To remove only the network tags while keeping the job connected to the VPC network:

  1. Remove the tags variable from the content in your job.yaml file, leaving the network and subnetwork variables in place, as shown in the following example:

     run.googleapis.com/network-interfaces: '[{"network":"NETWORK","subnetwork":"SUBNET"}]'

     Replace the following:

     • NETWORK: the name of your VPC network.
     • SUBNET: the name of your subnet.

  2. Redeploy the job by running the following command:

     gcloud run jobs replace job.yaml

Troubleshooting

Cannot delete subnet

To delete a subnet, you must first delete all resources that use it. If Cloud Run is using a subnet, you must disconnect Cloud Run from the VPC network or move it to a different subnet before deleting the subnet.

Note: After you delete or move your Cloud Run resources, wait 1-2 hours for Cloud Run to release the IPs before you delete the subnet.

Direct VPC subnet runs out of IP addresses

If the subnet of the VPC network runs out of IP addresses, it is logged by Cloud Logging. When this occurs, Cloud Run cannot start any more service instances or job tasks until more IP addresses become available.

View allocated IP addresses

To see which IP addresses Cloud Run has allocated, go to the IP addresses page in the Google Cloud console or run the following command from the Google Cloud CLI:

gcloud compute addresses list