Using protocol forwarding

Compute Engine supports protocol forwarding, which lets you create forwarding rule objects that can send packets to a non-NAT’ed target instance. Each target instance contains a single virtual machine (VM) instance that receives and handles traffic from the corresponding forwarding rules.

Protocol forwarding can be used in a number of scenarios, including:

Virtual hosting by IPs

You can set up multiple forwarding rules to point to a single target instance, allowing you to use multiple external IP addresses with one VM instance. You can use this in scenarios where you may want to serve data from just one VM instance, but through different external IP addresses. This is especially useful for setting up SSL virtual hosting.
Virtual private networks (VPN)

Refer to:
- Creating a Classic VPN using dynamic routing
- Creating a Classic VPN using static routing
Private VIPs

You can use protocol forwarding for private regional forwarding rules with VPC subnet ranges. Use this feature to configure private forwarding rules that send TCP or UDP traffic to a target instance in the same region. You can also switch private regional forwarding rules from using target instances to backend services and vice versa.
Load balancing

Refer to Forwarding rule concepts for information about forwarding traffic to a load balancer.

For virtual hosting by IPs, virtual private networks (VPN), and load balancing, Compute Engine supports protocol forwarding for the following protocols:

AH: Specifies the IP Authentication Header protocol.
ESP: Specifies the IP Encapsulating Security Payload protocol.
ICMP: Specifies the Internet Control Message Protocol.
SCTP: Specifies the Stream Control Transmission Protocol.
TCP: Specifies the Transmission Control Protocol.
UDP: Specifies the User Datagram Protocol.

For private VIPs, only TCP and UDP are supported:

TCP: Specifies the Transmission Control Protocol.
UDP: Specifies the User Datagram Protocol.

Protocol forwarding is charged at the same rates as the load balancing service. For more information, read the pricing page.

Before you begin

Make sure that you have the correct permissions to create forwarding rules.
Make sure that an Identity and Access Management (IAM) Condition doesn't grant member conditions that allow members to create internal forwarding rules only. If such a condition exists, forwarding rule creation fails with a permissions error.

For more information, see Access control overview.

Quickstart

This quickstart assumes you are familiar with bash.

To create a forwarding rule that sends traffic to a single instance, you must:

Create a target instance.

Your target instance is a single VM instance, but this instance can exist at the time you create the target instance, or it can be created afterward.
Create a forwarding rule.

Your target instance must exist before you create a forwarding rule. If incoming packets match the IP, protocol, and (if applicable) the port range that is being served by your forwarding rule, the forwarding rule directs that traffic to your target instance.

The rest of this quickstart demonstrates the above steps end-to-end by:

Setting up an Apache server on a VM instance.
Creating a target instance and corresponding forwarding rules.
Send traffic to a single target instance.

At the end of this quickstart, you should know how to set up protocol forwarding from multiple forwarding rules to a single target instance.

Set up a VM instance and install Apache

To create a single VM instance with Apache installed:

Create some startup scripts for your new instance.

Depending on your operating system, your startup script contents might differ.

If you plan to use a Debian image on your instance, run the following command:

me@local:~$ echo "sudo apt update && sudo apt -y install apache2 && mkdir -p /var/www1 &&
mkdir -p /var/www2 && mkdir -p /var/www3 && hostname > /var/www/html/index.html &&
echo w1 > /var/www1/index.html && echo w2 > /var/www2/index.html && echo w3 > /var/www3/index.html" > 

$HOME/pf_startup.sh

If you plan to use a CentOS image on your instance, run the following command:

me@local:~$ echo "sudo yum -y install httpd && sudo service httpd restart && mkdir -p /var/www1 &&
mkdir -p /var/www2 && mkdir /var/www3 && hostname > /var/www/html/index.html &&
echo w1 > /var/www1/index.html && echo w2 > /var/www2/index.html && echo w3 > /var/www3/index.html" > 

$HOME/pf_startup.sh

Create a tag for your future VM, to help apply a firewall to it later:
```
me@local:~$ TAG="www-tag"
```

Create a VM instance to handle traffic for your forwarding rules:

gcloud compute instances create pf-instance \
    --image-project debian-cloud --image-family debian-9 --tags $TAG \
    --metadata-from-file startup-script=$HOME/pf_startup.sh

Create a firewall rule to allow external traffic to this VM instance:

gcloud compute firewall-rules create www-firewall --target-tags $TAG --allow tcp

You have successfully set up a VM instance. Now you can start setting up your protocol forwarding configuration.

Create a target instance and corresponding forwarding rules

Create a target instance.

Target instances contain a single VM instance that receives and handles traffic from a forwarding rule. Target instances do not have a NAT policy, so you can use them to set up your own VPN connections using IPsec protocols directly.

You must create a target instance before you can create a forwarding rule object because forwarding rules must reference an existing target resource. It is not possible to create a forwarding rule that directs traffic to a non-existing target resource. For this example, create a target instance as follows:
```
gcloud compute target-instances create pf-target-instance --instance pf-instance
```
Create forwarding rule objects.

A forwarding rule object directs traffic that matches the IP protocol and port to a specified target instance. For more information, review the forwarding rules documentation.

For this example, the following commands create three forwarding rules, each with an ephemeral IP address that forwards TCP traffic to your target instance. Optionally, if you have some static external IP addresses, you can use them with these forwarding rules by specifying the --address IP-ADDRESS flag.
```
gcloud compute forwarding-rules create pf-rule1 --ip-protocol TCP \
    --ports 80 --target-instance pf-target-instance
```
```
gcloud compute forwarding-rules create pf-rule2 --ip-protocol TCP \
    --ports 80 --target-instance pf-target-instance
```
```
gcloud compute forwarding-rules create pf-rule3 --ip-protocol TCP \
    --ports 80 --target-instance pf-target-instance
```

You can start sending traffic to your target instance.

Send traffic to your instance

Get the external IP addresses of your new forwarding rules.

Run gcloud compute forwarding-rules list to get the external IP addresses of your forwarding rules. For example, the following table lists the ephemeral IP addresses that are allocated for the forwarding rules created earlier.

If you opted to use reserved IP addresses, they are listed here in place of the ephemeral IP addresses.
```
gcloud compute forwarding-rules list
```
```
NAME     REGION      IP_ADDRESS     IP_PROTOCOL TARGET
pf-rule1 us-central1 [ADDRESS_1]    TCP         us-central1-a/targetInstances/pf-target-instance
pf-rule2 us-central1 [ADDRESS_2]    TCP         us-central1-a/targetInstances/pf-target-instance
pf-rule3 us-central1 [ADDRESS_3]    TCP         us-central1-a/targetInstances/pf-target-instance
```
Make a note of the IP addresses for the next step.

Configure the VM instance's Apache virtual hosts to serve different information based on the destination URL.

First, connect to your instance:

gcloud compute ssh pf-instance

Then, edit the /etc/apache2/sites-enabled/000-default.conf file and add the following lines. For VirtualHostcd .. lines, use the IP addresses you saw when you ran the previous step.

<VirtualHost [ADDRESS_1]>
 DocumentRoot /var/www1
 <Directory /var/www1>
  Require all granted
 </Directory>
</VirtualHost>
<VirtualHost [ADDRESS_2]>
 DocumentRoot /var/www2
 <Directory /var/www2>
  Require all granted
 </Directory>
</VirtualHost>
<VirtualHost [ADDRESS_3]>
 DocumentRoot /var/www3
 <Directory /var/www3>
  Require all granted
 </Directory>
</VirtualHost>

Lastly, restart Apache:

user@myinst:~$ sudo /etc/init.d/apache2 restart

Try sending some traffic to your instance.

On your local machine, make a request to the external IP addresses served by the forwarding rules you created.

Use curl to send traffic to the IP addresses. The response returns w1, w2, or w3 depending on the IP address.
```
me@local:~$curl [ADDRESS_1]
w1
```
```
me@local:~$ curl [ADDRESS_2]
w2
```
```
me@local:~$ curl [ADDRESS_3]
w3
```
Your protocol forwarding configuration is ready.

Forwarding rules

Forwarding rules work in conjunction with target pools and target instances to support load balancing and protocol forwarding features. To use load balancing and protocol forwarding, you must create a forwarding rule that directs traffic to specific target pools (for load balancing) or target instances (for protocol forwarding). It is not possible to use these features without a forwarding rule.

Forwarding Rule resources live in the Forwarding Rules collection. Each forwarding rule matches a particular IP address, protocol, and optionally, port range to a single target pool or target instance. When traffic is sent to an external IP address that is served by a forwarding rule, the forwarding rule directs that traffic to the corresponding target pool or target instances. You can create up to 50 forwarding rule objects per project.

Some points to keep in mind when working with forwarding rules:

The name of a forwarding rule must be unique in this project, from 1-63 characters long and match the regular expression: [a-z]([-a-z0-9]*[a-z0-9])? which means the first character must be a lowercase letter, and all following characters must be a dash, lowercase letter, or digit, except the last character, which cannot be a dash.
If you do not specify a forwarding rule protocol, the default TCP is used. Also note that certain protocols can only be used with target pools or target instances:
- If you use ESP, AH, SCTP or ICMP, you must specify a target instance. It is not possible to specify a target pool when using these protocols.
- If you use TCP or UDP, you can specify either a target pool or a target instance.
Port ranges can only be specified for TCP, UDP, and SCTP protocols.

Add a forwarding rule

To add a new forwarding rule, you can use the gcloud compute forwarding-rules create command or create a HTTP POST request to the Forwarding Rules collection. An example of creating a forwarding rule to a target instance using the gcloud command-line tool is:

gcloud compute forwarding-rules create [FORWARDING_RULE] \
    --load-balancing-scheme internal | external \
    --region [REGION] \
    [--target-instance-zone [ZONE]] \
    --ip-protocol TCP --ports 80 \
    --target-instance [TARGET_INSTANCE]

If you omit the --target-instance-zone flag, the gcloud command-line tool prompts you to choose a zone if you did not set the compute/zone property with gcloud config set compute/zone. For more information, see Set default properties.

The gcloud compute forwarding-rules create command supports setting a target instance as a target when the load balancing scheme is internal.

The following example shows creating a forwarding rule to a target pool:

gcloud compute forwarding-rules create [FORWARDING_RULE] \
    --target-pool [TARGET_POOL] \
    [--region [REGION]]

If you omit the --region flag, the gcloud command-line tool prompts you to choose a region if you did not set the compute/region property with gcloud config set compute/region. For more information, see Set default properties.

For a complete description of the flags you can use, see the forwarding rules create command, or type gcloud compute forwarding-rules create --help.

To add a forwarding rule using the API, perform a HTTP POST request to the following URI:

https://compute.googleapis.com/compute/v1/projects/myproject/regions/us-central1/forwardingRules

Your request body should contain the following fields:

{
 "name": "example-forwarding-rule",
 "IPAddress": "`10.1.1.1",
 "IPProtocol": "TCP",
 "portRange": "80",
 "target": "zones/us-central1-f/targetInstances/example-target-instances"
}

For information about listing forwarding rules, getting information about a specific forwarding rule, and deleting forwarding rules, see the gcloud SDK and API reference pages:

Target instances

A Target Instance resource contains one VM instance that handles traffic from one or more forwarding rules and is ideal for forwarding certain types of protocol traffic that should be managed by a single source (e.g., ESP and AH), but you can also use a target instance for TCP and UDP protocols. Target instances do not have a NAT policy applied to them, so they can be used for traffic that requires non-NAT'ed IPsec traffic for virtual private networks (VPN).

Target instances must be in the same region as the forwarding rule. Target instances must also be in the same zone as the VM instance. For example, if your forwarding rule is in us-central1 and the instance you want to use is in us-central1-a, the target instance must be in us-central1-a. If the instance is in us-central1-b, the target instance must also be in us-central1-b.

Add a target instance

To add a new target instance, you can use the gcloud compute target-instances command or create an HTTP POST request to the targetInstances collection. It is not possible to create target instance resources by using the Cloud Console. The following example shows creating a target instance by using the gcloud command-line tool:

gcloud compute target-instances create [TARGET_INSTANCE] --instance INSTANCE

If you omit the --zone flag, the gcloud command-line tool prompts you to choose a zone if you did not set the compute/zone property with gcloud config set compute/zone. For more information, see Set default properties.

For a complete description of the flags you can use, see the create command, or type gcloud compute target-instances create --help.

In the API, make an HTTP POST request to the following URI:

https://compute.googleapis.com/compute/v1/projects/myproject/zones/us-central1-f/targetInstances

With the following request body:

body = {
  "name": "example-target-instance",
  "instance": "zones/us-central1-f/instances/example-instance"
}

For information about listing target instances, getting information about a specific target instance, and deleting target instances, see the gcloud SDK and API reference pages:

Protocol forwarding for private VIPs

Protocol forwarding can be used for private regional forwarding rules. Use this feature to configure private forwarding rules that send TCP or UDP traffic to a target instance in the same region. You can also easily switch private regional forwarding rules from using target instances to backend services and vice versa.

A target instance contains one backend instance that handles traffic from one or more forwarding rules. You can only configure one private forwarding rule to point to each target instance.

Private forwarding rule with a target instance (click to enlarge). — Private forwarding rule with a target instance (click to enlarge)

Use protocol forwarding with private forwarding rules in the following circumstances.

You want to deploy a single backend instance for your service and you want to manage health checks and other aspects yourself.
You want to preserve a forwarding rule private IP address and you want to change the target instance to which the forwarding rule points.
You want to keep your deployment stable by having the ability to move easily to multiple backend instances (backend service) from a single instance (target instance) without changing your private forwarding rule IP address.

Transition between a target instance and a backend service

You can update a private forwarding rule from directing traffic to a target instance with a single VM instance to directing traffic to a backend service or the reverse. When you update a private forwarding rule in this way, the forwarding rule IP address is preserved. Existing connections might be interrupted during this transition because the change enables or disables load balancing for backend service instances.

Transition from a backend service to a target instance (click to enlarge). — Transition from a backend service to a target instance (click to enlarge)

You can transition between a target instance and backend service only with private forwarding rules. You cannot do this with protocol forwarding that uses external forwarding rules.

Updating the target of a private forwarding rule

If your forwarding rule is a private forwarding rule, you can change from pointing to a target instance to pointing to a backend service:

    gcloud compute forwarding-rules set-target my-forwarding-rule
        --backend-service my-backend-service

You can then transition back to a target instance:

    gcloud compute forwarding-rules set-target my-forwarding-rule
        --target-instance my-target-instance

Enforcing protocol forwarding settings across a project, folder, or organization

Use an organization policy to restrict the protocol forwarding types that can be created in your organization. Set the following organization policy constraint:

constraints/compute.restrictProtocolForwardingCreationForTypes

When you set the compute.restrictProtocolForwardingCreationForTypes constraint, you specify the protocol forwarding types that are disallowed. For a list of available types, see Restrict protocol forwarding creation.

Setting up an organization policy

Permissions required for this task

To perform this task, you must have the following permissions:

resourcemanager.organizationAdmin role. For more information see, Access control for organizations using IAM.
To learn more about managing policies at the organization level, see Add an organization policy administrator.

Console

To set a protocol forwarding organization policy from the console, complete the following steps:

In the Google Cloud Console, go to the Organization policies page.
Go to the Organization policies page
In the Filter field, type protocol and select constraints/compute.restrictProtocolForwardingCreationForTypes.
Click Restrict Protocol Forwarding Based on type of IP Address.
Click Edit to edit your existing protocol forwarding policy constraints.
To create a custom policy, select Customize.
After making any changes, click Save to apply the constraint settings.

gcloud

To set a protocol forwarding organization policy, use the gcloud beta resource-manager org-policies enable-enforce command.

Find your organization ID.
```
gcloud organizations list
```

Create the policy file, as shown in the following examples.

List the values to be denied

{
  "constraint": "constraints/compute.restrictProtocolForwardingCreationForTypes",
   "listPolicy": {
     "deniedValues": [
       "INTERNAL",
       "EXTERNAL"
     ]
   }
 }

Allow internal forwarding rules

{
  "constraint": "constraints/compute.restrictProtocolForwardingCreationForTypes",
   "listPolicy": {
     "deniedValues": [
       "EXTERNAL"
     ]
   }
 }

Deny all forwarding rules

{
  "constraint": "constraints/compute.restrictProtocolForwardingCreationForTypes",
  "listPolicy": {
    "allValues": "DENY"
  }
}

Set the constraint in your organization. Replace ORGANIZATION_ID with your organization ID.
```
gcloud beta resource-manager org-policies set-policy POLICY_FILE \
    --organization=ORGANIZATION_ID
```
You can also apply the a protocol forwarding organization policy to a folder or a project with the --folder or the --project flags, and the folder ID and project ID, respectively.

For folders, run the following command:
```
gcloud beta resource-manager org-policies set-policy POLICY_FILE \
    --folder=FOLDER_ID
```
For projects, run the following command:
```
gcloud beta resource-manager org-policies set-policy POLICY_FILE \
    --project=PROJECT_ID
```
Replace the following:
- FOLDER_ID: Your folder ID.
- PROJECT_ID: Your project ID.

After you set the policy, the policy is enforced when adding the respective Google Cloud forwarding rules.

The constraint is not enforced on existing protocol forwarding configurations.

If you attempt to create protocol forwarding of a type that violates the constraint, the attempt fails and an error message is generated. The error message has the following format:

Constraint constraints/compute.restrictProtocolForwardingCreationForTypes
violated for projects/PROJECT_NAME. Forwarding Rule projects/PROJECT_NAME/region/REGION/forwardingRules/FORWARDING_RULE_NAME
of type SCHEME is not allowed.

If you set multiple restrictProtocolForwardingCreationForTypes constraints at different resource levels, and if you set the inheritFromParent field to true, then the constraints are enforced hierarchically.

To learn more about setting organization policies, including descriptions of the available options, see Creating and managing organization policies and Using constraints.

Troubleshooting protocol forwarding for private forwarding rules

Regional restriction

Protocol forwarding for private forwarding rules is a regional product. All clients and target instance VMs must be in the same region.

Error message: "An internal target instance can only be the target of one forwarding rule"

If you see the error message An internal target instance can only be the target of one forwarding rule, you might be trying to configure two forwarding rules pointing to the same target instance. You cannot point multiple forwarding rules to the same target instance.

Restrictions

When you use target instances with private forwarding rules, the only protocols supported are TCP and UDP.

Limits

You can configure 100 private forwarding rules pointing to target instances per network.
You can have 5 ports per private forwarding rule.