Use an HTTP proxy

This document shows you how to route traffic from the GKE on AWS through an HTTP/HTTPS proxy. You specify proxy configuration when you create a cluster.

Overview

GKE on AWS can route outbound internet traffic through a proxy for the following reasons:

  • To register clusters with Google Cloud through Connect
  • To run the Connect Agent
  • To download images from Container Registry

Limitations

  • The httpProxy and httpsProxy fields do not support URLs beginning with https://. You must use http://. Requests to port 443 use HTTPS.
  • You must set values for httpProxy, httpsProxy, and noProxy.
  • You might need to add additional domains, IPs, or CIDRs to the noProxy field. We recommend adding the VPC IP range. As of Google Kubernetes Engine version 1.22, GKE on AWS automatically adds the Pod address CIDR and Service address CIDR.

Prerequisites

This section describes the prerequisites you must apply before using a proxy.

Enable VPC endpoints

Before you configure a proxy, you must create VPC endpoints for your GKE on AWS installation.

VPC endpoints let resources in private subnets access AWS services without public internet access.

The following table lists the AWS services that GKE on AWS requires VPC endpoints for, along with the type of endpoint and the Security Groups that require access to the endpoint.

Service Endpoint type Security groups
Auto Scaling Interface Control plane, node pools
EC2 Interface Control plane, node pools
EFS Interface Control plane
Load Balancing Interface Control plane, node pools
Key Management Service Interface Control plane, node pools
S3 Gateway Control plane, node pools
Secrets Manager Interface Control plane, node pools
Security Token Service (STS) Interface Control plane, node pools

You can create endpoints from the AWS VPC Console. The options you set when creating VPC endpoints depend on your VPC configuration.

Define a security group

GKE on AWS must be able to connect to the proxy server to download software components. Create or locate an AWS security group that allows outbound connections to your proxy server. The security group should allow outbound access from your Control plane, and Node pool security groups to the proxy address and port. Save the ID of this security group— for example, sg-12345678.

Type Protocol From port To port Address
Egress TCP Proxy port Proxy port Proxy security group

Proxy Allowlist

For GKE on AWS to connect to Google Cloud services, the proxy server must allow traffic to the following domains.

.gcr.io
cloudresourcemanager.googleapis.com
container.googleapis.com
gkeconnect.googleapis.com
gkehub.googleapis.com
oauth2.googleapis.com
securetoken.googleapis.com
storage.googleapis.com
sts.googleapis.com
www.googleapis.com
servicecontrol.googleapis.com
logging.googleapis.com
monitoring.googleapis.com
opsconfigmonitoring.googleapis.com
GCP_LOCATION-gkemulticloud.googleapis.com

Replace GCP_LOCATION with the Google Cloud region in which your GKE Enterprise cluster resides. Specify us-west1 or another supported region.

Update AWS IAM roles

For GKE on AWS to read proxy configuration from AWS Secrets Manager, you must add the secretsmanager:GetSecretValue to your cluster's Control plane role and Node pool role.

To add this permission, add it to your control plane and node pool policy. For more information, see Editing IAM policies.

Create a proxy configuration file

The proxy configuration is stored in an AWS Secrets Manager secret as a JSON string. You can pass this configuration to the aws command-line tool as a file. This section describes how to create that file.

The following table describes the contents of this file.

Field Description Examples Required
httpProxy A proxy server URL. The value should include a hostname/IP address and optionally a port, username, and password. "http://user:password@10.184.37.42:80"
"10.184.37.42"
Yes
httpsProxy A proxy URL for encrypted, HTTPS traffic. The httpProxy URL will be used if httpsProxy has an empty value. "http://10.101.16.31:80" Yes
noProxy A comma-separated list of URLs to exclude from proxying. Each value can be an IP address, a CIDR range, a domain name, or the asterix character (*). Domains specified with a leading dot (for example, `.google.com`) indicate that a subdomain is required. A single asterix * ignores all proxy configuration. "1.2.3.4,10.0.0.0/16,example.com,.site.com" Yes
  1. To create the configuration file, create a JSON file that contains values for httpProxy, noProxy, and optional httpsProxy keys.

    {
      "httpProxy": "AUTHENTICATION_URL",
      "httpsProxy": "AUTHENTICATION_URL",
      "noProxy": "NO_PROXY_ADDRESSES"
    }
    

    Replace the following:

    • AUTHENTICATION_URL: Encoded URL containing the proxy username and the password
    • NO_PROXY_ADDRESSES: Comma-separated list of CIDR blocks and URLs— for example 10.0.0.0/16,http://example.com

    Save the file to use in the following section.

  2. Create a secret with this JSON data as a secret in AWS Secrets Manager using the aws command-line tool.

    aws secretsmanager create-secret \
    --name SECRET_NAME \
    --secret-string file://PROXY_CONFIGURATION_FILE
    

    Replace the following:

    • SECRET_NAME: the name of the new secret
    • PROXY_CONFIGURATION_FILE: the path to your proxy configuration file.

    The output includes the secret's Amazon resource name (ARN) and contents. You can now reference this secret when you create a cluster.

Create a cluster that uses a proxy

To configure GKE on AWS to use an HTTP proxy for outbound connectivity, perform the following steps:

Follow the steps in Create a cluster and pass the proxy-secret-arn and proxy-secret-version-id flags.

gcloud container aws clusters create CLUSTER_NAME \
  --proxy-secret-arn=PROXY_SECRET_ARN \
  --proxy-secret-version-id=PROXY_SECRET_VERSION \

Replace the following:

  • CLUSTER_NAME: your cluster's name
  • PROXY_SECRET_ARN: the ARN of the secret that contains proxy settings— for example arn:aws::secretsmanager:us-east-2:111122223333:secret:example/ExampleSecret-jiObOV
  • PROXY_SECRET_VERSION: the secrets's version ID— for example, EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE

Update proxy configuration

You can update the proxy configuration for a cluster control plane or a node pool. To update the proxy configuration ARN, you must first update the control plane or node pool AWS IAM role.

Update AWS IAM roles

Before you change the ARN where the proxy configuration is stored, you need to confirm that your cluster's Control plane role and Node pool role have read access to the secret ARN. If your IAM statement with the secretsmanager:GetSecretValue permission is scoped to specific resource ARNs, add the new secret ARN to that list before updating proxy configuration.

Update cluster proxy configuration

To update your cluster's proxy configuration, use the Google Cloud CLI.

gcloud container aws clusters update  CLUSTER_NAME \
    --location GOOGLE_CLOUD_LOCATION \
    --proxy-secret-arn=PROXY_SECRET_ARN \
    --proxy-secret-version-id=PROXY_SECRET_VERSION

Replace the following:

  • CLUSTER_NAME: your cluster's name
  • GOOGLE_CLOUD_LOCATION: the supported Google Cloud region that manages your cluster
  • PROXY_SECRET_ARN: the ARN of the secret that contains proxy settings
  • PROXY_SECRET_VERSION: the secrets's version ID— for example, EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE

Update node pool proxy configuration

To update your node pool's proxy configuration, use the Google Cloud CLI.

gcloud container aws node-pools update NODE_POOL_NAME
    --cluster CLUSTER_NAME \
    --location GOOGLE_CLOUD_LOCATION \
    --proxy-secret-arn=PROXY_SECRET_ARN \
    --proxy-secret-version-id=PROXY_SECRET_VERSION

Replace the following:

  • NODE_POOL_NAME: your node pool's name
  • CLUSTER_NAME: your cluster's name
  • GOOGLE_CLOUD_LOCATION: the supported Google Cloud region that manages your cluster
  • PROXY_SECRET_ARN: the ARN of the secret that contains proxy settings
  • PROXY_SECRET_VERSION: the secrets's version ID— for example, EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE

Remove proxy configuration

You can remove proxy configuration from the cluster control plane or node pools. These operations are independent. Removing configuration from the control plane doesn't remove it from the cluster's node pools.

Remove control plane proxy configuration

To remove your cluster control plane's proxy configuration, use the Google Cloud CLI.

gcloud container aws clusters update  CLUSTER_NAME \
  --location GOOGLE_CLOUD_LOCATION \
  --clear-proxy-config

Replace the following:

Remove node pool proxy configuration

To remove proxy configuration from a node pool, use the Google Cloud CLI.

gcloud container aws node-pools update NODE_POOL_NAME
  --cluster CLUSTER_NAME \
  --location GOOGLE_CLOUD_LOCATION \
  --clear-proxy-config

Replace the following:

  • NODE_POOL_NAME: your node pool's name
  • CLUSTER_NAME: your cluster's name
  • GOOGLE_CLOUD_LOCATION: the supported Google Cloud region that manages your cluster—for example, us-west1

What's next