This document shows you how to route traffic from the GKE on AWS through an HTTP/HTTPS proxy. You specify proxy configuration when you create a cluster.
Overview
GKE on AWS can route outbound internet traffic through a proxy for the following reasons:
- To register clusters with Google Cloud through Connect
- To run the Connect Agent
- To download images from Container Registry
Limitations
- The
httpProxy
andhttpsProxy
fields do not support URLs beginning withhttps://
. You must usehttp://
. Requests to port 443 use HTTPS. - You must set values for
httpProxy
,httpsProxy
, andnoProxy
. - You might need to add additional domains, IPs, or CIDRs to the
noProxy
field. We recommend adding the VPC IP range. As of Google Kubernetes Engine version 1.22, GKE on AWS automatically adds the Pod address CIDR and Service address CIDR.
Prerequisites
This section describes the prerequisites you must apply before using a proxy.
Enable VPC endpoints
Before you configure a proxy, you must create VPC endpoints for your GKE on AWS installation.
VPC endpoints let resources in private subnets access AWS services without public internet access.
The following table lists the AWS services that GKE on AWS requires VPC endpoints for, along with the type of endpoint and the Security Groups that require access to the endpoint.
Service | Endpoint type | Security groups |
---|---|---|
Auto Scaling | Interface | Control plane, node pools |
EC2 | Interface | Control plane, node pools |
EFS | Interface | Control plane |
Load Balancing | Interface | Control plane, node pools |
Key Management Service | Interface | Control plane, node pools |
S3 | Gateway | Control plane, node pools |
Secrets Manager | Interface | Control plane, node pools |
Security Token Service (STS) | Interface | Control plane, node pools |
You can create endpoints from the AWS VPC Console. The options you set when creating VPC endpoints depend on your VPC configuration.
Define a security group
GKE on AWS must be able to connect to the proxy
server to download software components. Create or locate an AWS
security group
that allows outbound connections to your proxy server. The security group should
allow outbound access from your Control plane, and Node pool security
groups to the proxy address and port. Save the ID of this security group—
for example, sg-12345678
.
Type | Protocol | From port | To port | Address |
---|---|---|---|---|
Egress | TCP | Proxy port | Proxy port | Proxy security group |
Proxy Allowlist
For GKE on AWS to connect to Google Cloud services, the proxy server must allow traffic to the following domains.
.gcr.io
cloudresourcemanager.googleapis.com
container.googleapis.com
gkeconnect.googleapis.com
gkehub.googleapis.com
oauth2.googleapis.com
securetoken.googleapis.com
storage.googleapis.com
sts.googleapis.com
www.googleapis.com
servicecontrol.googleapis.com
logging.googleapis.com
monitoring.googleapis.com
opsconfigmonitoring.googleapis.com
GCP_LOCATION-gkemulticloud.googleapis.com
Replace GCP_LOCATION
with the Google Cloud region in
which your GKE Enterprise cluster resides. Specify us-west1
or another
supported region.
Update AWS IAM roles
For GKE on AWS to read proxy configuration from AWS Secrets Manager,
you must add the secretsmanager:GetSecretValue
to your cluster's
Control plane role
and Node pool role.
To add this permission, add it to your control plane and node pool policy. For more information, see Editing IAM policies.
Create a proxy configuration file
The proxy configuration is stored in an AWS Secrets Manager secret as a JSON string.
You can pass this configuration to the aws
command-line tool as a
file. This section describes how to create that file.
The following table describes the contents of this file.
Field | Description | Examples | Required |
---|---|---|---|
httpProxy |
A proxy server URL. The value should include a hostname/IP address and optionally a port, username, and password. | "http://user:password@10.184.37.42:80" "10.184.37.42" |
Yes |
httpsProxy |
A proxy URL for encrypted, HTTPS traffic. The httpProxy URL will be used if httpsProxy has an empty value. | "http://10.101.16.31:80"
|
Yes |
noProxy |
A comma-separated list of URLs to exclude from proxying. Each value can be an IP address, a CIDR range, a domain name, or the asterix character (*). Domains specified with a leading dot (for example, `.google.com`) indicate that a subdomain is required. A single asterix * ignores all proxy configuration. | "1.2.3.4,10.0.0.0/16,example.com,.site.com" |
Yes |
To create the configuration file, create a JSON file that contains values for
httpProxy
,noProxy
, and optionalhttpsProxy
keys.{ "httpProxy": "AUTHENTICATION_URL", "httpsProxy": "AUTHENTICATION_URL", "noProxy": "NO_PROXY_ADDRESSES" }
Replace the following:
AUTHENTICATION_URL
: Encoded URL containing the proxy username and the passwordNO_PROXY_ADDRESSES
: Comma-separated list of CIDR blocks and URLs— for example10.0.0.0/16,http://example.com
Save the file to use in the following section.
Create a secret with this JSON data as a secret in AWS Secrets Manager using the
aws
command-line tool.aws secretsmanager create-secret \ --name SECRET_NAME \ --secret-string file://PROXY_CONFIGURATION_FILE
Replace the following:
SECRET_NAME
: the name of the new secretPROXY_CONFIGURATION_FILE
: the path to your proxy configuration file.
The output includes the secret's Amazon resource name (ARN) and contents. You can now reference this secret when you create a cluster.
Create a cluster that uses a proxy
To configure GKE on AWS to use an HTTP proxy for outbound connectivity, perform the following steps:
Follow the steps in
Create a cluster
and pass the proxy-secret-arn
and proxy-secret-version-id
flags.
gcloud container aws clusters create CLUSTER_NAME \
--proxy-secret-arn=PROXY_SECRET_ARN \
--proxy-secret-version-id=PROXY_SECRET_VERSION \
Replace the following:
CLUSTER_NAME
: your cluster's namePROXY_SECRET_ARN
: the ARN of the secret that contains proxy settings— for examplearn:aws::secretsmanager:us-east-2:111122223333:secret:example/ExampleSecret-jiObOV
PROXY_SECRET_VERSION
: the secrets's version ID— for example,EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE
Update proxy configuration
You can update the proxy configuration for a cluster control plane or a node pool. To update the proxy configuration ARN, you must first update the control plane or node pool AWS IAM role.
Update AWS IAM roles
Before you change the ARN where the proxy configuration is stored, you
need to confirm that your cluster's
Control plane role
and Node pool role
have read access to the secret ARN. If your IAM statement with the
secretsmanager:GetSecretValue
permission is scoped to specific resource ARNs,
add the new secret ARN to that list before updating proxy configuration.
Update cluster proxy configuration
To update your cluster's proxy configuration, use the Google Cloud CLI.
gcloud container aws clusters update CLUSTER_NAME \
--location GOOGLE_CLOUD_LOCATION \
--proxy-secret-arn=PROXY_SECRET_ARN \
--proxy-secret-version-id=PROXY_SECRET_VERSION
Replace the following:
CLUSTER_NAME
: your cluster's nameGOOGLE_CLOUD_LOCATION
: the supported Google Cloud region that manages your clusterPROXY_SECRET_ARN
: the ARN of the secret that contains proxy settingsPROXY_SECRET_VERSION
: the secrets's version ID— for example,EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE
Update node pool proxy configuration
To update your node pool's proxy configuration, use the Google Cloud CLI.
gcloud container aws node-pools update NODE_POOL_NAME
--cluster CLUSTER_NAME \
--location GOOGLE_CLOUD_LOCATION \
--proxy-secret-arn=PROXY_SECRET_ARN \
--proxy-secret-version-id=PROXY_SECRET_VERSION
Replace the following:
NODE_POOL_NAME
: your node pool's nameCLUSTER_NAME
: your cluster's nameGOOGLE_CLOUD_LOCATION
: the supported Google Cloud region that manages your clusterPROXY_SECRET_ARN
: the ARN of the secret that contains proxy settingsPROXY_SECRET_VERSION
: the secrets's version ID— for example,EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE
Remove proxy configuration
You can remove proxy configuration from the cluster control plane or node pools. These operations are independent. Removing configuration from the control plane doesn't remove it from the cluster's node pools.
Remove control plane proxy configuration
To remove your cluster control plane's proxy configuration, use the Google Cloud CLI.
gcloud container aws clusters update CLUSTER_NAME \
--location GOOGLE_CLOUD_LOCATION \
--clear-proxy-config
Replace the following:
CLUSTER_NAME
: your cluster's nameGOOGLE_CLOUD_LOCATION
: the supported Google Cloud region that manages your cluster—for example,us-west1
Remove node pool proxy configuration
To remove proxy configuration from a node pool, use the Google Cloud CLI.
gcloud container aws node-pools update NODE_POOL_NAME
--cluster CLUSTER_NAME \
--location GOOGLE_CLOUD_LOCATION \
--clear-proxy-config
Replace the following:
NODE_POOL_NAME
: your node pool's nameCLUSTER_NAME
: your cluster's nameGOOGLE_CLOUD_LOCATION
: the supported Google Cloud region that manages your cluster—for example,us-west1
What's next
- Read additional information on how to Create a cluster.