Using a proxy with GKE on AWS

This topic shows you how to route traffic from the GKE on AWS management service and Connect through an HTTP/HTTPS proxy.

GKE on AWS requires internet access for the following reasons:

  • The management service registers user clusters with Connect.
  • User clusters run the Connect Agent.
  • All management service and user cluster nodes download images from Container Registry.

You can route this traffic through an HTTP or HTTPS proxy. As of GKE on AWS 1.8, you can configure individual proxy settings for your management service and each of your clusters. For more information on how to change proxy settings, see Changing cluster proxy settings.

You declare proxy settings in a JSON file that is referenced in anthos-gke.yaml.

Prerequisites

Before you configure GKE on AWS to use a proxy, you must enable AWS VPC endpoints, define a security group for the proxy, and allow outbound connections to certain Google Cloud services.

To use an HTTP proxy, your user clusters must be running Kubernetes 1.17 or higher.

Enable VPC endpoints

Before you configure a proxy, you must create VPC endpoints for your GKE on AWS installation. VPC endpoints let resources in private subnets access AWS services without public internet access.

The following table lists the AWS services that GKE on AWS requires VPC endpoints for, along with the type of endpoint and the Security Groups for each GKE on AWS component that requires access to the endpoint.

Service Endpoint type Security groups
Auto Scaling Interface Management, control plane, node pools
EC2 Interface Management, control plane, node pools
Load Balancing Interface Management, control plane, node pools
Key Management Service Interface Management, control plane, node pools
S3 Gateway Management, control plane, node pools
Security Token Service (STS) Interface Management, control plane, node pools

You can create endpoints from the AWS VPC Console. The options you set when creating VPC endpoints depend on your VPC configuration.

Define a security group

Your GKE on AWS components must be able to connect to the proxy server. Create or locate an AWS security group that allows outbound connections to your proxy server. The security group should allow outbound access from your Management, Control plane, and Node pool security groups to the proxy address and port. Save the ID of this security group (for example, sg-12345678).

Type Protocol From port To port Address
Egress TCP Proxy port Proxy port Proxy security group

Allow outbound connections to Google Cloud services

For GKE on AWS to connect to Google Cloud services, the proxy server must allow traffic to the following domains:

  • gkeconnect.googleapis.com
  • gkehub.googleapis.com
  • oauth2.googleapis.com
  • storage.googleapis.com
  • www.googleapis.com
  • gcr.io
  • k8s.gcr.io
  • EC2-REGION.ec2.archive.ubuntu.com

Replace EC2-REGION with the AWS EC2 region where your GKE on AWS installation runs. For example, us-west-1.ec2.archive.ubuntu.com/.

If you are using Cloud Service Mesh with Prometheus and Kiali, allow outbound access from the following domains:

  • docker.io
  • quay.io

Creating the proxy configuration JSON file

The proxy configuration JSON file contains one object that contains three key/value pairs, with key names httpProxy, httpsProxy, and noProxy. Usernames and passwords are envelope-encrypted with your AWS KMS key before applying them to the server.

{
  "httpProxy": "HTTP_PROXY",
  "httpsProxy": "HTTPS_PROXY",
  "noProxy": "NO_PROXY"
}

Replace the following:

  • HTTP_PROXY with the proxy server address to route HTTP requests through.
  • HTTPS_PROXY with the proxy server to route HTTPS requests through.
  • NO_PROXY with an optional list of IPs, CIDR ranges, and domains where proxy is not used.

See Configuration reference for more information. For example, the following configuration uses a proxy at 10.0.0.254 and excludes a number of sites.

{
  "httpProxy": "http://user:password@10.0.0.254:80",
  "httpsProxy": "http://user:password@10.0.0.254:443",
  "noProxy": "localhost,127.0.0.1,169.254.169.254,10.0.0.0/16,example.com"
}

Adding the proxy to anthos-gke.yaml

Configure your AWSManagementService by performing the following steps:

  1. Change to the directory with your GKE on AWS configuration. You created this directory when Installing the management service.

    cd anthos-aws

  2. Open anthos-gke.yaml in a text editor. Add a value for spec.proxy as in the following example.

    apiVersion: multicloud.cluster.gke.io/v1
    kind: AWSManagementService
    metadata:
      name: management
    spec:
      version: aws-1.14.1-gke.0
      proxy: PROXY_JSON_FILE
      securityGroupIDs:
      - SECURITY_GROUP_ID
      ...
    

    Replace the following:

    • PROXY_JSON_FILE with the relative path of your proxy's JSON configuration.
    • SECURITY_GROUP_ID with the ID of a security group with access to the proxy server.

    Save the file.

  3. Run anthos-gke aws management init to generate an anthos-gke.status.yaml file with additional configuration. The init command also validates the AWSManagementService object in your anthos-gke.yaml file.

    anthos-gke aws management init
    
  4. Run anthos-gke aws management apply to update your management service on AWS.

    anthos-gke aws management apply
    
  5. Next, Create a user cluster. Include the proxy's security group in the AWSCluster and AWSNodePool's spec.controlPlane.securityGroupIDs and spec.securityGroupIDs, respectively. The cluster then has access to the proxy.

Configuration reference

Field Description Example Required
httpProxy A proxy server URL. The value should include a hostname/IP address and optionally a port, username, and password. "http://user:password@10.184.37.42:80" Yes
httpsProxy A proxy URL for encrypted, HTTPS traffic. The httpProxy URL will be used if httpsProxy isn't given. "http://user:password@10.101.16.31:80" No
noProxy A comma-separated list of URLs to exclude from proxying. Each value may be an IP address, a CIDR range, a domain name, or the asterix character (*). Domains specified with a leading dot (e.g. ".google.com") indicate that a subdomain is required. A single asterix * indicates that no proxying should be done. "1.2.3.4,10.0.0.0/16,example.com,.site.com" No

What's next

For more information about completing a GKE on AWS installation, see the following:

For information about changing your cluster's proxy settings, see: