Create an internal load balancer across VPC networks


This page explains how to create an internal passthrough Network Load Balancer on Google Kubernetes Engine (GKE) across VPC networks.

Before reading this page, ensure that you're familiar with the following concepts:

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Create an internal load balancer with Private Service Connect

As a service producer, you can use service attachments to make your services available to service consumers in other VPC networks using Private Service Connect. You can create, manage, and delete service attachments using a ServiceAttachment custom resource.

Requirements and limitations

  • Limitations for Private Service Connect apply.
  • You can create a service attachment in GKE versions 1.21.4-gke.300 and later.
  • You cannot use the same subnet in multiple service attachment configurations.
  • You must create a GKE service that uses an internal passthrough Network Load Balancer.
  • You cannot specify a subnet in a different project (Shared VPC) for GKE versions earlier than 1.22.4-gke.100. For Shared VPC, ensure all requirements for Shared VPC are met.

Create a ServiceAttachment

  1. Create a subnet.

    You must create a new subnet for each ServiceAttachment.

    gcloud beta compute networks subnets create SUBNET_NAME \
        --project PROJECT_ID \
        --network NETWORK_NAME \
        --region REGION \
        --range SUBNET_RANGE \
        --purpose PRIVATE_SERVICE_CONNECT
    

    Replace the following:

    • SUBNET_NAME: the name of the new subnet. In GKE versions 1.22.4-gke.100 and later, you can specify a subnet in a different project by using the fully qualified resource URL for this field. You can get the fully qualified resource URL using the command gcloud compute networks subnets describe.
    • PROJECT_ID: the ID of your Google Cloud project.
    • NETWORK_NAME: the name of the VPC network for the subnet.
    • REGION: the region for the new subnet. You must use the same region as the service that you create.
    • SUBNET_RANGE: the IP address range to use for the subnet.
  2. Deploy a workload.

    The following manifest describes a Deployment that runs a sample web application container image. Save the manifest as my-deployment.yaml:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: psc-ilb
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: psc-ilb
      template:
        metadata:
          labels:
            app: psc-ilb
        spec:
          containers:
          - name: whereami
            image: us-docker.pkg.dev/google-samples/containers/gke/whereami:v1.2.19
            ports:
              - name: http
                containerPort: 8080
            readinessProbe:
              httpGet:
                path: /healthz
                port: 8080
                scheme: HTTP
              initialDelaySeconds: 5
              timeoutSeconds: 1
    
  3. Apply the manifest to your cluster:

    kubectl apply -f my-deployment.yaml
    
  4. Create a service. The following manifest describes a service that creates an internal passthrough Network Load Balancer on TCP port 8080. Save the manifest as my-service.yaml:

     apiVersion: v1
     kind: Service
     metadata:
       name: SERVICE_NAME
       annotations:
         networking.gke.io/load-balancer-type: "Internal"
     spec:
       type: LoadBalancer
       selector:
         app: psc-ilb
       ports:
       - port: 80
         targetPort: 8080
         protocol: TCP
    

    Replace the following:

    • SERVICE_NAME: the name of the new service.
  5. Apply the manifest to your cluster:

    kubectl apply -f my-service.yaml
    
  6. Create ServiceAttachment.

    The following manifest describes a ServiceAttachment that exposes the service that you created to service consumers. Save the manifest as my-psc.yaml:

    apiVersion: networking.gke.io/v1
    kind: ServiceAttachment
    metadata:
     name: SERVICE_ATTACHMENT_NAME
     namespace: default
    spec:
     connectionPreference: ACCEPT_AUTOMATIC
     natSubnets:
     - SUBNET_NAME
     proxyProtocol: false
     resourceRef:
       kind: Service
       name: SERVICE_NAME
    

    Replace the following:

    • SERVICE_ATTACHMENT_NAME: the name of the new service attachment.
    • SUBNET_NAME: the name of the new subnet. In GKE versions 1.22.4-gke.100 and later, you can specify a subnet in a different project by using the fully qualified resource URL for this field. You can get the fully qualified resource URL using the command gcloud compute networks subnets describe. For a Shared VPC configuration, use the following format: projects/HOST_PROJECT_ID/regions/COMPUTE_REGION/subnetworks/SUBNET_NAME.

    For more information about the manifest fields, see the service attachment fields.

  7. Apply the manifest to your cluster:

    kubectl apply -f my-psc.yaml
    
  8. Verify that the Private Service Connect controller created the service attachment:

    gcloud beta compute service-attachments list
    

    The output shows a service attachment with an automatically generated name:

    NAME        REGION       PRODUCER_FORWARDING_RULE          CONNECTION_PREFERENCE
    k8s1-sa-... REGION_NAME  a3fea439c870148bdba5e59c9ea9451a  ACCEPT_AUTOMATIC
    

View a ServiceAttachment

You can view the details of a ServiceAttachment using the following command:

kubectl describe serviceattachment SERVICE_ATTACHMENT_NAME

The output is similar to the following:

Name:        <sa-name>
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  networking.gke.io/v1beta1
Kind:         ServiceAttachment
Metadata:
  ...
Status:
  Forwarding Rule URL:      https://www.googleapis.com/compute/beta/projects/<project>/regions/<region>/forwardingRules/<fr-name>
  Last Modified Timestamp:  2021-07-08T01:32:39Z
  Service Attachment URL:   https://www.googleapis.com/compute/beta/projects/<projects>/regions/<region>/serviceAttachments/<gce-service-attachment-name>
Events:                     <none>

Consume a ServiceAttachment

To consume your service from another project, perform the following steps:

  1. Get the URL of the ServiceAttachment:

    kubectl get serviceattachment SERVICE_ATTACHMENT_NAME -o=jsonpath="{.status.serviceAttachmentURL}"
    

    The output is similar to the following:

      serviceAttachmentURL: https://www.googleapis.com/compute/alpha/projects/<project>/region/<region>/serviceAttachments/k8s1-...my-sa
    
  2. Create a Private Service Connect endpoint using the URL of the ServiceAttachment.

  3. Verify that you can connect to the Service that you deployed in the producer project by using a curl command from a VM in the consumer project:

    curl PSC_IP_ADDRESS
    

    Replace PSC_IP_ADDRESS with the IP address of the forwarding rule in the consumer project.

    The output is similar to the following:

    {
      "cluster_name":"cluster",
      "host_header":"10.128.15.200",
      "node_name":"gke-psc-default-pool-be9b6e0e-dvxg.c.gke_project.internal",
      "pod_name":"foo-7bf648dcfd-l5jf8",
      "pod_name_emoji":"👚",
      "project_id":"gke_project",
      "timestamp":"2021-06-29T21:32:03",
      "zone":"ZONE_NAME"
    }
    

Update a ServiceAttachment

You can update a ServiceAttachment using the following steps:

  1. Edit the ServiceAttachment manifest in my-psc.yaml:

    apiVersion: networking.gke.io/v1
    kind: ServiceAttachment
    metadata:
      name: my-sa
      namespace: default
    spec:
      connectionPreference: ACCEPT_AUTOMATIC
      natSubnets:
      - my-nat-subnet
      proxyProtocol: false
      resourceRef:
        kind: Service
        name: ilb-service
    
  2. Apply the manifest to your cluster:

    kubectl apply -f my-psc.yaml
    

Delete a ServiceAttachment

You cannot delete an internal passthrough Network Load Balancer that is connected to a service attachment. You must delete the service attachment and GKE Service separately.

  1. Delete the service attachment:

    kubectl delete serviceattachment SERVICE_ATTACHMENT_NAME --wait=false
    

    This command marks the service attachment for deletion, but the resource continues to exist. You can also wait for the deletion to finish by omitting the --wait flag.

  2. Delete the Service:

    kubectl delete svc SERVICE_NAME
    
  3. Delete the subnet:

    gcloud compute networks subnets delete SUBNET_NAME
    

ServiceAttachment fields

The ServiceAttachment has the following fields:

  • connectionPreference: the connection preference that determines how customers connect to the service. You can either use automatic project approval using ACCEPT_AUTOMATIC or explicit project approval using ACCEPT_MANUAL. For more information, see Publishing services using Private Service Connect.
  • natSubnets: a list of subnetwork resource names to use for the service attachment.
  • proxyProtocol: when set to true, the consumer source IP and Private Service Connect connection ID are available in the requests. This field is optional and defaults to false if not provided.
  • consumerAllowList: the list of consumer projects that are allowed to connect to the ServiceAttachment. This field can only be used when connectionPreference is ACCEPT_MANUAL. For more information about this field, see Publishing services using Private Service Connect.
    • project: the project ID or number for the consumer project.
    • connectionLimit: the connection limit for the consumer project. This field is optional.
    • forceSendFields: the field names to send to include in API requests. This field is optional.
    • nullFields: the field names to include in API requests with a null value. This field is optional.
  • consumerRejectList: the list of consumer project IDs or numbers that are not allowed to connect to the ServiceAttachment. This field can only be used when connectionPreference is ACCEPT_MANUAL. For more information about this field, see Publishing services using Private Service Connect.
  • resourceRef: a reference to the Kubernetes resource.

    • kind: the type of Kubernetes resource. You must use Service.
    • name: the name of the Kubernetes resource that must be in the same namespace as the internal passthrough Network Load Balancer.

Troubleshooting

You can view error messages using the following command:

kubectl get events -n NAMESPACE

Replace NAMESPACE with the namespace of the internal passthrough Network Load Balancer.

An error message similar to the following occurs if you try to delete an internal passthrough Network Load Balancer that is being used by a service attachment. You must delete the ServiceAttachment before you can delete the internal passthrough Network Load Balancer.

Error syncing load balancer: failed to ensure load balancer: googleapi:
Error 400: The forwarding_rule resource '<fwd-rule-URL>' is already being used
by '<svc-attachment-URL>', resourceInUseByAnotherResource.

What's next