This page explains how to create an internal passthrough Network Load Balancer on Google Kubernetes Engine (GKE) across VPC networks.
Before reading this page, ensure that you're familiar with the following concepts:
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
Create an internal load balancer with Private Service Connect
As a service producer, you can use service attachments to make your services
available to service consumers in other VPC networks using
Private Service Connect. You can
create, manage, and delete service attachments using a ServiceAttachment
custom resource.
Requirements and limitations
- Limitations for Private Service Connect apply.
- You can create a service attachment in GKE versions 1.21.4-gke.300 and later.
- You cannot use the same subnet in multiple service attachment configurations.
- You must create a GKE service that uses an internal passthrough Network Load Balancer.
- You cannot specify a subnet in a different project (Shared VPC) for GKE versions earlier than 1.22.4-gke.100. For Shared VPC, ensure all requirements for Shared VPC are met.
Create a ServiceAttachment
Create a subnet.
You must create a new subnet for each
ServiceAttachment
.gcloud beta compute networks subnets create SUBNET_NAME \ --project PROJECT_ID \ --network NETWORK_NAME \ --region REGION \ --range SUBNET_RANGE \ --purpose PRIVATE_SERVICE_CONNECT
Replace the following:
SUBNET_NAME
: the name of the new subnet. In GKE versions 1.22.4-gke.100 and later, you can specify a subnet in a different project by using the fully qualified resource URL for this field. You can get the fully qualified resource URL using the commandgcloud compute networks subnets describe
.PROJECT_ID
: the ID of your Google Cloud project.NETWORK_NAME
: the name of the VPC network for the subnet.REGION
: the region for the new subnet. You must use the same region as the service that you create.SUBNET_RANGE
: the IP address range to use for the subnet.
Deploy a workload.
The following manifest describes a Deployment that runs a sample web application container image. Save the manifest as
my-deployment.yaml
:apiVersion: apps/v1 kind: Deployment metadata: name: psc-ilb spec: replicas: 3 selector: matchLabels: app: psc-ilb template: metadata: labels: app: psc-ilb spec: containers: - name: whereami image: us-docker.pkg.dev/google-samples/containers/gke/whereami:v1.2.19 ports: - name: http containerPort: 8080 readinessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 1
Apply the manifest to your cluster:
kubectl apply -f my-deployment.yaml
Create a service. The following manifest describes a service that creates an internal passthrough Network Load Balancer on TCP port 8080. Save the manifest as
my-service.yaml
:apiVersion: v1 kind: Service metadata: name: SERVICE_NAME annotations: networking.gke.io/load-balancer-type: "Internal" spec: type: LoadBalancer selector: app: psc-ilb ports: - port: 80 targetPort: 8080 protocol: TCP
Replace the following:
SERVICE_NAME
: the name of the new service.
Apply the manifest to your cluster:
kubectl apply -f my-service.yaml
Create
ServiceAttachment
.The following manifest describes a
ServiceAttachment
that exposes the service that you created to service consumers. Save the manifest asmy-psc.yaml
:apiVersion: networking.gke.io/v1 kind: ServiceAttachment metadata: name: SERVICE_ATTACHMENT_NAME namespace: default spec: connectionPreference: ACCEPT_AUTOMATIC natSubnets: - SUBNET_NAME proxyProtocol: false resourceRef: kind: Service name: SERVICE_NAME
Replace the following:
SERVICE_ATTACHMENT_NAME
: the name of the new service attachment.SUBNET_NAME
: the name of the new subnet. In GKE versions 1.22.4-gke.100 and later, you can specify a subnet in a different project by using the fully qualified resource URL for this field. You can get the fully qualified resource URL using the commandgcloud compute networks subnets describe
. For a Shared VPC configuration, use the following format:projects/HOST_PROJECT_ID/regions/COMPUTE_REGION/subnetworks/SUBNET_NAME
.
For more information about the manifest fields, see the service attachment fields.
Apply the manifest to your cluster:
kubectl apply -f my-psc.yaml
Verify that the Private Service Connect controller created the service attachment:
gcloud beta compute service-attachments list
The output shows a service attachment with an automatically generated name:
NAME REGION PRODUCER_FORWARDING_RULE CONNECTION_PREFERENCE k8s1-sa-... REGION_NAME a3fea439c870148bdba5e59c9ea9451a ACCEPT_AUTOMATIC
View a ServiceAttachment
You can view the details of a ServiceAttachment
using the following command:
kubectl describe serviceattachment SERVICE_ATTACHMENT_NAME
The output is similar to the following:
Name: <sa-name>
Namespace: default
Labels: <none>
Annotations: <none>
API Version: networking.gke.io/v1beta1
Kind: ServiceAttachment
Metadata:
...
Status:
Forwarding Rule URL: https://www.googleapis.com/compute/beta/projects/<project>/regions/<region>/forwardingRules/<fr-name>
Last Modified Timestamp: 2021-07-08T01:32:39Z
Service Attachment URL: https://www.googleapis.com/compute/beta/projects/<projects>/regions/<region>/serviceAttachments/<gce-service-attachment-name>
Events: <none>
Consume a ServiceAttachment
To consume your service from another project, perform the following steps:
Get the URL of the
ServiceAttachment
:kubectl get serviceattachment SERVICE_ATTACHMENT_NAME -o=jsonpath="{.status.serviceAttachmentURL}"
The output is similar to the following:
serviceAttachmentURL: https://www.googleapis.com/compute/alpha/projects/<project>/region/<region>/serviceAttachments/k8s1-...my-sa
Create a Private Service Connect endpoint using the URL of the
ServiceAttachment
.Verify that you can connect to the Service that you deployed in the producer project by using a
curl
command from a VM in the consumer project:curl PSC_IP_ADDRESS
Replace
PSC_IP_ADDRESS
with the IP address of the forwarding rule in the consumer project.The output is similar to the following:
{ "cluster_name":"cluster", "host_header":"10.128.15.200", "node_name":"gke-psc-default-pool-be9b6e0e-dvxg.c.gke_project.internal", "pod_name":"foo-7bf648dcfd-l5jf8", "pod_name_emoji":"👚", "project_id":"gke_project", "timestamp":"2021-06-29T21:32:03", "zone":"ZONE_NAME" }
Update a ServiceAttachment
You can update a ServiceAttachment
using the following steps:
Edit the
ServiceAttachment
manifest inmy-psc.yaml
:apiVersion: networking.gke.io/v1 kind: ServiceAttachment metadata: name: my-sa namespace: default spec: connectionPreference: ACCEPT_AUTOMATIC natSubnets: - my-nat-subnet proxyProtocol: false resourceRef: kind: Service name: ilb-service
Apply the manifest to your cluster:
kubectl apply -f my-psc.yaml
Delete a ServiceAttachment
You cannot delete an internal passthrough Network Load Balancer that is connected to a service attachment. You must delete the service attachment and GKE Service separately.
Delete the service attachment:
kubectl delete serviceattachment SERVICE_ATTACHMENT_NAME --wait=false
This command marks the service attachment for deletion, but the resource continues to exist. You can also wait for the deletion to finish by omitting the
--wait
flag.Delete the Service:
kubectl delete svc SERVICE_NAME
Delete the subnet:
gcloud compute networks subnets delete SUBNET_NAME
ServiceAttachment
fields
The ServiceAttachment
has the following fields:
connectionPreference
: the connection preference that determines how customers connect to the service. You can either use automatic project approval usingACCEPT_AUTOMATIC
or explicit project approval usingACCEPT_MANUAL
. For more information, see Publishing services using Private Service Connect.natSubnets
: a list of subnetwork resource names to use for the service attachment.proxyProtocol
: when set to true, the consumer source IP and Private Service Connect connection ID are available in the requests. This field is optional and defaults to false if not provided.consumerAllowList
: the list of consumer projects that are allowed to connect to theServiceAttachment
. This field can only be used whenconnectionPreference
isACCEPT_MANUAL
. For more information about this field, see Publishing services using Private Service Connect.project
: the project ID or number for the consumer project.connectionLimit
: the connection limit for the consumer project. This field is optional.forceSendFields
: the field names to send to include in API requests. This field is optional.nullFields
: the field names to include in API requests with a null value. This field is optional.
consumerRejectList
: the list of consumer project IDs or numbers that are not allowed to connect to theServiceAttachment
. This field can only be used whenconnectionPreference
isACCEPT_MANUAL
. For more information about this field, see Publishing services using Private Service Connect.resourceRef
: a reference to the Kubernetes resource.kind
: the type of Kubernetes resource. You must useService
.name
: the name of the Kubernetes resource that must be in the same namespace as the internal passthrough Network Load Balancer.
Troubleshooting
You can view error messages using the following command:
kubectl get events -n NAMESPACE
Replace NAMESPACE
with the namespace of the
internal passthrough Network Load Balancer.
An error message similar to the following occurs if you try to delete an
internal passthrough Network Load Balancer that is being used by a service attachment. You must delete
the ServiceAttachment
before you can delete the internal passthrough Network Load Balancer.
Error syncing load balancer: failed to ensure load balancer: googleapi:
Error 400: The forwarding_rule resource '<fwd-rule-URL>' is already being used
by '<svc-attachment-URL>', resourceInUseByAnotherResource.
What's next
- Read the GKE network overview.
- Learn more about Compute Engine load balancers.
- Learn how to create a VPC-native cluster.
- Learn about configuring authorized networks.
- Troubleshoot load balancing in GKE.