This tutorial shows how to combine Anthos Service Mesh with Cloud Load Balancing to expose applications in a service mesh to internet clients.
Anthos Service Mesh is a managed service mesh, based on Istio, that provides a security-enhanced, observable, and standardized communication layer for applications. Whether you use Anthos Service Mesh, Traffic Director, or Istio, a service mesh provides a holistic communications platform for clients that are communicating in the mesh. However, a challenge remains in how to connect clients that are outside the mesh to applications hosted in the mesh.
You can expose an application to clients in many ways, depending on where the client is. This tutorial shows you how to expose an application to clients by combining Cloud Load Balancing with Anthos Service Mesh in order to integrate load balancers with a service mesh. This tutorial is intended for advanced practitioners who run Anthos Service Mesh but it works for Istio on Google Kubernetes Engine too.
Mesh ingress gateway
Istio 0.8 introduced the mesh ingress gateway that provides a dedicated set of proxies whose ports are exposed to traffic coming from outside the service mesh. These mesh ingress proxies let you control L4 exposure behavior separately from application routing behavior. The proxies also let you apply routing and policy to mesh-external traffic before it arrives at an application sidecar. Mesh ingress defines the treatment of traffic when it reaches a node in the mesh, but external components must define how traffic first arrives at the mesh.
To manage this external traffic, you need a load balancer that is external to the mesh. This tutorial uses Google Cloud Load Balancing provisioned through GKE Ingress resources to automate deployment. The canonical example of this setup is an external load balancing service that (in the case of Google Cloud) deploys a public TCP/UDP load balancer. That load balancer points at the NodePorts of a GKE cluster. These NodePorts expose the Istio ingress gateway Pods, which route traffic to downstream mesh sidecar proxies. The following diagram illustrates this topology. Load balancing for internal private traffic looks similar to this architecture, except that you deploy an internal TCP/UDP load balancer instead.
Using L4 transparent load balancing with a mesh ingress gateway offers the following advantages:
- This setup simplifies deploying the load balancer.
- The load balancer provides a stable virtual IP (VIP), health checking, and reliable traffic distribution when cluster changes, node outages, or process outages occur.
- All routing rules, TLS termination, and traffic policy is handled in a single location at the mesh ingress gateway.
GKE Ingress and Services
You can provide access to applications for clients that are outside the cluster in many ways. The following table lists the Kubernetes primitives available for deploying load balancers on Google Cloud. The type of load balancer you use to expose applications to clients depends largely on whether the clients are external or internal, what kind of protocol support is required, and whether the service mesh spans multiple GKE clusters or is contained in a single cluster.
Any of the load balancer types in the following table can expose mesh-hosted applications, depending on the use case.
GKE resource | Cloud-based load balancer | Characteristics |
---|---|---|
Ingress for External HTTP(S) Load Balancing | External HTTP(S) load balancer |
L7 proxies in Google edge points of presence (PoPs) Public VIP Globally scoped Single cluster |
Ingress for Internal HTTP(S) Load Balancing | Internal HTTP(S) load balancer |
L7 proxies inside virtual private cloud (VPC) network Private VIP Regionally scoped Single cluster |
External LoadBalancer Service |
External TCP/UDP load balancer |
L4 pass-through at Google edge PoPs Public VIP Regionally scoped Single cluster |
Internal LoadBalancer Service |
Internal TCP/UDP load balancer |
L4 pass-through in VPC routing network Private VIP Regionally scoped Single cluster |
Multi Cluster Ingress (multi-cluster, external ingress) | External HTTP(S) load balancer |
L7 proxies in Google edge PoPs Public VIP Globally scoped Multi-cluster |
Although the default load balancer for Anthos Service Mesh is the external TCP/UDP load balancer, this tutorial focuses on the external HTTP(S) load balancer that you deploy by using External HTTP(S) Load Balancing. The external HTTP(S) load balancer provides integration with edge services like Identity-Aware Proxy (IAP), Google Cloud Armor, and Cloud CDN, as well as a globally distributed network of edge proxies. The next section describes the architecture and advantages of using two layers of HTTP load balancing.
Cloud ingress and mesh ingress
Deploying external L7 load balancing outside of the mesh along with a mesh ingress layer offers significant advantages, especially for internet traffic. Even though Anthos Service Mesh and Istio ingress gateways provide advanced routing and traffic management in the mesh, some functions are better served at the edge of the network. Taking advantage of internet-edge networking through Google Cloud's External HTTP(S) Load Balancing might provide significant performance, reliability, or security-related benefits over mesh-based ingress. These benefits include the following:
- Global Anycast VIP advertisement and globally distributed TLS and HTTP termination
- DDoS defense and traffic filtering at the edge with Google Cloud Armor
- API gateway functionality with IAP
- Automatic public certificate creation and rotation with Google-managed certificates
- Multi-cluster and multi-regional load balancing at the edge with Multi Cluster Ingress
This external layer of L7 load balancing is referred to as cloud ingress because it is built on cloud-managed load balancers rather than the self-hosted proxies that are used by mesh ingress. The combination of cloud ingress and mesh ingress utilizes complementary capabilities of the Google Cloud infrastructure and the mesh. The following diagram illustrates how you can combine cloud ingress and mesh ingress to serve as two load balancing layers for internet traffic.
In this topology, the cloud ingress layer sources traffic from outside of the service mesh and directs that traffic to the mesh ingress layer. The mesh ingress layer then directs traffic to the mesh-hosted application backends.
Cloud and mesh ingress topology
This section describes the complementary roles that each ingress layer fulfills when you use them together. These roles are not concrete rules, but rather guidelines that use the advantages of each layer. Variations of this pattern are likely, depending on your use case.
- Cloud ingress. When paired with mesh ingress, the cloud ingress layer is best used for edge security and global load balancing. Because the cloud ingress layer is integrated with DDoS protection, cloud firewalls, authentication, and encryption products at the edge, this layer excels at running these services outside of the mesh. The routing logic is typically straightforward at this layer, but the logic can be more complex for multi-cluster and multi-region environments. Because of the critical function of internet-facing load balancers, the cloud ingress layer is likely managed by an infrastructure team that has exclusive control over how applications are exposed and secured on the internet. This control also makes this layer less flexible and dynamic than a developer-driven infrastructure, a consideration that may impact who and how you provide administrative access to this layer.
- Mesh ingress. When paired with cloud ingress, the mesh ingress layer provides flexible routing that is close to the application. Because of this flexibility, the mesh ingress is better than cloud ingress for complex routing logic and application-level visibility. The separation between ingress layers also makes it easier for application owners to directly control this layer without impacting other teams. When you expose service mesh applications through an L4 load balancer instead of an L7 load balancer, you should terminate client TLS at the mesh ingress layer inside the mesh to help secure applications.
Health checking
One complexity of using two layers of L7 load balancing is health checking. You must configure each load balancer to check the health of the next layer to ensure that it can receive traffic. The topology in the following diagram shows how cloud ingress checks the health of the mesh ingress proxies, and the mesh, in return, checks the health of the application backends.
This topology has the following considerations:
- Cloud ingress. In this tutorial, you configure the Google Cloud load balancer through ingress to check the health of the mesh ingress proxies on their exposed health check ports. If a mesh proxy is down, or if the cluster, mesh, or region is unavailable, the Google Cloud load balancer detects this condition and doesn't send traffic to the mesh proxy.
- Mesh ingress. In the mesh application, you perform health checks on the backends directly so that you can execute load balancing and traffic management locally.
Security
The preceding topology involves several elements of security. One of the most critical elements is in how you configure encryption and deploy certificates. Ingress for External HTTP(S) Load Balancing has deep integration with Google-managed certificates. This integration automatically provisions public certificates, attaches them to a load balancer, and renews and rotates certificates all through the declarative GKE Ingress interface. Internet clients authenticate against the public certificates and connect to the external load balancer as the first hop in the Virtual Private Cloud (VPC).
The next hop, which is between the Google Front End (GFE) and the mesh ingress proxy, is encrypted by default. Network-level encryption between the GFEs and their backends is applied automatically. However, if your security requirements dictate that the platform owner retain ownership of the encryption keys, then you can enable HTTP/2 with TLS encryption between the cluster ingress (the GFE) and the mesh ingress (the envoy proxy instance). When you enable HTTP/2 with TLS encryption for this path, you can use a self-signed or public certificate to encrypt traffic because the GFE doesn't authenticate against it. This additional layer of encryption is demonstrated in this guide. To help prevent the mishandling of certificates, do not use the public certificate for the public load balancer elsewhere. Instead, we recommend that you use separate certificates in the service mesh.
If the service mesh mandates TLS, then all traffic is encrypted between sidecar proxies and to the mesh ingress. The following diagram illustrates HTTPS encryption from the client to the Google Cloud load balancer, from the load balancer to the mesh ingress proxy, and from the ingress proxy to the sidecar proxy.
Objectives
- Deploy a Google Kubernetes Engine (GKE) cluster on Google Cloud.
- Deploy an Istio-based Anthos Service Mesh on your GKE cluster.
- Configure GKE Ingress to terminate public HTTPS traffic and direct that traffic to service mesh-hosted applications.
- Deploy the Online Boutique application on the GKE cluster that you expose to clients on the internet.
Costs
This tutorial uses the following billable components of Google Cloud:
- Google Kubernetes Engine
- Compute Engine
- Cloud Load Balancing
- Anthos Service Mesh
- Google Cloud Armor
- Cloud Endpoints
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.
Before you begin
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.
-
In the Google Cloud console, activate Cloud Shell.
You run all of the terminal commands for this tutorial from Cloud Shell.
Upgrade to the latest version of the Google Cloud CLI:
gcloud components update
Set your default Google Cloud project:
export PROJECT=PROJECT export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT} --format="value(projectNumber)") gcloud config set project ${PROJECT}
Replace
PROJECT
with the project ID that you want to use for this tutorial.Create a working directory:
mkdir -p ${HOME}/edge-to-mesh cd ${HOME}/edge-to-mesh export WORKDIR=`pwd`
After you finish the tutorial, you can delete the working directory.
Creating GKE clusters
The features that are described in this tutorial require a GKE cluster version 1.16 or later.
In Cloud Shell, create a new
kubeconfig
file. This step ensures that you don't create a conflict with your existing (default)kubeconfig
file.touch edge2mesh_kubeconfig export KUBECONFIG=${WORKDIR}/edge2mesh_kubeconfig
Define environment variables for the GKE cluster:
export CLUSTER_NAME=edge-to-mesh export CLUSTER_LOCATION=us-west1-a
Enable the Google Kubernetes Engine API.
gcloud
gcloud services enable container.googleapis.com
Config Connector
This tutorial includes Config Connector resources. You can use these resources to complete the same tasks that you complete in the
gcloud
tab. To utilize these resources, install Config Connector and apply the resources in the way that works best for your environment.Use the following
Services
manifest:apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1 kind: Service metadata: annotations: cnrm.cloud.google.com/deletion-policy: "abandon" cnrm.cloud.google.com/disable-dependent-services: "false" name: container.googleapis.com spec: resourceID: container.googleapis.com projectRef: external: PROJECT
Create a GKE cluster.
gcloud
gcloud container clusters create ${CLUSTER_NAME} \ --machine-type=e2-standard-4 \ --num-nodes=4 \ --zone ${CLUSTER_LOCATION} \ --enable-ip-alias \ --workload-pool=${PROJECT}.svc.id.goog \ --release-channel rapid \ --addons HttpLoadBalancing \ --labels mesh_id=proj-${PROJECT_NUMBER}
Config Connector
Use the following
ContainerCluster
andContainerNodePool
manifests:apiVersion: container.cnrm.cloud.google.com/v1beta1 kind: ContainerNodePool metadata: annotations: cnrm.cloud.google.com/project-id: PROJECT name: edge-to-mesh spec: clusterRef: name: edge-to-mesh location: us-west1-a nodeConfig: machineType: e2-standard-4 nodeCount: 4 --- apiVersion: container.cnrm.cloud.google.com/v1beta1 kind: ContainerCluster metadata: annotations: cnrm.cloud.google.com/project-id: PROJECT cnrm.cloud.google.com/remove-default-node-pool: "true" labels: mesh_id: proj-PROJECT_NUMBER name: edge-to-mesh spec: addonsConfig: httpLoadBalancing: disabled: false location: us-west1-a initialNodeCount: 1 releaseChannel: channel: RAPID workloadIdentityConfig: workloadPool: PROJECT.svc.id.goog
Replace
PROJECT_NUMBER
with the value of thePROJECT_NUMBER
environment variable retrieved earlier.To use a cloud ingress, you must have the HTTP load balancing add-on enabled. GKE clusters have HTTP load balancing enabled by default; you must not disable it.
To use the managed Anthos Service Mesh, you must apply the
mesh_id
label on the cluster.Ensure that the cluster is running:
gcloud container clusters list
The output is similar to the following:
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS edge-to-mesh us-west1-a v1.22.6-gke.300 35.233.195.59 e2-standard-4 v1.22.6-gke.300 4 RUNNING
Connect to the cluster:
gcloud container clusters get-credentials ${CLUSTER_NAME} \ --zone ${CLUSTER_LOCATION} \ --project ${PROJECT}
Installing a service mesh
In this section, you configure the managed Anthos Service Mesh with fleet API.
Enable the required APIs:
gcloud
gcloud services enable mesh.googleapis.com
Config Connector
Use the following
Services
manifest:apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1 kind: Service metadata: annotations: cnrm.cloud.google.com/deletion-policy: "abandon" cnrm.cloud.google.com/disable-dependent-services: "false" name: mesh.googleapis.com spec: resourceID: mesh.googleapis.com projectRef: external: PROJECT
Enable Anthos Service Mesh on the fleet:
gcloud
gcloud container fleet mesh enable
Config Connector
Use the following
GKEHubFeature
manifest:apiVersion: gkehub.cnrm.cloud.google.com/v1beta1 kind: GKEHubFeature metadata: name: servicemesh spec: projectRef: external: PROJECT location: global resourceID: servicemesh
Register the cluster to the fleet:
gcloud
gcloud container fleet memberships register ${CLUSTER_NAME} \ --gke-cluster ${CLUSTER_LOCATION}/${CLUSTER_NAME} \ --enable-workload-identity
Config Connector
Use the following
GKEHubMembership
manifest:apiVersion: gkehub.cnrm.cloud.google.com/v1beta1 kind: GKEHubMembership metadata: annotations: cnrm.cloud.google.com/project-id: PROJECT name: edge-to-mesh spec: location: global authority: issuer: https://container.googleapis.com/v1/projects/PROJECT/locations/us-west1-a/clusters/edge-to-mesh endpoint: gkeCluster: resourceRef: name: edge-to-mesh
Enable automatic control plane management and managed data plane:
gcloud
gcloud container fleet mesh update \ --management automatic \ --memberships ${CLUSTER_NAME}
Config Connector
Use the following
GKEHubFeatureMembership
manifest:apiVersion: gkehub.cnrm.cloud.google.com/v1beta1 kind: GKEHubFeatureMembership metadata: name: servicemesh-membership spec: projectRef: external: PROJECT_ID location: global membershipRef: name: edge-to-mesh featureRef: name: servicemesh mesh: management: MANAGEMENT_AUTOMATIC
After a few minutes, verify that the control plane status is
ACTIVE
:gcloud container fleet mesh describe
The output is similar to the following:
... membershipSpecs: projects/841956571429/locations/global/memberships/edge-to-mesh: mesh: management: MANAGEMENT_AUTOMATIC membershipStates: projects/841956571429/locations/global/memberships/edge-to-mesh: servicemesh: controlPlaneManagement: details: - code: REVISION_READY details: 'Ready: asm-managed-rapid' state: ACTIVE dataPlaneManagement: details: - code: OK details: Service is running. state: ACTIVE state: code: OK description: 'Revision(s) ready for use: asm-managed-rapid.' updateTime: '2022-09-29T05:30:28.320896186Z' name: projects/your-project/locations/global/features/servicemesh resourceState: state: ACTIVE ...
Deploying GKE Ingress
In the following steps, you deploy the external HTTP(S) load balancer through GKE's Ingress controller. The Ingress resource automates the provisioning of the load balancer, its TLS certificates, and backend health checking. Additionally, you use Cloud Endpoints to automatically provision a public DNS name for the application.
Install an ingress gateway
As a security best practice, we recommend that you deploy the ingress gateway in a different namespace from the control plane.
In Cloud Shell, create a dedicated
asm-ingress
namespace:kubectl create namespace asm-ingress
Add a namespace label to the
asm-ingress
namespace:kubectl label namespace asm-ingress istio-injection=enabled
The output is similar to the following:
namespace/asm-ingress labeled
Labeling the
asm-ingress
namespace withistio-injection=enabled
instructs Anthos Service Mesh to automatically inject Envoy sidecar proxies when an application is deployed.Run the following command to create the
Deployment
manifest asingress-deployment.yaml
:cat <<EOF > ingress-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: asm-ingressgateway namespace: asm-ingress spec: selector: matchLabels: asm: ingressgateway template: metadata: annotations: # This is required to tell Anthos Service Mesh to inject the gateway with the # required configuration. inject.istio.io/templates: gateway labels: asm: ingressgateway spec: securityContext: fsGroup: 1337 runAsGroup: 1337 runAsNonRoot: true runAsUser: 1337 containers: - name: istio-proxy securityContext: allowPrivilegeEscalation: false capabilities: drop: - all privileged: false readOnlyRootFilesystem: true image: auto # The image will automatically update each time the pod starts. resources: limits: cpu: 2000m memory: 1024Mi requests: cpu: 100m memory: 128Mi serviceAccountName: asm-ingressgateway --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: asm-ingressgateway spec: maxReplicas: 5 minReplicas: 3 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: asm-ingressgateway --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: asm-ingressgateway namespace: asm-ingress rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "watch", "list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: asm-ingressgateway namespace: asm-ingress roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: asm-ingressgateway subjects: - kind: ServiceAccount name: asm-ingressgateway --- apiVersion: v1 kind: ServiceAccount metadata: name: asm-ingressgateway namespace: asm-ingress EOF
This
Deployment
has its ownServiceAccount
with associatedRole
andRoleBinding
allowing the gateway to access certificates.Deploy
ingress-deployment.yaml
in your cluster to create theDeployment
resource:kubectl apply -f ingress-deployment.yaml
The output is similar to the following:
deployment.apps/asm-ingressgateway configured role.rbac.authorization.k8s.io/asm-ingressgateway configured rolebinding.rbac.authorization.k8s.io/asm-ingressgateway configured serviceaccount/asm-ingressgateway created
Ensure that all deployments are up and running:
kubectl wait --for=condition=available --timeout=600s deployment --all -n asm-ingress
The output is similar to the following:
deployment.apps/asm-ingressgateway condition met
Run the following command to create the
Service
manifest asingress-service.yaml
:cat <<EOF > ingress-service.yaml apiVersion: v1 kind: Service metadata: name: asm-ingressgateway namespace: asm-ingress annotations: cloud.google.com/neg: '{"ingress": true}' cloud.google.com/backend-config: '{"default": "ingress-backendconfig"}' cloud.google.com/app-protocols: '{"https":"HTTP2"}' # HTTP/2 with TLS encryption labels: asm: ingressgateway spec: ports: # status-port exposes a /healthz/ready endpoint that can be used with GKE Ingress health checks - name: status-port port: 15021 protocol: TCP targetPort: 15021 # Any ports exposed in Gateway resources should be exposed here. - name: http2 port: 80 targetPort: 8080 - name: https port: 443 targetPort: 8443 selector: asm: ingressgateway type: ClusterIP EOF
This
Service
has the following annotations that set parameters for the Ingress load balancer when it's deployed:cloud.google.com/backend-config
refers to the name of a custom resource calledBackendConfig
. The Ingress controller usesBackendConfig
to set parameters on the Google CloudBackendService
resource. You use this resource in the next step to define custom parameters of the Google Cloud health check.cloud.google.com/neg: '{"ingress": true}'
enables the Ingress backends (the mesh ingress proxies in this case) for container-native load balancing. For more efficient and stable load balancing, these backends use network endpoint groups (NEGs) instead of instance groups.cloud.google.com/app-protocols: '{"https":"HTTP2"}'
directs the GFE to connect to the service mesh's ingress gateway using HTTP2 with TLS as described in Ingress for External HTTP(S) Load Balancing and External HTTP(S) Load Balancing overview, for an additional layer of encryption.
Deploy
ingress-service.yaml
in your cluster to create theService
resource:kubectl apply -f ingress-service.yaml
The output is similar to the following:
service/asm-ingressgateway created
Apply backend Service settings
In Cloud Shell, run the following command to create the
BackendConfig
manifest asingress-backendconfig.yaml
:cat <<EOF > ingress-backendconfig.yaml apiVersion: cloud.google.com/v1 kind: BackendConfig metadata: name: ingress-backendconfig namespace: asm-ingress spec: healthCheck: requestPath: /healthz/ready port: 15021 type: HTTP securityPolicy: name: edge-fw-policy EOF
BackendConfig
is a Custom Resource Definition (CRD) that defines backend parameters for Ingress load balancing. For a complete list of the backend and frontend parameters that you can configure through GKE Ingress, see Ingress features.In this tutorial, the
BackendConfig
manifest specifies custom health checks for the mesh ingress proxies. Anthos Service Mesh and Istio expose their sidecar proxy health checks on port15021
at the/healthz/ready
path. Custom health check parameters are required because the serving port (443
) of mesh ingress proxies is different from their health check port (15021
). GKE Ingress uses the following health check parameters inBackendConfig
to configure the Google Cloud load balancer health checks. A security policy that helps protect load balancing traffic from different kinds of network attacks is also referenced.healthCheck.port
defines the port that receives a health check by the Google Cloud load balancer on each Pod's IP address.healthCheck.requestPath
defines the HTTP path that receives a health check on the specified port.type
defines the protocol of the health check (in this case, HTTP).securityPolicy.name
refers to the name of a Cloud Armor security policy.
Deploy
ingress-backendconfig.yaml
in your cluster to create theBackendConfig
resource:kubectl apply -f ingress-backendconfig.yaml
The output is similar to the following:
backendconfig.cloud.google.com/ingress-backendconfig created
The
BackendConfig
parameters and theasm-ingressgateway
Service annotations are not applied to a Google Cloud load balancer until the Ingress resource is deployed. The Ingress deployment ties all of these resources together.
Define security policies
Google Cloud Armor provides DDoS defense and customizable security policies that you can attach to a load balancer through Ingress resources. In the following steps, you create a security policy that uses preconfigured rules to block cross-site scripting (XSS) attacks. This rule helps block traffic that matches known attack signatures but allows all other traffic. Your environment might use different rules depending on what your workload is.
gcloud
In Cloud Shell, create a security policy that is called
edge-fw-policy
:gcloud compute security-policies create edge-fw-policy \ --description "Block XSS attacks"
Create a security policy rule that uses the preconfigured XSS filters:
gcloud compute security-policies rules create 1000 \ --security-policy edge-fw-policy \ --expression "evaluatePreconfiguredExpr('xss-stable')" \ --action "deny-403" \ --description "XSS attack filtering"
Config Connector
Use the following ComputeSecurityPolicy
manifest:
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeSecurityPolicy
metadata:
annotations:
cnrm.cloud.google.com/project-id: PROJECT_ID
name: edge-fw-policy
spec:
rule:
- action: allow
description: "Default rule"
match:
versionedExpr: SRC_IPS_V1
config:
srcIpRanges:
- "*"
priority: 2147483647
- action: deny-403
description: "XSS attack filtering"
match:
expr:
expression: "evaluatePreconfiguredExpr('xss-stable')"
priority: 1000
The edge-fw-policy
was referenced by ingress-backendconfig
in the previous
section. When the Ingress resource is deployed, it binds this security policy
with the load balancer to help protect any backends of the
asm-ingressgateway
Service.
Configure IP addressing and DNS
In Cloud Shell, create a global static IP for the Google Cloud load balancer:
gcloud
gcloud compute addresses create ingress-ip --global
Config Connector
Use the following
ComputeAddress
manifest:apiVersion: compute.cnrm.cloud.google.com/v1beta1 kind: ComputeAddress metadata: annotations: cnrm.cloud.google.com/project-id: PROJECT_ID name: ingress-ip spec: location: global
This static IP is used by the Ingress resource and allows the IP to remain the same, even if the external load balancer changes.
Get the static IP address:
export GCLB_IP=$(gcloud compute addresses describe ingress-ip --global --format "value(address)") echo ${GCLB_IP}
To create a stable, human-friendly mapping to your Ingress IP, you must have a public DNS record. You can use any DNS provider and automation that you want. This tutorial uses Endpoints instead of creating a managed DNS zone. Endpoints provides a free Google-managed DNS record for a public IP.
Run the following command to create the YAML specification file named
dns-spec.yaml
:cat <<EOF > dns-spec.yaml swagger: "2.0" info: description: "Cloud Endpoints DNS" title: "Cloud Endpoints DNS" version: "1.0.0" paths: {} host: "frontend.endpoints.${PROJECT}.cloud.goog" x-google-endpoints: - name: "frontend.endpoints.${PROJECT}.cloud.goog" target: "${GCLB_IP}" EOF
The YAML specification defines the public DNS record in the form of
frontend.endpoints.${PROJECT}.cloud.goog
, where${PROJECT}
is your unique project number.Deploy the
dns-spec.yaml
file in your Cloud project:gcloud endpoints services deploy dns-spec.yaml
The output is similar to the following:
Operation finished successfully. The following command can describe the Operation details: gcloud endpoints operations describe operations/rollouts.frontend.endpoints.edge2mesh.cloud.goog:442b2b38-4aee-4c60-b9fc-28731657ee08 Service Configuration [2021-11-14r0] uploaded for service [frontend.endpoints.edge2mesh.cloud.goog]
Now that the IP and DNS are configured, you can generate a public certificate to secure the Ingress frontend. GKE Ingress supports Google-managed certificates as Kubernetes resources, which lets you provision them through declarative means.
Provision a TLS certificate
In Cloud Shell, run the following command to create the
ManagedCertificate
manifest asmanaged-cert.yaml
:cat <<EOF > managed-cert.yaml apiVersion: networking.gke.io/v1 kind: ManagedCertificate metadata: name: gke-ingress-cert namespace: asm-ingress spec: domains: - "frontend.endpoints.${PROJECT}.cloud.goog" EOF
This YAML file specifies that the DNS name created through Endpoints is used to provision a public certificate. Because Google fully manages the lifecycle of these public certificates, they are automatically generated and rotated on a regular basis without direct user intervention.
Deploy the
managed-cert.yaml
file in your GKE cluster:kubectl apply -f managed-cert.yaml
The output is similar to the following:
managedcertificate.networking.gke.io/gke-ingress-cert created
Inspect the
ManagedCertificate
resource to check the progress of certificate generation:kubectl describe managedcertificate gke-ingress-cert -n asm-ingress
The output is similar to the following:
Name: gke-ingress-cert Namespace: asm-ingress Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"networking.gke.io/v1","kind":"ManagedCertificate","metadata":{"annotations":{},"name":"gke-ingress-cert","namespace":"... API Version: networking.gke.io/v1 Kind: ManagedCertificate Metadata: Creation Timestamp: 2020-08-05T20:44:49Z Generation: 2 Resource Version: 1389781 Self Link: /apis/networking.gke.io/v1/namespaces/asm-ingress/managedcertificates/gke-ingress-cert UID: d74ec346-ced9-47a8-988a-6e6e9ddc4019 Spec: Domains: frontend.endpoints.edge2mesh.cloud.goog Status: Certificate Name: mcrt-306c779e-8439-408a-9634-163664ca6ced Certificate Status: Provisioning Domain Status: Domain: frontend.endpoints.edge2mesh.cloud.goog Status: Provisioning Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Create 44s managed-certificate-controller Create SslCertificate mcrt-306c779e-8439-408a-9634-163664ca6ced
When the certificate is ready, the
Certificate Status
isActive
.
Deploy the Ingress resource
In Cloud Shell, run the following command to create the
Ingress
manifest asingress.yaml
:cat <<EOF > ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gke-ingress namespace: asm-ingress annotations: kubernetes.io/ingress.allow-http: "false" kubernetes.io/ingress.global-static-ip-name: "ingress-ip" networking.gke.io/managed-certificates: "gke-ingress-cert" kubernetes.io/ingress.class: "gce" spec: defaultBackend: service: name: asm-ingressgateway port: number: 443 rules: - http: paths: - path: /* pathType: ImplementationSpecific backend: service: name: asm-ingressgateway port: number: 443 EOF
This manifest defines an Ingress resource that ties all of the previous resources together. The manifest specifies the following fields:
kubernetes.io/ingress.allow-http: "false"
disables HTTP traffic on port80
of the Google Cloud load balancer. This effectively prevents any clients connecting with unencrypted traffic because port443
only listens for HTTPS, and port80
is disabled.kubernetes.io/ingress.global-static-ip-name: "ingress-ip"
links the previously created IP address with the load balancer. This link allows the IP address to be created separately from the load balancer so that it can be reused separately from the load balancer lifecycle.networking.gke.io/managed-certificates: "gke-ingress-cert"
links this load balancer with the previously created Google-managed SSL Certificate resource.
Deploy
ingress.yaml
in your cluster:kubectl apply -f ingress.yaml
Inspect the Ingress resource to check the progress of the load balancer deployment:
kubectl describe ingress gke-ingress -n asm-ingress
The output is similar to the following:
... Annotations: ingress.kubernetes.io/https-forwarding-rule: k8s2-fs-fq3ng2uk-asm-ingress-gke-ingress-qm3qqdor ingress.kubernetes.io/ssl-cert: mcrt-306c779e-8439-408a-9634-163664ca6ced networking.gke.io/managed-certificates: gke-ingress-cert kubernetes.io/ingress.global-static-ip-name: ingress-ip ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-306c779e-8439-408a-9634-163664ca6ced ingress.kubernetes.io/backends: {"k8s-be-31610--07bdde06b914144a":"HEALTHY","k8s1-07bdde06-asm-ingress-asm-ingressgateway-443-228c1881":"HEALTHY"} ingress.kubernetes.io/forwarding-rule: k8s2-fr-fq3ng2uk-asm-ingress-gke-ingress-qm3qqdor ingress.kubernetes.io/https-target-proxy: k8s2-ts-fq3ng2uk-asm-ingress-gke-ingress-qm3qqdor ingress.kubernetes.io/target-proxy: k8s2-tp-fq3ng2uk-asm-ingress-gke-ingress-qm3qqdor ingress.kubernetes.io/url-map: k8s2-um-fq3ng2uk-asm-ingress-gke-ingress-qm3qqdor ...
The Ingress resource is ready when the
ingress.kubernetes.io/backends
annotations indicate that the backends areHEALTHY
. The annotations also show the names of different Google Cloud resources that are provisioned, including backend services, SSL certificates, and HTTPS target proxies.
Install the self-signed ingress gateway certificate
In the following steps, you generate and install a certificate (as a Kubernetes secret
resource) that enables the GFE to establish a TLS connection to the service mesh's ingress gateway. For more details about the requirements of the ingress gateway certificate, see the secure backend protocol considerations guide.
In Cloud Shell, create the private key and certificate using
openssl
:openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 \ -subj "/CN=frontend.endpoints.${PROJECT}.cloud.goog/O=Edge2Mesh Inc" \ -keyout frontend.endpoints.${PROJECT}.cloud.goog.key \ -out frontend.endpoints.${PROJECT}.cloud.goog.crt
Create the
Secret
in theasm-ingress
namespace:kubectl -n asm-ingress create secret tls edge2mesh-credential \ --key=frontend.endpoints.${PROJECT}.cloud.goog.key \ --cert=frontend.endpoints.${PROJECT}.cloud.goog.crt
Configure the ingress gateway for external load balancing
In the following steps, you create a shared Gateway
resource in the asm-ingress
namespace. Gateways are generally owned by the platform admins or network admins team. Therefore, the Gateway
resource is created in the asm-ingress
namespace owned by the platform admin and could be use in other namespaces via their own VirtualService
entries.
In Cloud Shell, run the following command to create the
Gateway
manifest asingress-gateway.yaml
:cat <<EOF > ingress-gateway.yaml apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: asm-ingressgateway namespace: asm-ingress spec: selector: asm: ingressgateway servers: - port: number: 443 name: https protocol: HTTPS hosts: - "*" # IMPORTANT: Must use wildcard here when using SSL, see note below tls: mode: SIMPLE credentialName: edge2mesh-credential EOF
Note that you must use the wildcard
*
entry in thehosts
field in theGateway
. GCLB does not use SNI extension to the backends. Using the wildcard entry sends the encrypted packet (from GCLB) to the ASM ingress gateway. ASM Ingress gateway decrypts the packet and uses the HTTP Host Header (in the decrypted packet) to make routing decisions (based onVirtualService
entries).Deploy
ingress-gateway.yaml
in your cluster:kubectl apply -f ingress-gateway.yaml
The output is similar to the following:
gateway.networking.istio.io/asm-ingressgateway created
Installing the Online Boutique sample app
In Cloud Shell, create a dedicated
onlineboutique
namespace:kubectl create namespace onlineboutique
Add a namespace label to the
onlineboutique
namespace:kubectl label namespace onlineboutique istio-injection=enabled
The output is similar to the following:
namespace/onlineboutique labeled
Labeling the
onlineboutique
namespace withistio-injection=enabled
instructs Anthos Service Mesh to automatically inject Envoy sidecar proxies when an application is deployed.Download the Kubernetes YAML files for the Online Boutique sample app:
curl -LO \ https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/main/release/kubernetes-manifests.yaml
Deploy the Online Boutique app:
kubectl apply -f kubernetes-manifests.yaml -n onlineboutique
The output is similar to the following:
deployment.apps/frontend created service/frontend created service/frontend-external created ...
Ensure that all deployments are up and running:
kubectl get pods -n onlineboutique
The output is similar to the following:
NAME READY STATUS RESTARTS AGE adservice-d854d8786-fjb7q 2/2 Running 0 3m cartservice-85b5d5b4ff-8qn7g 2/2 Running 0 2m59s checkoutservice-5f9bf659b8-sxhsq 2/2 Running 0 3m1s ...
Run the following command to create the
VirtualService
manifest asfrontend-virtualservice.yaml
:cat <<EOF > frontend-virtualservice.yaml apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: frontend-ingress namespace: onlineboutique spec: hosts: - "frontend.endpoints.${PROJECT}.cloud.goog" gateways: - asm-ingress/asm-ingressgateway http: - route: - destination: host: frontend port: number: 80 EOF
Note that the
VirtualService
is created in the application namespace (onlineboutique
). Typically, the application owner decides and configures how and what traffic gets routed to thefrontend
application soVirtualService
is deployed by the app owner.Deploy
frontend-virtualservice.yaml
in your cluster:kubectl apply -f frontend-virtualservice.yaml
The output is similar to the following:
virtualservice.networking.istio.io/frontend-virtualservice created
Access the following link:
echo "https://frontend.endpoints.${PROJECT}.cloud.goog"
Your Online Boutique frontend is displayed.
To display the details of your certificate, click
View site information in your browser's address bar, and then click Certificate (Valid).The certificate viewer displays details for the managed certificate, including the expiration date and who issued the certificate.
You now have a global HTTPS load balancer serving as a frontend to your service mesh-hosted application.
Clean up
After you've finished the tutorial, you can clean up the resources you created on Google Cloud so you won't be billed for them in the future. You can either delete the project entirely or delete cluster resources and then delete the cluster.
Delete the project
- In the Google Cloud console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
Delete the individual resources
If you want to keep the Cloud project you used in this tutorial, delete the individual resources:
Delete the Ingress resource:
kubectl delete -f ingress.yaml
Delete the managed certificate:
kubectl delete -f managed-cert.yaml
Delete the Endpoints DNS entry:
gcloud endpoints services delete "frontend.endpoints.${PROJECT}.cloud.goog"
The output is similar to the following:
Are you sure? This will set the service configuration to be deleted, along with all of the associated consumer information. Note: This does not immediately delete the service configuration or data and can be undone using the undelete command for 30 days. Only after 30 days will the service be purged from the system.
When you are prompted to continue, enter Y.
The output is similar to the following:
Waiting for async operation operations/services.frontend.endpoints.edge2mesh.cloud.goog-5 to complete... Operation finished successfully. The following command can describe the Operation details: gcloud endpoints operations describe operations/services.frontend.endpoints.edge2mesh.cloud.goog-5
Delete the static IP address:
gcloud compute addresses delete ingress-ip --global
The output is similar to the following:
The following global addresses will be deleted: - [ingress-ip]
When you are prompted to continue, enter Y.
The output is similar to the following:
Deleted [https://www.googleapis.com/compute/v1/projects/edge2mesh/global/addresses/ingress-ip].
Delete the GKE cluster:
gcloud container clusters delete $CLUSTER_NAME --zone $CLUSTER_LOCATION
What's next
- Learn about more features offered by GKE Ingress that you can use with your service mesh.
- Learn about the different types of cloud load balancing available for GKE.
- Learn about the features and functionality offered by Anthos Service Mesh.
- See how to deploy Ingress across multiple GKE clusters for multi-regional load balancing.
- Explore reference architectures, diagrams, tutorials, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.