Using multiple SSL certificates in HTTPS load balancing with Ingress

Autopilot Standard

---

This page shows you how to configure multiple SSL certificates for Ingress resources in Google Kubernetes Engine (GKE) clusters.

Overview

If you want to accept HTTPS requests from your clients, the Application Load Balancer must have a certificate so it can prove its identity to your clients. The load balancer must also have a private key to complete the HTTPS handshake.

When the load balancer accepts an HTTPS request from a client, the traffic between the client and the load balancer is encrypted using TLS. However, the load balancer terminates the TLS encryption, and forwards the request without encryption to the application. When you configure an Application Load Balancer through Ingress, you can configure the load balancer to present up to ten TLS certificates to the client.

The load balancer uses Server Name Indication (SNI) to determine which certificate to present to the client, based on the domain name in the TLS handshake. If the client does not use SNI, or if the client uses a domain name that does not match the Common Name (CN) in one of the certificates, the load balancer uses the first certificate listed in the Ingress.

The following diagram shows the load balancer sending traffic to different backends, depending on the domain name used in the request:

You can provide an Application Load Balancer with SSL certificates using the following methods:

• Google-managed SSL certificates. Refer to the managed certificates page for information on how to use them.

• Google Cloud SSL certificate that you manage yourself. The SSL Certificate uses a pre-shared certificate you upload to your Google Cloud project.

• Kubernetes Secrets. The Secret holds a certificate and key that you create yourself. You add the name of the Secret to the tls field of your Ingress manifest.

You can use more than one method in the same Ingress. This allows for no-downtime migrations between methods.

The big picture

Here's an overview of the steps in this document:

1. Create a Deployment.

2. Create a Service.

3. Create two certificate files and two key files or two ManagedCertificate objects. You must configure these certificates in the same project and the same namespace as where the load balancer is deployed.

4. Create an Ingress that uses either Secrets or pre-shared certificates. When you create the Ingress, GKE creates and configures an Application Load Balancer.

5. Test the Application Load Balancer.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

• You must own two domain names. The domain names must be no longer than 63 characters.

Limitations

• Google-managed certificates are only supported with GKE Ingress using the external Application Load Balancer. Google-managed certificates don't support third-party Ingress controllers.

• For internal Application Load Balancers, you must disable HTTP in the Ingress manifest. This is not required for the external load balancer.
• You must not manually change or update the configuration of the Application Load Balancer. This means that you must not edit any of the load balancer's components, including target proxies, URL maps, and backend services. Any changes that you make are overwritten by GKE.

Create a Deployment

1. Save the following manifest as my-mc-deployment.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-mc-deployment
   spec:
     selector:
       matchLabels:
         app: products
         department: sales
     replicas: 3
     template:
       metadata:
         labels:
           app: products
           department: sales
       spec:
         containers:
         - name: hello
           image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0"
           env:
           - name: "PORT"
             value: "50001"
         - name: hello-again
           image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0"
           env:
           - name: "PORT"
             value: "50002"
   

   This manifest describes a Deployment with three Pods. Each Pod has two containers. One container runs hello-app:1.0 and listens on TCP port 50001. The other container runs hello-app:2.0 and listens on TCP port 50002.

2. Apply the manifest to your cluster:

   kubectl apply -f my-mc-deployment.yaml
   

Create a Service

1. Save the following manifest as my-mc-service.yaml:

   apiVersion: v1
   kind: Service
   metadata:
     name: my-mc-service
   spec:
     type: NodePort
     selector:
       app: products
       department: sales
     ports:
     - name: my-first-port
       protocol: TCP
       port: 60001
       targetPort: 50001
     - name: my-second-port
       protocol: TCP
       port: 60002
       targetPort: 50002
   

   This manifest describes a Service with the following fields:

   • selector: specifies that any Pod that has the app: products label and the department: sales label is a member of this Service.
   • ports: specifies that when a client sends a request to the Service on my-first-port, GKE forwards the request to one of the member Pods on port 50001. When a client sends a request to the Service on my-second-port, GKE forwards the request to one of the member Pods on port 50002.

2. Apply the manifest to your cluster:

   kubectl apply -f my-mc-service.yaml
   

Create certificates and keys

To do the exercises on this page, you need two certificates, each with a corresponding key. Each certificate must have a Common Name (CN) that is equal to a domain name that you own.

You can create those certificates manually or use Google-managed certificates.

If you already have two certificate files with the appropriate values for Common Name, you can skip ahead to the next section.

User-managed certs

1. Create your first key:

   openssl genrsa -out test-ingress-1.key 2048
   

2. Create your first certificate signing request:

   openssl req -new -key test-ingress-1.key -out test-ingress-1.csr \
       -subj "/CN=FIRST_DOMAIN"
   

   Replace FIRST_DOMAIN with a domain name that you own, such as example.com.

3. Create your first certificate:

   openssl x509 -req -days 365 -in test-ingress-1.csr -signkey test-ingress-1.key \
       -out test-ingress-1.crt
   

4. Create your second key:

   openssl genrsa -out test-ingress-2.key 2048
   

5. Create your second certificate signing request:

   openssl req -new -key test-ingress-2.key -out test-ingress-2.csr \
       -subj "/CN=SECOND_DOMAIN"
   

   Replace SECOND_DOMAIN with another domain name that you own, such as examplepetstore.com.

6. Create your second certificate:

   openssl x509 -req -days 365 -in test-ingress-2.csr -signkey test-ingress-2.key \
       -out test-ingress-2.crt
   

For more information about certificates and keys, see the SSL certificates overview.

You now have two certificate files and two key files.

The remaining tasks use the following placeholders to refer to your domains, certificates, and keys:

• FIRST_CERT_FILE: the path to your first certificate file.
• FIRST_KEY_FILE: the path to the key file that goes with your first certificate.
• FIRST_DOMAIN: a domain name that you own.
• FIRST_SECRET_NAME: the name of the Secret containing your first certificate and key.
• SECOND_CERT_FILE: the path to your second certificate file.
• SECOND_KEY_FILE: the path to the key file that goes with your second certificate.
• SECOND_DOMAIN: a second domain name that you own.
• SECOND_SECRET_NAME: the name of the Secret containing your second certificate and key.

Note: The words FIRST and SECOND don't designate an order. They help to keep track of your two domains, your two certificate files, and the corresponding key files.

Google-managed certs

To create Google-managed certificates, you must add ManagedCertificate objects to the namespace of your Ingress. You can use the following template to define certificates for your domains:

  apiVersion: networking.gke.io/v1
  kind: ManagedCertificate
  metadata:
    name: FIRST_CERT_NAME
  spec:
    domains:
      - FIRST_DOMAIN
  ---
  apiVersion: networking.gke.io/v1
  kind: ManagedCertificate
  metadata:
    name: SECOND_CERT_NAME
  spec:
    domains:
      - SECOND_DOMAIN

Replace the following:

• FIRST_CERT_NAME: the name of your first ManagedCertificate object.
• FIRST_DOMAIN: the first domain that you own.
• SECOND_CERT_NAME: the name of the second ManagedCertificate object.
• SECOND_DOMAIN: the second domain that you own.

The names of the ManagedCertificate objects are different from the names of the actual certificates that they create. You only need to know the names of the ManagedCertificate objects to use them in your Ingress.

Specify certificates for your Ingress

The next step is to create an Ingress object. In your Ingress manifest, you can use one of the following methods to provide certificates for the load balancer:

• Secrets
• Pre-shared certificates

• Google-managed certificates

Secrets

1. Create a Secret that holds your first certificate and key:

   kubectl create secret tls FIRST_SECRET_NAME \
       --cert=FIRST_CERT_FILE \
       --key=FIRST_KEY_FILE
   

2. Create a Secret that holds your second certificate and key:

   kubectl create secret tls SECOND_SECRET_NAME \
       --cert=SECOND_CERT_FILE \
       --key=SECOND_KEY_FILE
   

Create an Ingress

1. Save the following manifest as my-mc-ingress.yaml:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: my-mc-ingress
   spec:
     tls:
     - secretName: FIRST_SECRET_NAME
     - secretName: SECOND_SECRET_NAME
     rules:
     - host: FIRST_DOMAIN
       http:
         paths:
         - pathType: ImplementationSpecific
           backend:
             service:
               name: my-mc-service
               port:
                 number: 60001
     - host: SECOND_DOMAIN
       http:
         paths:
         - pathType: ImplementationSpecific
           backend:
             service:
               name: my-mc-service
               port:
                 number: 60002
   

   Replace FIRST_DOMAIN and SECOND_DOMAIN with domain names that you own, for example example.com and examplepetstore.com.

2. Apply the manifest to your cluster:

   kubectl apply -f my-mc-ingress.yaml
   

3. Describe your Ingress:

   kubectl describe ingress my-mc-ingress
   

   The output is similar to the following:

   Name: my-mc-ingress
   Address: 203.0.113.1
   ...
   TLS:
     FIRST_SECRET_NAME terminates
     SECOND_SECRET_NAME terminates
   Rules:
     Host              Path  Backends
     ----              ----  --------
     FIRST_DOMAIN
                         my-mc-service:my-first-port (<none>)
     SECOND_DOMAIN
                         my-mc-service:my-second-port (<none>)
   Annotations:
   ...
   Events:
     Type    Reason  Age   From                     Message
     ----    ------  ----  ----                     -------
     Normal  ADD     3m    loadbalancer-controller  default/my-mc-ingress
     Normal  CREATE  2m    loadbalancer-controller  ip: 203.0.113.1
   

   The output shows that two Secrets are associated with the Ingress. The output also shows the external IP address of the load balancer. If the external IP address is not set, wait a few minutes and try the command again.

Pre-shared certs

1. Create a certificate:

   gcloud compute ssl-certificates create FIRST_CERT_NAME \
       --certificate=FIRST_CERT_FILE \
       --private-key=FIRST_KEY_FILE
   

   Replace the following:

   • FIRST_CERT_NAME: the name of your first certificate.
   • FIRST_CERT_FILE: your first certificate file.
   • FIRST_KEY_FILE: your first key file.

2. Create a second certificate:

   gcloud compute ssl-certificates create SECOND_CERT_NAME \
       --certificate=SECOND_CERT_FILE \
       --private-key=SECOND_KEY_FILE
   

   Replace the following:

   • SECOND_CERT_NAME: the name of your second certificate.
   • SECOND_CERT_FILE: your second certificate file.
   • SECOND_KEY_FILE: your second key file.

3. View your certificate resources:

   gcloud compute ssl-certificates list
   

   The output is similar to the following:

   NAME                   CREATION_TIMESTAMP
   FIRST_CERT_NAME      2018-11-03T12:08:47.751-07:00
   SECOND_CERT_NAME     2018-11-03T12:09:25.359-07:00
   

Create an Ingress

1. Save the following manifest as my-psc-ingress.yaml:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: my-psc-ingress
     annotations:
       ingress.gcp.kubernetes.io/pre-shared-cert: "FIRST_CERT_NAME,SECOND_CERT_NAME"
   spec:
     rules:
     - host: FIRST_DOMAIN
       http:
         paths:
         - pathType: ImplementationSpecific
           backend:
             service:
               name: my-mc-service
               port:
                 number: 60001
     - host: SECOND_DOMAIN
       http:
         paths:
         - pathType: ImplementationSpecific
           backend:
             service:
               name: my-mc-service
               port:
                 number: 60002
   

   Replace FIRST_DOMAIN and SECOND_DOMAIN with your domain names.

   This manifest describes an Ingress that lists pre-shared certificate resources in an annotation.

2. Apply the manifest to your cluster:

   kubectl apply -f my-psc-ingress.yaml
   

3. Describe your Ingress:

   kubectl describe ingress my-psc-ingress
   

   The output is similar to the following:

   Name:             my-psc-ingress
   Address:          203.0.113.2
   ...
   Rules:
     Host              Path  Backends
     ----              ----  --------
     FIRST_DOMAIN
                         my-mc-service:my-first-port (<none>)
     SECOND_DOMAIN
                         my-mc-service:my-second-port (<none>)
   Annotations:
     ...
     ingress.gcp.kubernetes.io/pre-shared-cert:    FIRST_CERT_NAME,SECOND_CERT_NAME
     ...
     ingress.kubernetes.io/ssl-cert:               FIRST_CERT_NAME,SECOND_CERT_NAME
   Events:
     Type    Reason  Age   From                     Message
     ----    ------  ----  ----                     -------
     Normal  ADD     2m    loadbalancer-controller  default/my-psc-ingress
     Normal  CREATE  1m    loadbalancer-controller  ip: 203.0.113.2
   

   The output shows that the Ingress is associated with pre-shared certificates named FIRST_CERT_NAME and SECOND_CERT_NAME. The output also shows the external IP address of the load balancer. If the external IP address is not set, wait a few minutes and try the command again.

Google-managed certs

Create an Ingress

1. Save the following manifest as my-gmc-ingress.yaml:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: my-gmc-ingress
     annotations:
       networking.gke.io/managed-certificates: "FIRST_CERT_NAME,SECOND_CERT_NAME"
   spec:
     rules:
     - host: FIRST_DOMAIN
       http:
         paths:
         - pathType: ImplementationSpecific
           backend:
             service:
               name: my-mc-service
               port:
                 number: 60001
     - host: SECOND_DOMAIN
       http:
         paths:
         - pathType: ImplementationSpecific
           backend:
             service:
               name: my-mc-service
               port:
                 number: 60002
   

   Replace FIRST_DOMAIN and SECOND_DOMAIN with your domain names.

   This manifest describes an Ingress that lists pre-shared certificate resources in an annotation.

2. Apply the manifest to your cluster:

   kubectl apply -f my-gmc-ingress.yaml
   

3. Describe your Ingress:

   kubectl describe ingress my-gmc-ingress
   

   The output is similar to the following:

   Name:             my-gmc-ingress
   Address:          203.0.113.2
   ...
   Rules:
     Host              Path  Backends
     ----              ----  --------
     FIRST_DOMAIN
                         my-mc-service:my-first-port (<none>)
     SECOND_DOMAIN
                         my-mc-service:my-second-port (<none>)
   Annotations:
     ...
     ingress.gcp.kubernetes.io/pre-shared-cert:    mcrt-a6e41ce4-2b39-4334-84ce-867ff543c424,mcrt-bbff4116-f014-4800-a43a-4095bffeb4f4
     ...
     ingress.kubernetes.io/ssl-cert:               mcrt-a6e41ce4-2b39-4334-84ce-867ff543c424,mcrt-bbff4116-f014-4800-a43a-4095bffeb4f4
     networking.gke.io/managed-certificates:       FIRST_CERT_NAME,SECOND_CERT_NAME
   Events:
     Type    Reason  Age   From                     Message
     ----    ------  ----  ----                     -------
     Normal  ADD     2m    loadbalancer-controller  default/my-gmc-ingress
     Normal  CREATE  1m    loadbalancer-controller  ip: 203.0.113.2
   

   The output shows that the Ingress is associated with managed certificates named FIRST_CERT_NAME and SECOND_CERT_NAME. GKE automatically populates the ingress.gcp.kubernetes.io/pre-shared-cert and ingress.kubernetes.io/ssl-cert annotations with the Google-managed certificates that you created using the ManagedCertificate objects. The output also shows the external IP address of the load balancer. If the external IP address is not set, wait a few minutes and try the command again.

Test the load balancer

Wait about five minutes for GKE to finish configuring the load balancer.

If you used Google-managed certificates, it might take considerably longer to complete the configuration, as the system needs to provision the certificates and verify the DNS configuration for given domains.

To test the load balancer, you must own two domain names, and both of your domain names must resolve the external IP address of the external Application Load Balancer.

1. Send a request to the load balancer by using your first domain name:

   curl -v https://FIRST_DOMAIN
   

   You might need to use the curl -k option to perform an insecure SSL transfer, so that curl will accept self-signed certificates.

   The output is similar to the following:

   ...
   *   Trying 203.0.113.1...
   ...
   * Connected to FIRST_DOMAIN (203.0.113.1) port 443 (#0)
   ...
   * TLSv1.2 (IN), TLS handshake, Certificate (11):
   ...
   * Server certificate:
   *  subject: CN=FIRST_DOMAIN
   ...
   > Host: FIRST_DOMAIN.com
   ...
   Hello, world!
   Version: 1.0.0
   ...
   

   This output shows that your first certificate was used in the TLS handshake.

2. Send a request to the load balancer by using your second domain name:

   curl -v https://SECOND_DOMAIN
   

   The output is similar to the following:

   ...
   *   Trying 203.0.113.1...
   ...
   * Connected to SECOND_DOMAIN (203.0.113.1) port 443 (#0)
   ...
   * Server certificate:
   *  subject: CN=SECOND_DOMAIN
   ...
   > Host: SECOND_DOMAIN
   ...
   Hello, world!
   Version: 2.0.0
   

   This output shows that your second certificate was used in the TLS handshake.

The hosts field of an Ingress object

An IngressSpec has a tls field that is an array of IngressTLS objects. Each IngressTLS object has a hosts field and SecretName field. In GKE, the hosts field is not used. GKE reads the Common Name (CN) from the certificate in the Secret. If the Common Name matches the domain name in a client request, then the load balancer presents the matching certificate to the client.

Which certificate is presented?

The load balancer chooses a certificate according to these rules:

• If both Secrets and pre-shared certificates are listed in the Ingress, the pre-shared certificates take priority over Secrets. In other words, Secrets are still included but pre-shared certificates are presented first.

• If no certificate has a Common Name (CN) that matches the domain name in the client request, the load balancer presents the primary certificate.

• For Secrets listed in the tls block, the primary certificate is in the first Secret in the list.

• For pre-shared certificates listed in the annotation, the primary certificate is the first certificate in the list.

Certificate rotation best practices

If you want to rotate the contents of your Secret or pre-shared certificate, here are some best practices:

• Create a new Secret or pre-shared certificate with a different name that contains the new certificate data. Attach this resource (along with the existing one) to your Ingress using instructions provided earlier. Once satisfied with the changes, you can remove the old cert from the Ingress.
• If you don't mind disrupting traffic, you can remove the old resource from the Ingress, provision a new resource with the same name but different contents and then reattach it to the Ingress.

To avoid managing certificate rotation yourself, see Use Google-managed SSL certificates.

Troubleshooting

Specifying invalid or non-existent Secrets results in a Kubernetes event error. You can check Kubernetes events for an Ingress as follows:

kubectl describe ingress

The output is similar to the following:

Name:             my-ingress
Namespace:        default
Address:          203.0.113.3
Default backend:  hello-server:8080 (10.8.0.3:8080)
TLS:
  my-faulty-Secret terminates
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     my-service:443 (10.8.0.3:443)
Events:
   Error during sync: cannot get certs for Ingress default/my-ingress:
 Secret "my-faulty-ingress" has no 'tls.crt'

What's next

• Read the GKE network overview.

• Learn how to configure domain names with static IP addresses.

• If you have an application running on multiple GKE clusters in different regions, configure a Multi Cluster Ingress to route traffic to a cluster in the region closest to the user.