Using multiple SSL certificates in HTTP(S) load balancing with Ingress

Beta

This is a Beta release of multi-SSL. This feature is not covered by any SLA or deprecation policy and might be subject to backward-incompatible changes.

Overview

The HTTP(S) load balancer terminates user SSL/TLS connections at the global load balancing layer, then balances the connections across your instances via SSL. If your application uses HTTP(S) Load Balancing configured through Ingress, you can present up to ten SSL certificates to the load balancer.

The load balancer uses Server Name Indication (SNI) to determine which certificate to present to the client, based on the hostname in the TLS handshake message. If the client does not use SNI, or if the SNI domain does not match, then the system uses the certificate listed first in the annotation directive of your Ingress configuration YAML. The following diagram depicts the system function:

Prerequisites

This document is specific to HTTP(S) load balancing configured via Ingress. To use this procedure, you need a Google Kubernetes Engine project with HTTP(S) Load Balancing configured through Ingress. If you do not wish to use Secrets, you need to have SSL certificates already uploaded to GCP.

If you wish to perform load balancing without Ingress, please refer to the Google Cloud Load Balancing documentation.

Specifying Certificates for an Ingress

You can specify certificates for an Ingress using one of two methods:

1. Kubernetes Secrets.
2. Pre-shared certificates. These certificates already exist in GCP and are referenced via an annotation.

Note: You cannot use both methods simultaneously. All certificates must be of the same type. If you attempt to use both types, the system ignores Secrets and uses pre-shared certificates only.

Specifying Certificates using Kubernetes Secrets

Secrets are intended to hold sensitive information, such as passwords, OAuth tokens, and SSH keys. Storing sensitive information in a Secret is safer and more flexible than specifying it in plain text in a Pod definition or Docker image.

To use Kubernetes Secrets, you need to first create the certificates and upload them to your GCP project. Kubernetes Secrets are managed in the same way as other Kubernetes objects using the kubectl command-line interface. (For more information about creating and uploading Secrets, see the Secrets documentation.)

Once certificates are uploaded, declare them using the tls: directive of the Ingress configuration YAML:

kubectl create secret tls my-tls-secret \
  --key /path/to/tls.key --cert /path/to/tls.cert

When successful, this command has the following output:

Secret "my-tls-secret" created

The data directive of the Ingress spec now reflects the certificate and key specified in the kubectl command:

kind: secret
type: Opaque
apiVersion: v1
data:
  tls.crt: <base64-encoded cert>
  tls.key: <base64-encoded cert>
metadata:
  name: my-tls-secret-1
  namespace: default

To add multiple Secrets, edit the tls directive of the Ingress spec. The Ingress definition below assumes you have created three TLS Secrets secret-1, secret-2, secret-3:

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: my-ingress
spec:
  tls:
  - secretName: secret-1
  - secretName: secret-2
  - secretName: secret-3
  backend:
    serviceName: my-service
    servicePort: 443

Once you have edited the Ingress configuration, apply it to your Kubernetes cluster with kubectl:

kubectl apply -f my-ingress.yaml

Specifying pre-shared certificates

Pre-shared certificates are references to SSL certificates that already exist in GCP. The Ingress instance does not manage the pre-shared certificates; you must manage these certificates yourself. Pre-shared certificates must be created (via the API, GCP Console, or gcloud command-line interface) before you create the Ingress instance.

To specify the certificate names in the Ingress configuration, use the annotation ingress.gcp.kubernetes.io/pre-shared-cert. Pre-shared certificates take precedence over certificates named in the tls: block of the Ingress definition.

To create certificates in your GCP project, you must use the gcloud CLI (for more details about creating certificates, see the SSL Certificate documentation).

The following example demonstrates the creation of certificates my-cert1 and my-cert2 to an Ingress specification:

gcloud compute ssl-certificates create my-cert1 \
  --private-key=/path/to/tls.key --certificate=/path/to/tls.cert
gcloud compute ssl-certificates create my-cert2 \
--private-key=/path/to/tls.key --certificate=/path/to/tls.cert

Then, reference the certificates in a comma-separated list using the pre-shared-cert annotation:

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: my-ingress
  annotations:
    ingress.gcp.kubernetes.io/pre-shared-cert: "my-cert1,my-cert2"
spec:
  backend:
    serviceName: my-service
    servicePort: 443

Now, push the new configuration using kubectl:

kubectl apply -f my-ingress.yaml

Which certificate is presented?

The load balancer specifies the order in which certificates are presented:

• The primary certificate for a Secret is the first certificate specified in the tls: block.
• The primary certificate for a pre-shared certificate is the first element listed in the annotation.
• In the event that your app specifies both a Secret and a pre-shared certificate, the system ignores the Secret. The primary certificate in this case is the first element listed in the pre-shared certificate annotation.

Getting Ingress Certificate Status

Use kubectl describe to show the status of the Ingress SSL certificates currently in use:

kubectl describe ingress

Status for certificates using Secrets appears similar to:

Name:             my-ingress
Namespace:        default
Address:          35.227.232.191
Default backend:  hello-server:8080 (10.8.0.3:8080)
TLS:
  my-Secret-1 terminates
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     my-service:443 (10.8.0.3:443)
Annotations:
  target-proxy:     k8s-tp-default-basic-ingress--2222a22222d2222e
  url-map:          k8s-um-default-basic-ingress--2222a22222d2222e
  backends:         {"k8s-be-31881--2222a22222d2222e":"HEALTHY"}
  forwarding-rule:  k8s-fw-default-basic-ingress--2222a22222d2222e
  ssl-cert:         k8s-ssl-be37b1b910190168-184aeba12df7c106--2222a22222d2222e

Status for pre-shared certificates using annotations appears similar to:

Name:             my-ingress
Namespace:        default
Address:          35.227.232.191
Default backend:  hello-server:8080 (10.8.0.3:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     my-service:443 (10.8.0.3:443)
Annotations:
  ingress.gcp.kubernetes.io/pre-shared-cert: "my-cert1,mycert2"
  target-proxy:     k8s-tp-default-basic-ingress--2222a22222d2222e
  url-map:          k8s-um-default-basic-ingress--2222a22222d2222e
  backends:         {"k8s-be-31881--2222a22222d2222e":"HEALTHY"}
  forwarding-rule:  k8s-fw-default-basic-ingress--2222a22222d2222e
  ssl-cert:         my-cert-1, my-cert2

Troubleshooting

TLS certificates are invalid or do not exist

Specifying invalid or non-existent secrets results in a Kubernetes event error. You can check Kubernetes events for an Ingress as follows:

kubectl describe ingress

This command produces output similar to:

Name:             my-ingress
Namespace:        default
Address:          35.227.232.191
Default backend:  hello-server:8080 (10.8.0.3:8080)
TLS:
  my-faulty-Secret terminates
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     my-service:443 (10.8.0.3:443)
Events:
   Error during sync: cannot get certs for Ingress default/my-ingress:
 Secret "my-faulty-ingress" has no 'tls.crt'

Pre-shared certificates take precedence over TLS certificates

If you specify certificates using both tls: block and pre-shared certificates, the certificates in the annotation take precedence. As you will see when you run kubectl describe ingress, the ssl-certannotation contains only the certificates specified through the pre-shared annotation, and not the tls: block:

Name:             my-ingress
Namespace:        default
Address:          35.227.232.191
Default backend:  hello-server:8080 (10.8.0.3:8080)
TLS:
  my-Secret-1 terminates
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     my-service:443 (10.8.0.3:443)
Annotations:
  ingress.gcp.kubernetes.io/pre-shared-cert: "my-pre-shared"
  target-proxy:     k8s-tp-default-basic-ingress--2222a22222d2222e
  url-map:          k8s-um-default-basic-ingress--2222a22222d2222e
  backends:         {"k8s-be-31881--2222a22222d2222e":"HEALTHY"}
  forwarding-rule:  k8s-fw-default-basic-ingress--2222a22222d2222e
  ssl-cert:         my-pre-shared

If you receive this error, you need to choose which mechanism your app uses to handle SSL and standardize your certificates into that method.

What's next

• Read the GKE network overview.
• Read the Ingress user guide for details about Ingress features.
• Configure HTTP(S) load balancing with Ingress.
• Configure static IP and a domain name for your Ingress application using Ingress.
• If you have an application running on multiple GKE clusters in different regions, configure a multi-cluster Ingress to route traffic to a cluster in the region closest to the user.