This page shows you how to use multiple SSL certificates for Ingress with Internal and External load balancing.
Overview
If you want to accept HTTPS requests from your clients, the Internal or External HTTP(S) load balancer must have a certificate so it can prove its identity to your clients. The load balancer must also have a private key to complete the HTTPS handshake.
When the load balancer accepts an HTTPS request from a client, the traffic between the client and the load balancer is encrypted using TLS. However, the load balancer terminates the TLS encryption, and forwards the request without encryption to the application. When you configure an HTTP(S) load balancer through Ingress, you can configure the load balancer to present up to ten TLS certificates to the client.
The load balancer uses Server Name Indication (SNI) to determine which certificate to present to the client, based on the domain name in the TLS handshake. If the client does not use SNI, or if the client uses a domain name that does not match the Common Name (CN) in one of the certificates, the load balancer uses the first certificate listed in the Ingress. The following diagram depicts the load balancer sending traffic to different backends, depending on the domain name used in the request:
You can provide an HTTPS load balancer with SSL certificates using one of three methods:
Google-managed SSL certificates. Refer to the managed certificates page for information on how to use them.
Google Cloud SSL Certificate that you are managing yourself. It uses a pre-shared certificate previously uploaded to your Google Cloud project.
Kubernetes Secrets. The Secret holds a certificate and key that you create yourself. To use a Secret, add its name in the
tls
field of your Ingress manifest.
You can use more than one method in the same Ingress. This allows for no-downtime migrations between methods.
Minimum GKE version
You must have GKE version 1.10.2 or later to use pre-shared certificates or to specify multiple certificates in an Ingress.
Prerequisites
To do the exercises on this page, you must own two domain names. You can use Google Domains or another registrar.
The big picture
Here's an overview of the steps in this topic:
Create a Deployment.
Create a Service.
Create two certificate files and two key files or two
ManagedCertificate
objects. Ensure you configure these certificates in the same project and same namespace as where the load balancer is deployed.Create an Ingress that uses either Secrets or pre-shared certificates. As a result of creating the Ingress, GKE creates and configures an HTTP(S) load balancer.
Test the HTTP(S) load balancer.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
Creating a Deployment
Here is a manifest for a Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-mc-deployment
spec:
selector:
matchLabels:
app: products
department: sales
replicas: 3
template:
metadata:
labels:
app: products
department: sales
spec:
containers:
- name: hello
image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0"
env:
- name: "PORT"
value: "50001"
- name: hello-again
image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0"
env:
- name: "PORT"
value: "50002"
The Deployment has three Pods, and each Pod has two containers. One container
runs hello-app:1.0
and listens on TCP port 50001. The other container runs
hello-app:2.0
and listens on TCP port 50002.
Copy the manifest to a file named my-mc-deployment.yaml
, and create the
Deployment:
kubectl apply -f my-mc-deployment.yaml
Creating a Service
Here is a manifest for a Service:
apiVersion: v1
kind: Service
metadata:
name: my-mc-service
spec:
type: NodePort
selector:
app: products
department: sales
ports:
- name: my-first-port
protocol: TCP
port: 60001
targetPort: 50001
- name: my-second-port
protocol: TCP
port: 60002
targetPort: 50002
The selector
field in the Service manifest says any Pod that has the
app: products
label and the department: sales
label is a member of this
Service. So the Pods of the Deployment you created in the preceding step are
members of the Service.
The ports
field of the Service manifest is an array of
ServicePort
objects. When a client sends a request to the Service on
my-first-port
, the request is forwarded to one of the member Pods on port
50001. When a client sends a request to the Service on my-second-port
, the
request is forwarded to one of the member Pods on port 50002.
Copy the manifest to a file named my-mc-service.yaml
, and create the
Service:
kubectl apply -f my-mc-service.yaml
Creating certificates and keys
To do the exercises on this page, you need two certificates, each with a corresponding key. Each certificate must have a Common Name (CN) that is equal to a domain name that you own. You can create those certificates manually or use Google-managed certificates. If you already have two certificate files with the appropriate values for Common Name, you can skip ahead to the next section.
User-managed certs
Create your first key:
openssl genrsa -out test-ingress-1.key 2048
Create your first certificate signing request:
openssl req -new -key test-ingress-1.key -out test-ingress-1.csr \ -subj "/CN=FIRST_DOMAIN"
Replace
FIRST_DOMAIN
with a domain name that you own.For example, suppose you want the load balancer to serve requests from the
example.com
domain. Your certificate signing request would look like this:openssl req -new -key test-ingress-1.key -out test-ingress-1.csr \ -subj "/CN=example.com"
Create your first certificate:
openssl x509 -req -days 365 -in test-ingress-1.csr -signkey test-ingress-1.key \ -out test-ingress-1.crt
Create your second key:
openssl genrsa -out test-ingress-2.key 2048
Create your second certificate signing request:
openssl req -new -key test-ingress-2.key -out test-ingress-2.csr \ -subj "/CN=SECOND_DOMAIN"
Replace
SECOND_DOMAIN
with another domain name that you own.For example, suppose you want the load balancer to serve requests from the
examplepetstore.com
domain. Your certificate signing request would look like this:openssl req -new -key test-ingress-2.key -out test-ingress-2.csr \ -subj "/CN=examplepetstore.com"
Create your second certificate:
openssl x509 -req -days 365 -in test-ingress-2.csr -signkey test-ingress-2.key \ -out test-ingress-2.crt
For more information about certificates and keys, see the SSL certificates overview.
You now have two certificate files and two key files.
The remaining tasks use the following placeholders to refer to your domains, certificates, and keys:
FIRST_CERT_FILE
: the path to your first certificate file.FIRST_KEY_FILE
: the path to the key file that goes with your first certificate.FIRST_DOMAIN
: a domain name that you own.FIRST_SECRET_NAME
: the name of the Secret containing your first certificate and key.SECOND_CERT_FILE
: the path to your second certificate file.SECOND_KEY_FILE
: the path to the key file that goes with your second certificate.SECOND_DOMAIN
: a second domain name that you own.SECOND_SECRET_NAME
: the name of the Secret containing your second certificate and key.
Google-managed certs
To create Google-managed certificates, you must add ManagedCertificate
objects to the namespace of your Ingress. You can
use the following template to define certificates for your domains:
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
name: FIRST_CERT_NAME
spec:
domains:
- FIRST_DOMAIN
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
name: SECOND_CERT_NAME
spec:
domains:
- SECOND_DOMAIN
Replace the following:
FIRST_CERT_NAME
: the name of your firstManagedCertificate
object.FIRST_DOMAIN
: the first domain that you own.SECOND_CERT_NAME
: the name of the secondManagedCertificate
object.SECOND_DOMAIN
: the second domain that you own.
Specifying certificates for your Ingress
The next step is to create an Ingress object. In your Ingress manifest, you can use one of the following methods to provide certificates for the load balancer:
- Secrets
- Pre-shared certificates
- Google-managed certificates
Choose one of the methods by selecting one of the tabs:
Secrets
Creating Secrets
Create a Secret that holds your first certificate and key:
kubectl create secret tls FIRST_SECRET_NAME \ --cert FIRST_CERT_FILE --key FIRST_KEY_FILE
Create a Secret that holds your second certificate and key:
kubectl create secret tls SECOND_SECRET_NAME \ --cert SECOND_CERT_FILE --key SECOND_KEY_FILE
Creating an Ingress
Here is a manifest for an Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-mc-ingress
spec:
tls:
- secretName: FIRST_SECRET_NAME
- secretName: SECOND_SECRET_NAME
rules:
- host: FIRST_DOMAIN
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: my-mc-service
port:
number: 60001
- host: SECOND_DOMAIN
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: my-mc-service
port:
number: 60002
Copy the manifest to a file named
my-mc-ingress.yaml
. ReplaceFIRST_DOMAIN
andSECOND_DOMAIN
with domain names that you own, for exampleexample.com
andexamplepetstore.com
.Create the Ingress:
kubectl apply -f my-mc-ingress.yaml
When you create an Ingress, the GKE Ingress controller creates an HTTP(S) load balancer. Wait a minute for GKE assign an external IP address to the load balancer.
Describe your Ingress:
kubectl describe ingress my-mc-ingress
The output shows that two Secrets are associated with the Ingress. The output also shows the external IP address of the load balancer.
Name: my-mc-ingress Address: 203.0.113.1 ... TLS: FIRST_SECRET_NAME terminates SECOND_SECRET_NAME terminates Rules: Host Path Backends ---- ---- -------- FIRST_DOMAIN my-mc-service:my-first-port (<none>) SECOND_DOMAIN my-mc-service:my-second-port (<none>) Annotations: ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 3m loadbalancer-controller default/my-mc-ingress Normal CREATE 2m loadbalancer-controller ip: 203.0.113.1
Pre-shared certs
Creating pre-shared certificates
Create a certificate resource in your Google Cloud project:
gcloud compute ssl-certificates create FIRST_CERT_NAME \ --certificate FIRST_CERT_FILE --private-key FIRST_KEY_FILE
Replace the following:
FIRST_CERT_NAME
: the name of your first certificate.FIRST_CERT_FILE
: your first certificate file.FIRST_KEY_FILE
: your first key file.
Create a second certificate resource in your Google Cloud project:
gcloud compute ssl-certificates create SECOND_CERT_NAME \ --certificate SECOND_CERT_FILE --private-key SECOND_KEY_FILE
Replace the following:
SECOND_CERT_NAME
: the name of your second certificate.SECOND_CERT_FILE
: your second certificate file.SECOND_KEY_FILE
: your second key file.
View your certificate resources:
gcloud compute ssl-certificates list
The output shows you have certificate resources named
FIRST_CERT_NAME
andSECOND_CERT_NAME
:NAME CREATION_TIMESTAMP FIRST_CERT_NAME 2018-11-03T12:08:47.751-07:00 SECOND_CERT_NAME 2018-11-03T12:09:25.359-07:00
Creating GKE Ingress
Here's a manifest for an Ingress that lists pre-shared certificate resources in an annotation:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-psc-ingress
annotations:
ingress.gcp.kubernetes.io/pre-shared-cert: "FIRST_CERT_NAME,SECOND_CERT_NAME"
spec:
rules:
- host: FIRST_DOMAIN
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: my-mc-service
port:
number: 60001
- host: SECOND_DOMAIN
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: my-mc-service
port:
number: 60002
Copy the manifest to a file named
my-psc-ingress.yaml
. ReplaceFIRST_DOMAIN
andSECOND_DOMAIN
with your domain names.Create the Ingress:
kubectl apply -f my-psc-ingress.yaml
Wait a minute for GKE to assign an external IP address to the load balancer.
Describe your Ingress:
kubectl describe ingress my-psc-ingress
The output shows that the Ingress is associated with pre-shared certificates named
FIRST_CERT_NAME
andSECOND_CERT_NAME
. The output also shows the external IP address of the load balancer:Name: my-psc-ingress Address: 203.0.113.2 ... Rules: Host Path Backends ---- ---- -------- FIRST_DOMAIN my-mc-service:my-first-port (<none>) SECOND_DOMAIN my-mc-service:my-second-port (<none>) Annotations: ... ingress.gcp.kubernetes.io/pre-shared-cert: FIRST_CERT_NAME,SECOND_CERT_NAME ... ingress.kubernetes.io/ssl-cert: FIRST_CERT_NAME,SECOND_CERT_NAME Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 2m loadbalancer-controller default/my-psc-ingress Normal CREATE 1m loadbalancer-controller ip: 203.0.113.2
Google-managed certs
Creating GKE Ingress
Here's a manifest for an Ingress that lists pre-shared certificate resources in an annotation:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-gmc-ingress
annotations:
networking.gke.io/managed-certificates: "FIRST_CERT_NAME,SECOND_CERT_NAME"
spec:
rules:
- host: FIRST_DOMAIN
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: my-mc-service
port:
number: 60001
- host: SECOND_DOMAIN
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: my-mc-service
port:
number: 60002
Copy the manifest to a file named
my-gmc-ingress.yaml
. ReplaceFIRST_DOMAIN
andSECOND_DOMAIN
with your domain names.Create the Ingress:
kubectl apply -f my-gmc-ingress.yaml
Wait a minute for GKE to assign an external IP address to the load balancer.
Describe your Ingress:
kubectl describe ingress my-gmc-ingress
The output shows that the Ingress is associated with managed certificates named
FIRST_CERT_NAME
andSECOND_CERT_NAME
. GKE automatically fills theingress.gcp.kubernetes.io/pre-shared-cert
andingress.kubernetes.io/ssl-cert
annotations to point to the Google-managed certificates that you created using theManagedCertificate
objects The output also shows the external IP address of the load balancer:Name: my-gmc-ingress Address: 203.0.113.2 ... Rules: Host Path Backends ---- ---- -------- FIRST_DOMAIN my-mc-service:my-first-port (<none>) SECOND_DOMAIN my-mc-service:my-second-port (<none>) Annotations: ... ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-a6e41ce4-2b39-4334-84ce-867ff543c424,mcrt-bbff4116-f014-4800-a43a-4095bffeb4f4 ... ingress.kubernetes.io/ssl-cert: mcrt-a6e41ce4-2b39-4334-84ce-867ff543c424,mcrt-bbff4116-f014-4800-a43a-4095bffeb4f4 networking.gke.io/managed-certificates: FIRST_CERT_NAME,SECOND_CERT_NAME Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 2m loadbalancer-controller default/my-gmc-ingress Normal CREATE 1m loadbalancer-controller ip: 203.0.113.2
Testing the load balancer
Wait about five minutes for GKE to finish configuring the load balancer. In case of Google-managed certificates, it might take considerably longer to complete the configuration, as the system needs to provision the certificates and verify the DNS configuration for given domains.
To do this step, you need to own two domain names, and both of your domain names must resolve the external IP address of the HTTP(S) load balancer.
Send a request to the load balancer by using your first domain name:
curl -v https://FIRST_DOMAIN
The output shows that your first certificate was used in the TLS handshake.
If your first domain is example.com
, the output is similar to this:
...
* Trying 203.0.113.1...
...
* Connected to example.com (203.0.113.1) port 443 (#0)
...
* TLSv1.2 (IN), TLS handshake, Certificate (11):
...
* Server certificate:
* subject: CN=example.com
...
> Host: example.com
...
Hello, world!
Version: 1.0.0
...
Send a request to the load balancer by using your second domain name:
curl -v https://SECOND_DOMAIN
The output shows that your second certificate was used in the TLS handshake.
If your second domain is examplepetstore.com
, the output is similar
to this:
...
* Trying 203.0.113.1...
...
* Connected to examplepetstore.com (203.0.113.1) port 443 (#0)
...
* Server certificate:
* subject: CN=examplepetstore.com
...
> Host: examplepetstore.com
...
Hello, world!
Version: 2.0.0
The hosts field of an Ingress object
An
IngressSpec
has a tls
field that is an array of
IngressTLS
objects. Each IngressTLS
object has a hosts
field and SecretName
field.
In GKE, the hosts
field is not used. GKE
reads the Common Name (CN) from the certificate in the Secret. If the Common
Name matches the domain name in a client request, then the load balancer
presents the matching certificate to the client.
Which certificate is presented?
The load balancer chooses a certificate according to these rules:
If both Secrets and pre-shared certificates are listed in the Ingress, the pre-shared certificates take priority over Secrets. In other words, Secrets are still included but pre-shared certificates are presented first.
If no certificate has a Common Name (CN) that matches the domain name in the client request, the load balancer presents the primary certificate.
For Secrets listed in the
tls
block, the primary certificate is in the first Secret in the list.For pre-shared certificates listed in the annotation, the primary certificate is the first certificate in the list.
Certificate rotation best practices
If you wish to rotate the contents of your certificate (Secret or pre-shared), here are some best practices:
- Create a new Secret or pre-shared certificate with a different name that contains the new certificate data. Attach this resource (along with the existing one) to your Ingress using instructions provided earlier. Once satisfied with the changes, you can remove the old cert from the Ingress.
- If you don't mind disrupting traffic, you can remove the old resource from the Ingress, provision a new resource with the same name but different contents and then reattach it to the Ingress.
To avoid managing certificate rotation yourself, see the Google-managed SSL feature.
Troubleshooting
Specifying invalid or non-existent Secrets results in a Kubernetes event error. You can check Kubernetes events for an Ingress as follows:
kubectl describe ingress
This output is similar to this:
Name: my-ingress
Namespace: default
Address: 203.0.113.3
Default backend: hello-server:8080 (10.8.0.3:8080)
TLS:
my-faulty-Secret terminates
Rules:
Host Path Backends
---- ---- --------
* * my-service:443 (10.8.0.3:443)
Events:
Error during sync: cannot get certs for Ingress default/my-ingress:
Secret "my-faulty-ingress" has no 'tls.crt'
What's next
Read about using a Kubernetes Ingress object to configure an HTTP(S) load balancer.
Read the GKE network overview.
Do the tutorial on how to Configure HTTP(S) load balancing with Ingress.
Learn how to configure a static IP address and domain name
If you have an application running on multiple GKE clusters in different regions, configure a Multi Cluster Ingress to route traffic to a cluster in the region closest to the user.