This solution provides a blueprint and methodology for onboarding your workloads more securely, reliably, and cost-effectively to Kubernetes Engine. It provides guidance for configuring administrative and network access to clusters. This article assumes a working understanding of Kubernetes resources and cluster administration as well as familiarity with Google Cloud Platform (GCP) networking features.
Structuring projects, networks, and clusters
The following diagram shows the best structure for projects, networks, regions, subnetworks, zones, and clusters.
GCP creates all of its resources within a project entity. Projects are the unit of billing and allow administrators to associate IAM roles with users. When roles are applied at the project level they apply to all resources encapsulated within project. You should use projects to encapsulate your various operating environments. For example, you may have production and staging projects for operations teams as well as a test-dev project for developers. You can apply more granular and strict policies to the projects that hold your most mission critical and sensitive data and workloads, namely production, while applying permissive and flexible policies for developers in the test-dev environment to experiment.
Networks and subnetworks
Within each project you have networks with subnetworks. The Network resource holds a collection of other resources including subnetworks, external IP addresses, firewall rules, routes, your VPN, and Cloud Router. On GCP, for performance and simplicity of configuration, a Network can span many regions. Subnetworks offer you the flexibility to isolate and control traffic into or out of each region between your Kubernetes Engine clusters. By default, you will have one created in your project but it is recommended that you create and configure your own network to map to your existing IP address management (IPAM) convention. Firewall rules can then be applied with this network to filter traffic ingressing or egressing your Kubernetes Engine nodes. By default all internet traffic to your Kubernetes Engine nodes is denied.
can span multiple
zones but not regions. To control communication between subnetworks, you will
create firewall rules
allowing traffic to pass between them. Use the
during cluster or node-pool creation to appropriately tag your Kubernetes Engine
nodes for the firewall rules to take effect. In addition to firewalling, tags
can be used to create routes between your custom subnetworks. In auto-mode
routing between subnets is enabled by default,
with custom subnetworks you will need to
create the routes manually.
Managing identity and access
Below is an illustration of IAM policy layout that provides the principle of least privilege for a dev project that is set up for developers to develop and test their upcoming features and bug fixes, as well as a prod project for production traffic:
As the following table shows, there are 4 groups of users within the organization with varying levels of permissions, granted through IAM roles across the 2 projects:
||dev||Can create Kubernetes resources for the existing clusters within the project, is not allowed to create or delete clusters.|
||prod||Full administrative access to the clusters and Kubernetes resources running within the project.|
||prod||Create, modify, and delete firewall rules and SSL certificates as well as view resources that were created within each cluster including the logs of the running pods.|
||prod||Create, modify, and delete networking resources, except for firewall rules and SSL certificates.|
In addition to the 3 teams with access to the prod project, an additional
is given the
container.developer role for prod, allowing it to create, list,
and delete resources within the cluster. Service accounts can be used to give
automation scripts or deployment frameworks the ability to act on your behalf.
Deployments to your production project and clusters should go through an
In the dev project there are multiple developers working on the same application within the same cluster. This is facilitated by namespaces, which the cluster user can create. Each developer can create resources within their own namespace, therefore avoiding naming conflicts. They can also reuse the same YAML configuration files for their deployments so that their configurations stay as similar as possible during development iterations. Namespaces can also be used to create quotas on CPU, memory, and storage usage within the cluster, ensuring that one developer isn't using too many resources within the cluster. The next section discusses restricting users to operating within certain namespaces.
Kubernetes Engine clusters running Kubernetes 1.6 and above can take advantage of further restrictions to what users are authorized to do in individual clusters. Cloud IAM can provide users access to full clusters and the resources within them, but Kubernetes' Role Based Access Control (RBAC) allows you to use the Kubernetes API to further constrain the actions users can perform inside their clusters.
With RBAC, cluster administrators apply fine-grained policies to individual
namespaces within their clusters or to the cluster as a whole. The Kubernetes
command line interface
kubectl uses the active credentials from the
tool, allowing cluster admins to map roles to GCP identities (users and
service accounts) as subjects in RoleBindings.
For example, in the figure below there are two users, user-a and user-b, who have been granted the config-reader and pod-reader roles on the app-a namespace.
As another example, there are GCP project-level IAM roles that give certain users access to all clusters in a project. In addition, individual namespace- and cluster-level role bindings are added through RBAC to give fine-grained access to resources within particular clusters or namespaces.
Kubernetes includes some default roles, but as a cluster
administrator, you can create your own that map more closely to your
organizational needs. Below is an example role that allows users only to view,
edit, and update ConfigMaps but not delete them, because the
delete verb is
kind:Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata namespace: default name: config-editor rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "watch", "create", "update", "patch"]
After you have defined roles, you can apply those roles to the cluster or namespace through "bindings". Bindings associate roles to their users, groups, or service accounts. Below is an example of binding our previously created role (config-editor) to the email@example.com user and the "development" namespace.
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: config-editors namespace: development subjects: - kind: User name: firstname.lastname@example.org apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: config-editor apiGroup: rbac.authorization.k8s.io
For more information on RBAC, see the Kubernetes Engine documentation.
Image access and sharing
Making images public
You can make images public by making the objects and buckets backing them public. For more detailed instructions, see the Container Registry Access Control documentation.
Accessing images across projects
You can share container images between projects by ensuring that your Kubernetes
nodes use a service account. The
default service account
associated with your project is in the form
After you have this identifier, you can grant it access
projects where you want to use the Container Registry. Use a custom service
account that has restricted permissions, however, because the default has
to the entire project.
To use a different service account for your clusters, provide the
service account at cluster or node-pool creation by using the
--service-account flag. For example, to use the gke-sa service account in
the project my-project:
gcloud container clusters create west --service-account \ email@example.com
Kubernetes provides the Service abstraction that provides load-balancing and service discovery across sets of pods within a cluster as well as to legacy systems running outside the cluster. The sections below describe best practices for communication between Kubernetes pods and with other systems, including other Kubernetes clusters.
Communicating within the same cluster
Kubernetes allows you to define services that group pods that are running in the cluster based on a set of labels. This group of pods can be discovered within your cluster using DNS. For more information on service discovery in Kubernetes, go to the Connecting Applications with Services documentation.
A cluster-local DNS server, kube-dns, is deployed in each Kubernetes Engine cluster that handles mapping service names to healthy pod IPs. By default, the Kubernetes DNS server returns the service's cluster IP address. This IP address is static throughout the lifetime of the service. When sending traffic to this IP the iptables on the node will load balance packets across the ready pods that match the selectors of the service. These iptables are programmed automatically by the kube-proxy service running on each node.
If you want service discovery and health monitoring but would rather have the DNS service return you the IPs of pods rather than a virtual IP, you can provision the service with the ClusterIP field set to "None," which makes the service headless. In this case, the DNS server returns a list of A records that map the DNS name of your service to the A records of the ready pods that match the label selectors defined by the service. The records in the response rotate to facilitate spreading load across the various pods. Some client-side DNS resolvers may cache DNS replies, rendering the A record rotation ineffective. The advantages of using the ClusterIP are listed here in the Kubernetes documentation.
One typical use case for headless services is with StatefulSets. StatefulSets are well-suited to run stateful applications that require stable storage and networking among their replicas. This type of deployment provisions pods that have a stable network identity, meaning their hostnames can be resolved in the cluster. Although the pod's IP may change, its hostname DNS entry will be kept up to date and resolvable.
Packet flow: ClusterIP
The following diagram shows the DNS response and packet flow of a standard Kubernetes service. While pod IPs are routable from outside the cluster, a service's cluster IP is only accessible within the cluster. These virtual IPs are implemented by doing destination network address translation (DNAT) in each Kubernetes node. The kube-proxy service running on each node keeps forwarding rules up to date on each node that map the cluster IP to the IPs of healthy pods across the cluster. If there is a pod of the service running on the local node, then that pod is used, otherwise a random pod in the cluster is chosen.
For more information on how Service IPs are implemented, go to the Kubernetes documentation. For a deep dive into Kubernetes Engine networking watch the Next 2017 talk on YouTube:
Below is an example of the DNS response and traffic pattern for a headless service. Pod IPs are routable through the default GCP subnetwork route tables and are accessed by your application directly.
Connecting to a Kubernetes Engine cluster from inside GCP
To connect to your services from outside of your cluster but within the GCP network's private IP space, you should use internal load balancing. For your service to be reachable outside the cluster, you must expose it using a NodePort.
With this service type, Kubernetes allocates a port on all your cluster nodes that can be used to access your service from outside the cluster. Because Kubernetes Engine nodes are all members of an instance group, you can use it as a backend service for an internal load balancer. This instance group includes all your cluster nodes and is kept up to date as the cluster is scaled or upgraded. Clients of the internal load balancer must be in the same region as your cluster. To ensure that traffic from the local network can't access your services, apply ingress firewall rules to your nodes.
A connection to the internal load balancer will first hit one of the nodes in the cluster based on the configured session affinity. The kube-proxy process keeps an up-to-date set of iptables rules that can forward traffic to ready pods for the service. If no ready pods matching your service label selectors are available, then it sends your traffic to a pod on an adjacent node.
Connecting from inside a cluster to external services
In many cases it may be necessary to connect your applications running inside of Kubernetes with a service, database, or application that lives outside of the cluster. You have 3 options, as outlined below.
In Kubernetes 1.6 and above, you can configure the cluster internal DNS service (kube-dns) to forward DNS queries for a certain domain to an external DNS server. This is useful when you have authoritative DNS servers that should be queried for a domain that your Kubernetes pods will need to leverage.
External name services
External name services allow you to map a DNS record to a service name within the cluster. In this case, DNS lookups for the in-cluster service will return a CNAME record of your choosing. You should use this if you only have a few records that you want to map back to existing DNS services.
Services without selectors
You can create services without a selector and then manually add endpoints to it to populate service discovery with the correct values. This allows you to use the same service discovery mechanism for your in- cluster services while ensuring that systems without service discovery through DNS are still reachable. While this approach is the most flexible, it also requires the most configuration and maintenance in the long term.
For more information on DNS, go to the Kubernetes DNS Pods and Services documentation page.
Receiving traffic from the Internet to your cluster
Traffic from the Internet can be directed to your services running in Kubernetes by using two different methods, namely network- or HTTP(s)-load balancing.
Kubernetes services should be created as LoadBalancer type for external TCP/UDP load balancing. Kubernetes creates a Network Load Balancer in your GCP project and maps it to the nodes of your Kubernetes Engine cluster. This is an easy way to get load balancing for your TCP and UDP workloads with minimal configuration. The Network Load Balancer is scoped regionally, so it can only balance traffic against pods running within the same region.
For HTTP(S) Load Balancing, you should take advantage of
Google's Global HTTP(S) Load Balancer,
which can load balance traffic across multiple regions using a single anycast IP
address. In Kubernetes, you can create Ingress Resources that allow you to map hostnames and paths to services
within your cluster. For ingress to work properly your services must be created
Kubernetes Engine nodes are provisioned as
in Compute Engine. As such,
they adhere to the same
stateful firewall mechanism as other instances. These
firewall rules are applied within your network to instances by
Each node pool
receives its own set of
tags that you can use in rules. By default, each node pool receives a single tag
that identifies it. You can add your own custom tags at either cluster or node
pool creation time using the
gcloud command line.
For example, to allow an internal load balancer to access port 8080 on all your nodes, you would use the following commands:
gcloud compute firewall-rules create \ allow-8080-fwr --target-tags allow-8080 --allow tcp:8080 \ --network gke --source-range 220.127.116.11/22 gcloud container clusters create my-cluster --tags allow-8080
Below is an example tagging one cluster so that internet traffic can access nodes on port 30000 while the other cluster is tagged to allow traffic from the VPN to port 40000. This is useful when exposing a service through a NodePort that should only be accessible using privileged networks like a VPN back to a corporate data center, or from another cluster within your project.
Connecting to an on-premises data center
There are several Cloud Interconnect options for connecting to on-premises data centers. These options are not mutually exclusive, so you may have a combination, based on workload and requirements:
- Internet for workloads that aren't data intensive or latency sensitive. Google has more than 100 points of presence (PoPs) connecting to service providers across the world.
- Direct Peering for workloads that require dedicated bandwidth, are latency sensitive, and access to all Google services, including the full suite of GCP products. Direct Peering is a Layer 3 connection, done by exchanging BGP routes, and thus requires a registered ASN.
- Carrier Interconnect is that same as Direct Peering, but done through a service provider. This is a great option if you don't have a registered ASN, or have existing relationships with a preferred service provider.
- Cloud VPN is configured over Layer 3 interconnect and internet options (1, 2, and 3), if IPsec encryption is required, or if you want to extend your private network into your private Compute Engine network.
- Try out other Google Cloud Platform features for yourself. Have a look at our tutorials.