7 best practices from Google for operating containers
Théo Chamley
Solutions Architect, Google Cloud
Editor's note: This is the second in a two part series on Google's best practices for building and operating containers. You can find the first post in the series here.
Google Kubernetes Engine (GKE) is one of the best places to run your workloads at scale. Kubernetes will happily run most of your applications without trouble, as long as they are containerized. But if you want your application to be easy to manage and take full advantage of Kubernetes, you need to follow some best practices. They will make your application easier to operate, monitor and debug and harder to break into.
This post goes over what you need to know and do to efficiently run containers in Kubernetes. You can go into more details on this topic by reading our new solution Best Practices for Operating Containers, or you can check out our earlier post on building containers.
1. Use the native logging mechanisms of containers
If your application runs on a Kubernetes cluster, you don’t need to do much regarding logs. A centralized logging system is probably already integrated in your Kubernetes cluster. If you are using Kubernetes Engine, this service is provided by Cloud Logging. So just keep it simple and use the native logging mechanisms of containers. Write your logs to stdout and stderr and they will be shipped, stored and indexed automatically for you.
If you want, you can also write your logs in a JSON format. This allows you to easily add metadata about your logs. You will then be able to search through your logs in Cloud Logging using those metadata.
2. Ensure that your containers are stateless and immutable
To properly work in a Kubernetes cluster, your containers should be both stateless and immutable. When they are, Kubernetes can do its job by easily creating and destroying instances of your application when and where needed.
Stateless means that any state (persistent data of any kind) is stored outside of the container. You can use different kinds of external storage if needed: Cloud Storage, Persistent Disks, Redis, Cloud SQL or another managed database.
Immutable means that a container will not be modified during its life: no updates, no patches, no configuration changes. If you need to update the application code or apply a patch, you build a new image and deploy it. We recommend that you externalize the container configuration (listening port, runtime options, and so on) in Secrets and ConfigMaps. You will be able to update them without having to build a new container image. You can use Cloud Build to easily create image build pipelines.
3. Avoid privileged containers
You don’t run your application as root on your servers, do you? If an attacker were to break into your application, they would end up as root. For the same reasons, you should not use privileged containers. If you need to modify settings on the host, you can give specific capabilities to the container through the securityContext option of Kubernetes. If you need to modify sysctls, Kubernetes has a dedicated annotation for that. As much as possible, use an Init container or a sidecar container for those privileged operations. They don’t need to be exposed to either internal or external traffic.
If you are a cluster administrator, you can use a Pod Security Policy to restrict the use of privileged containers.
4. Avoid running as root
You already know to avoid privileged containers, but if you can also avoid running your application as root inside the container, it’s even better. If an attacker finds a remote code execution vulnerability in your application running as root, and then manages to escape your container via an as-yet-unknown vulnerability, the attacker can end up as root on the host.
The best way to avoid this is to not run as root in the first place. You can do this with the USER instruction in your Dockerfile or the runAsUser in Kubernetes. If you are a cluster administrator, you can enforce this behavior with a Pod Security Policy.
5. Make your application easy to monitor
Like logging, monitoring is an integral part of application management. A popular option in the Kubernetes community for monitoring is Prometheus, a system that can automatically discover the pods and services it has to monitor. Google Cloud's operations suite for GKE is able to monitor Kubernetes clusters and also comes with its own version of Prometheus to monitor your applications.
Prometheus expects metrics to be exposed on an HTTP endpoint by your application. To easily achieve this, you can use the Prometheus client libraries. Other tools such as OpenCensus or Istio also use this format.
6. Expose the health of your application
To facilitate its management in production, an application must communicate its state to the overall system. Is the application running? Is it healthy? Is it ready to receive traffic? How is it behaving? The most common way of addressing this problem is to implement health checks. Kubernetes has two types of health checks: liveness and readiness probes.
For the liveness probe, your application should expose an HTTP endpoint that returns a “200 OK” response if the application is running and its main dependencies are met. For the readiness probe, your application should expose another HTTP endpoint that returns a “200 OK” response if the application is healthy, initialization steps are completed and any valid request does not result in an error. Kubernetes will only send traffic to your container if it is considered ready by these standards. Those two endpoints can be merged if there is no difference between the healthy and ready states.
You can read more on this topic from Google Developer Advocate Sandeep Dinesh here: Kubernetes best practices: Setting up health checks with readiness and liveness probes.
7. Carefully choose the image version
Most public and private images follow a tagging system similar to the one described in Best Practices for Building Containers. If the image uses a system close to Semantic Versioning, you must consider some tagging specifics. For example, the “latest” tag can be moved frequently from image to image; you cannot rely on it for predictable or reproducible builds or deployments.
You can use a “X.Y.Z” tag (they are almost always immutable), but if you do, you have to keep track of all the patches and updates to this image. If the image you use has a “X.Y” tag, it may be a good middle ground. By using it, you get automatic patches, but you stay on a stable version of the application.
