Migrate from Kubernetes to Cloud Run

Cloud Run and Kubernetes both use standard container images as deployment artifacts, they both use a declarative API model with resources that can be represented in YAML files with the same standard structure.

Introduction

The Cloud Run Admin API v1 is designed to maximize portability with Kubernetes, for example, the Cloud Run Admin API resources share the same structure conventions and attribute names as Kubernetes resources. See the Cloud Run service YAML reference.

The Cloud Run Admin API v1 implements the Knative Serving API specification, but you do not need to be using Knative in your existing Kubernetes cluster in order to migrate some of your Kubernetes workloads to Cloud Run.

The primary resource of Cloud Run is the service. You can think of a Cloud Run service as a high level abstraction that looks like a Kubernetes deployment with a built-in pod autoscaler and unique endpoint. A "Pod" in Kubernetes corresponds to an "instance" in Cloud Run. We recommend transforming your Kubernetes deployments into Cloud Run services, one service at a time. You will also be able to merge some configuration of your Kubernetes Horizontal Pod Autoscaler and Kubernetes services into the Cloud Run service.

Cloud Run does not have the concept of namespace, instead the Google Cloud project is used as an isolation boundary between resources. When migrating to Cloud Run from Kubernetes, we recommend creating one Google Cloud project for each namespace. In the YAML of a Cloud Run service, the namespace value is the Google Cloud project number.

Cloud Run has a built-in zonal redundancy, this means that you do not need to provision replica to ensure your service is resilient to a zonal outage in the selected Google Cloud region.

Quickstart

This quickstart is an example of a simple migration example.

Simple resource comparison

Compare the following simple Kubernetes deployment named my-app with the equivalent Cloud Run service. Note how the YAML files are almost identical.

The parts in blue are different and need to be changed. The parts in red should be deleted because Cloud Run has a built-in autoscaler.

Kubernetes deployment	Cloud Run service
apiVersion: apps/v1 kind: Deployment metadata: name: my-app namespace: default labels: app: my-app spec: template: metadata: labels: app: my-app spec: containers: - image: gcr.io/cloudrun/hello env: - name: HELLO value: world replicas: 3 selector: matchLabels: app: my-app	apiVersion: serving.knative.dev/v1 kind: Service metadata: name: my-app namespace: 'PROJECT_NUMBER' labels: app: my-app spec: template: metadata: labels: app: my-app spec: containers: - image: gcr.io/cloudrun/hello env: - name: HELLO value: world

Kubernetes deployment

Cloud Run service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app
spec:
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: gcr.io/cloudrun/hello
        env:
        - name: HELLO
          value: world
  replicas: 3
  selector:
    matchLabels:
      app: my-app

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: my-app
  namespace: 'PROJECT_NUMBER'
  labels:
    app: my-app
spec:
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: gcr.io/cloudrun/hello
        env:
        - name: HELLO
          value: world

Migrate a simple Kubernetes deployment to Cloud Run

Download the YAML file of your deployment in the current directory with:
```
kubectl get deployment  my-app -o yaml > my-app.yaml
```
Modify the YAML to match a Cloud Run service. Update the my-app.yaml file:
- For the "kind" attribute: replace the value "Deployment" with "Service"
- For the "apiVersion" attribute: replace the value "apps/v1" with "serving.knative.dev/v1"
- Delete the metadata.namespace attribute or change its value to match your Google Cloud project number.
- Delete spec.replicas and spec.selector
Deploy the my-app.yaml file to Cloud Run using the following command, replacing REGION with the desired Google Cloud region, for example us-central1:
```
gcloud run services replace my-app.yaml --region REGION
```

Invoking a Cloud Run service is protected by an IAM permission. If you want to expose the new Cloud Run service publicly to the internet and allow unauthenticated invocations, run the following command:

gcloud run services add-iam-policy-binding my-app --member="allUsers" --role="roles/run.invoker" --region REGION

Features not supported by Cloud Run

Only workloads that are a fit for Cloud Run can be migrated.

Notably, the following Kubernetes features are not supported by Cloud Run:

Fixed numbers of replicas (a workaround is to use the same number of minimum and maximum instances
Config Maps (workaround available)
Custom horizontal Pod autoscaler strategies
Service Discovery

When migrating a YAML file from a Kubernetes deployment to a Cloud Run service, the gcloud run services replace command will return a clear error message for any attribute that is not supported by Cloud Run. Delete or update these attributes, then repeat the command until it succeeds.

You can refer to the YAML Reference for an exhaustive list of attributes support by Cloud Run.

Migrate Kubernetes resources

Migrate Kubernetes secrets

Like Kubernetes, Cloud Run supports mounting secrets as environment variables or volumes, but secrets should be stored in Secret Manager.

There are several important differences between Secret Manager secrets and Kubernetes secrets:

Allowed characters in names:
- Kubernetes secrets: [a-z0-9-.]{1,253}
- Secret Manager secrets: [a-zA-Z0-9_-]{1,255}
Versioning: Secrets from Secret Manager are versioned, while Kubernetes secrets aren't.
Payload: Secrets from Secret Manager contain a single []byte, while Kubernetes secrets contain a map<string, string>.

Follow the Secret Manager documentation to create a secret and add a new secret version for every secret key your Kubernetes app depends on.

Migrate Kubernetes ConfigMaps

Cloud Run does not have an equivalent of Kubernetes ConfigMaps, but because ConfigMaps can be seen as unencrypted Secrets, you can transform your ConfigMaps into secrets in Secret Manager instead. See instructions under Migrate Kubernetes secrets.

Migrate a Kubernetes deployment

The Kubernetes deployment is the resource that maps the closest to the Cloud Run service. We recommend starting from the YAML of your Kubernetes deployment and edit it to transform it into a Cloud Run service.

The main required changes are:

The namespace must be replaced with the Google Cloud project number.
Labels (metadata.labels and spec.template.metadata.labels) must be valid Google Cloud labels.
Containers must be stored in a supported container registry.
CPU and memory limits might need to be adjusted.
When referencing a secret, the "key" attribute is used to capture the version in Secret Manager, with "latest" referencing the latest version of the secret.
serviceAccountName should reference a Service Account in the current Google Cloud project
References to ConfigMaps (configMapKeyRef) should be replaced with references to secrets (secretKeyRef)

If your Kubernetes deployment is accessing other resources in your Kubernetes cluster or resources on a VPC, you must connect the Cloud Run service to the appropriate VPC

Migrate a Kubernetes Service

Cloud Run services automatically expose a unique endpoint that routes traffic to the container with a containerPort. After you have migrated your Kubernetes deployment to a Cloud Run service, you don't need to migrate the Kubernetes services that was routing traffic to this deployment.

Migrate a Kubernetes HorizontalPodAutoscaler

Cloud Run services have a built-in horizontal autoscaler: Cloud Run automatically scales pods (called "instances") using a combination of factors within the boundaries of its defined minimum and maximum number of instances.

Migrate the minReplicas and maxReplicas attributes of the HorizontalPodAutoscaler to the autoscaling.knative.dev/minScale and autoscaling.knative.dev/maxScale annotations of the Cloud Run service. Refer to the documentation on configuring minimum instances and maximum instances.

Migrate a Kubernetes Job

Because a Kubernetes job is similar to a Cloud Run job execution. You can migrate to a Cloud Run job and execute the job.

The following samples show the structural difference between a Kubernetes job and a Cloud Run job:

Kubernetes job	Cloud Run job
apiVersion: batch/v1 kind: Job metadata: name: my-job spec: template: spec: containers: - image: us-docker.pkg.dev/cloudrun/container/job	apiVersion: run.googleapis.com/v1 kind: Job metadata: name: my-job spec: template: spec: template: spec: containers: - image: us-docker.pkg.dev/cloudrun/container/job

Kubernetes job

Cloud Run job

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  template:
    spec:
      containers:
      - image: us-docker.pkg.dev/cloudrun/container/job

apiVersion: run.googleapis.com/v1
kind: Job
metadata:
  name: my-job
spec:
  template:
    spec:
      template:
        spec:
          containers:
          - image: us-docker.pkg.dev/cloudrun/container/job

Migration strategy

After you create the equivalent resources, exposing external endpoints behind a global external Application Load Balancer enables you to gradually migrate traffic between Cloud Run and Google Kubernetes Engine (GKE).