Running Django on the Google Kubernetes Engine

Django apps that run on GKE scale dynamically according to traffic.

This tutorial assumes that you're familiar with Django web development. If you're new to Django development, it's a good idea to work through writing your first Django app before continuing.

While this tutorial demonstrates Django specifically, you can use this deployment process with other Django-based frameworks, such as Wagtail and Django CMS.

This tutorial uses Django 3, which requires at least Python 3.7.

You also need to have Docker installed.

Objectives

In this tutorial, you will:

• Create and connect a Cloud SQL database.
• Create and use Kubernetes secret values.
• Deploy a Django app to Google Kubernetes Engine.

Costs

This tutorial uses the following billable components of Google Cloud:

• GKE
• Cloud SQL

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

4. Enable the Cloud SQL, GKE and Compute Engine APIs.

   Enable the APIs

5. Install and initialize the Cloud SDK.
   Note: You can run the gcloud tool in the Cloud Console without installing the Cloud SDK. To run the gcloud tool in the Cloud Console, use Cloud Shell .

6. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

7. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

8. Enable the Cloud SQL, GKE and Compute Engine APIs.

   Enable the APIs

9. Install and initialize the Cloud SDK.
   Note: You can run the gcloud tool in the Cloud Console without installing the Cloud SDK. To run the gcloud tool in the Cloud Console, use Cloud Shell .

Prepare your environment

Clone a sample app

The code for the Django sample app is in the GoogleCloudPlatform/python-docs-samples repository on GitHub.

1. You can either download the sample as a ZIP file and extract it or clone the repository to your local machine:

   git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
   

2. Go to the directory that contains the sample code:

   Linux/macOS

   cd python-docs-samples/kubernetes_engine/django_tutorial
   

   Windows

   cd python-docs-samples\kubernetes_engine\django_tutorial
   

Confirm your Python setup

This tutorial relies on Python to run the sample application on your machine. The sample code also requires installing dependencies

For more details, refer to the Python development environment guide.

1. Confirm your Python is at least version 3.7.

    python -V
   

   You should see Python 3.7.3 or higher.

   Note: If you see a version number starting with "2", you may need to run python3 instead of python. If so, remember to reference your chosen Python installation when running python commands.

2. Create a Python virtual environment and install dependencies:

   Linux/macOS

   python -m venv venv
   source venv/bin/activate
   pip install --upgrade pip
   pip install -r requirements.txt
   

   Windows

   python -m venv env
   venv\scripts\activate
   pip install --upgrade pip
   pip install -r requirements.txt
   

Download Cloud SQL Auth proxy to connect to Cloud SQL from your local machine

Note: If you are completing this tutorial from Cloud Shell, this section is not required.

When deployed, your app uses the Cloud SQL Auth proxy that is built into the Google Kubernetes Engine environment to communicate with your Cloud SQL instance. However, to test your app locally, you must install and use a local copy of the proxy in your development environment. For more details, refer to the Cloud SQL Auth proxy guide.

The Cloud SQL Auth proxy uses the Cloud SQL API to interact with your SQL instance. To do this, it requires application authenication through the gcloud.

1. Authenticate and acquire credentials for the API:

   gcloud auth application-default login
   

2. Download and install the Cloud SQL Auth proxy to your local machine.

   Linux 64-bit

   1. Download the Cloud SQL Auth proxy:

      wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
      

   2. Make the Cloud SQL Auth proxy executable:

      chmod +x cloud_sql_proxy
      

   Linux 32-bit

   1. Download the Cloud SQL Auth proxy:

      wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.386 -O cloud_sql_proxy
      

   2. If the wget command is not found, run sudo apt-get install wget and repeat the download command.
   3. Make the Cloud SQL Auth proxy executable:

      chmod +x cloud_sql_proxy
      

   macOS 64-bit

   1. Download the Cloud SQL Auth proxy:

      curl -o cloud_sql_proxy https://dl.google.com/cloudsql/cloud_sql_proxy.darwin.amd64
      

   2. Make the Cloud SQL Auth proxy executable:

      chmod +x cloud_sql_proxy
      

   macOS 32-bit

   1. Download the Cloud SQL Auth proxy:

      curl -o cloud_sql_proxy https://dl.google.com/cloudsql/cloud_sql_proxy.darwin.386
      

   2. Make the Cloud SQL Auth proxy executable:

      chmod +x cloud_sql_proxy
      

   Windows 64-bit

   Right-click https://dl.google.com/cloudsql/cloud_sql_proxy_x64.exe and select Save Link As to download the Cloud SQL Auth proxy. Rename the file to cloud_sql_proxy.exe.

   Windows 32-bit

   Right-click https://dl.google.com/cloudsql/cloud_sql_proxy_x86.exe and select Save Link As to download the Cloud SQL Auth proxy. Rename the file to cloud_sql_proxy.exe.

   Cloud SQL Auth proxy Docker image

   For convenience, several container images that contain the Cloud SQL Auth proxy are available on GitHub in the Cloud SQL Auth proxy repo. You can pull the latest image to your local machine using Docker with the following command:

   docker pull gcr.io/cloudsql-docker/gce-proxy:1.19.1
   

   Other OS

   For other operating systems not included here, you can compile the Cloud SQL Auth proxy from source.

   You can choose to move the download to somewhere common, such as a location on your PATH, or your home directory. If you choose to do this, when you start the Cloud SQL Auth proxy later on in the tutorial, remember to reference your chosen location when using cloud_sql_proxy commands.

Create backing services

This tutorial uses several Google Cloud services to provide the database, media storage, and secret storage that support the deployed Django project. These services are deployed in a specific region. For efficiency between services, all services should be deployed in the same region. For more information about the closest region to you, see Products available by region.

Set up a Cloud SQL for PostgreSQL instance

Django officially supports multiple relational databases, but offers the most support for PostgreSQL. PostgreSQL is supported by Cloud SQL, so this tutorial chooses to use that type of database.

The following section describes the creation of a PostgreSQL instance, database, and database user for the app.

1. Create the PostgreSQL instance:

   Console

   1. In the Cloud Console, go to the Cloud SQL Instances page.

      Go to the Cloud SQL Instances page

   2. Click Create Instance.

   3. Click PostgreSQL.

   4. In the Instance ID field, enter INSTANCE_NAME.

   5. Enter a password for the postgres user.

   6. Keep the default values for the other fields.

   7. Click Create.

   It takes a few minutes to create the instance and for it to be ready for use.

   gcloud

   • Create the PostgreSQL instance:

     gcloud sql instances create INSTANCE_NAME \
         --project PROJECT_ID \
         --database-version POSTGRES_13 \
         --tier db-f1-micro \
         --region REGION
     

   Replace the following:

   • INSTANCE_NAME: the Cloud SQL instance name
   • PROJECT_ID: the Google Cloud project ID
   • REGION: the Google Cloud region

   It takes a few minutes to create the instance and for it to be ready for use.

2. Within the created instance, create a database:

   Console

   1. Within your instance page, go to the Databases tab.
   2. Click Create database.
   3. In the Database name dialog, enter DATABASE_NAME.
   4. Click Create.

   gcloud

   • Create the database within the recently created instance:

     gcloud sql databases create DATABASE_NAME \
         --instance INSTANCE_NAME
     

     Replace DATABASE_NAME with a name for the database inside the instance.

3. Create a database user:

   Note: Users created this way get additional database rights. See Limit the database user privileges for an alternative method.

   Console

   1. Within your instance page, go to the Users tab.
   2. Click Add User Account.
   3. In the Add a user account to instance dialog under "Built-in Authentication":
   4. Enter the username DATABASE_USERNAME.
   5. Enter the password DATABASE_PASSWORD
   6. Click Add.

   gcloud

   • Create the user within the recently created instance:

     gcloud sql users create DATABASE_USERNAME \
         --instance INSTANCE_NAME \
         --password DATABASE_PASSWORD
     

     Replace PASSWORD with a secure password.

Create a service account

The proxy requires a service account with Editor privileges for your Cloud SQL instance. For more information about service accounts, see the Google Cloud authentication overview.

If you are connecting from Compute Engine, make sure your VM has the proper scope to connect using the Cloud SQL API.

Configure the service account to have either of the following access scopes:

• https://www.googleapis.com/auth/sqlservice.admin
• https://www.googleapis.com/auth/cloud-platform

1. Go to the Service accounts page of the Google Cloud Console.

   Go to the Service accounts page

2. Select the project that contains your Cloud SQL instance.
3. Click Create service account.
4. In the Create service account dialog, enter a descriptive name for the service account.
5. Change the Service account ID to a unique, recognizable value and then click Create.
6. For Role, select one of the following roles, click Continue, and then click Done:
   • Cloud SQL > Cloud SQL Client
   • Cloud SQL > Cloud SQL Editor
   • Cloud SQL > Cloud SQL Admin
   Note: To create a service account with the required permissions, you must have resourcemanager.projects.setIamPolicy permission. This permission is included in the Project Owner, Project IAM Admin, and Organization Administrator roles.
   You must also have enabled the Cloud SQL Admin API. If you are using the legacy project roles (Viewer, Editor, Owner), the service account must have at least the Editor role.
7. Click the action menu for your new service account and then select Manage keys.
8. Click the Add key drop-down menu and then click Create new key.
9. Confirm that the key type is JSON and then click Create.

   The private key file is downloaded to your machine. You can move it to another location. Keep the key file secure.

Configure the database settings

Use the following commands to set environment variables for database access. These environment variables are used for local testing.

Linux/MacOS

export DATABASE_NAME=DATABASE_NAME
export DATABASE_USER=DATABASE_USERNAME
export DATABASE_PASSWORD=DATABASE_PASSWORD

Windows

set DATABASE_USER=DATABASE_USERNAME
set DATABASE_PASSWORD=DATABASE_PASSWORD

Set up your GKE configuration

1. This application is represented in a single Kubernetes configuration called polls. In polls.yaml replace <your-project-id> with your Google Cloud project ID (PROJECT_ID).

2. Run the following command and note the value of connectionName:

   gcloud sql instances describe INSTANCE_NAME --format "value(connectionName)"
   

3. In the polls.yaml file, replace <your-cloudsql-connection-string> with the connectionName value.

Run the app on your local computer

With the backing services configured, you can now run the app on your computer. This setup allows for local development, creating a superuser, and applying database migrations.

1. In a separate terminal, start the Cloud SQL Auth proxy:

   Linux/macOS

   ./cloud_sql_proxy -instances="PROJECT_ID:REGION:INSTANCE_NAME"=tcp:5432
   

   Windows

   cloud_sql_proxy.exe -instances="PROJECT_ID:REGION:INSTANCE_NAME"=tcp:5432
   

   This step establishes a connection from your local computer to your Cloud SQL instance for local testing purposes. Keep the Cloud SQL Auth proxy running the entire time you test your app locally. Running this process in a separate terminal allows you to keep working while this process runs.

2. In a new terminal, set the Project ID locally:

   Linux/macOS

     export GOOGLE_CLOUD_PROJECT=PROJECT_ID
   

   Windows

     set GOOGLE_CLOUD_PROJECT=PROJECT_ID
   

3. Run the Django migrations to set up your models and assets:

   python manage.py makemigrations
   python manage.py makemigrations polls
   python manage.py migrate
   python manage.py collectstatic
   

4. Start the Django web server:

   python manage.py runserver
   

5. In your browser, go to http://localhost:8000.

   The page displays the following text: "Hello, world. You're at the polls index." The Django web server running on your computer delivers the sample app pages.

6. Press Ctrl/Cmd+C to stop the local web server.

Use the Django admin console

In order to log into Django's admin console, you need to create a superuser. Since you have a locally accessible connection to the database, you can run management commands:

1. Create a superuser. You will be prompted to enter a username, email, and password.

   python manage.py createsuperuser
   

2. Start a local web server:

   python manage.py runserver
   

3. In your browser, go to http://localhost:8000/admin.

4. Log in to the admin site using the username and password you used when you ran createsuperuser.

Note: Since you are connected to your Cloud SQL database, you will log into your deployed Django application with the same credentials.

Deploy the app to GKE

When the app is deployed to Google Cloud, it uses the Gunicorn server. Gunicorn doesn't serve static content, so the app uses Cloud Storage to serve static content.

Collect and upload static resources

1. Create a Cloud Storage bucket and make it publicly readable.

   gsutil mb gs://PROJECT_ID_MEDIA_BUCKET
   gsutil defacl set public-read gs://PROJECT_ID_MEDIA_BUCKET
   

2. Gather all the static content locally into one folder:

   python manage.py collectstatic
   

3. Upload the static content to Cloud Storage:

   gsutil -m rsync -r ./static gs://PROJECT_ID_MEDIA_BUCKET/static
   

4. In mysite/settings.py, set the value of STATIC_URL to the following URL, replacing [YOUR_GCS_BUCKET] with your bucket name:

   http://storage.googleapis.com/PROJECT_ID_MEDIA_BUCKET/static/
   

Set up GKE

1. To initialize GKE, go to the Clusters page.

   Go to the Clusters page

   When you use GKE for the first time in a project, you need to wait for the "Kubernetes Engine is getting ready. This may take a minute or more" message to disappear.

2. Create a GKE cluster:

   gcloud container clusters create polls \
     --scopes "https://www.googleapis.com/auth/userinfo.email","cloud-platform" \
     --num-nodes 4 --zone "us-central1-a"
   

   Did you get the error: "Project [PROJECT_ID] is not fully initialized with the default service accounts."?

   Initialize GKE

   If you received an error, go to the Google Cloud Console to initialize GKE in your project.

   Go to the Clusters page

   Wait for the "Kubernetes Engine is getting ready. This can take a minute or more" message to disappear.

   OK

3. After the cluster is created, use the kubectl command-line tool, which is integrated with the gcloud, to interact with your GKE cluster. Because gcloud and kubectl are separate tools, make sure kubectl is configured to interact with the right cluster.

   gcloud container clusters get-credentials polls --zone "us-central1-a"
   

Set up Cloud SQL

1. You need several secrets to enable your GKE app to connect with your Cloud SQL instance. One is required for instance-level access (connection), while the other two are required for database access. For more information about the two levels of access control, see Instance access control.

   1. To create the secret for instance-level access, provide the location ([PATH_TO_CREDENTIAL_FILE]) of the JSON service account key you downloaded when you created your service account (see Creating a service account):

      kubectl create secret generic cloudsql-oauth-credentials \
        --from-file=credentials.json=[PATH_TO_CREDENTIAL_FILE]
      

   2. To create the secrets for database access, use the SQL database, user name, and password defined in step 2 of Initializing your Cloud SQL instance:

      kubectl create secret generic cloudsql \
        --from-literal=database=DATABASE_NAME \
        --from-literal=username=DATABASE_USERNAME \
        --from-literal=password=DATABASE_PASSWORD
      

2. Retrieve the public Docker image for the Cloud SQL proxy.

   docker pull b.gcr.io/cloudsql-docker/gce-proxy
   

3. Build a Docker image, replacing <your-project-id> with your project ID.

   docker build -t gcr.io/PROJECT_ID/polls .
   

4. Configure Docker to use gcloud as a credential helper, so that you can push the image to Container Registry:

   gcloud auth configure-docker
   

5. Push the Docker image. Replace <your-project-id> with your project ID.

   docker push gcr.io/PROJECT_ID/polls
   

   Note: This command requires write access to Cloud Storage. If you run this tutorial on a Compute Engine instance, your access to Cloud Storage might be read-only. To get write access, create a service account and use the service account to authenticate on your instance.

6. Create the GKE resource:

   kubectl create -f polls.yaml
   

   Note: If you used different names (other than cloudsql-oauth-credentials and cloudsql) when creating the secrets in the previous commands, then you need to update the polls.yaml file to match those new names.

Deploy the app to GKE

After the resources are created, there are three polls pods on the cluster. Check the status of your pods:

kubectl get pods

Wait a few minutes for the pod statuses to display as Running. If the pods aren't ready or if you see restarts, you can get the logs for a particular pod to figure out the issue. [YOUR-POD-ID] is a part of the output returned by the previous kubectl get pods command.

kubectl logs [YOUR_POD_ID]

See the app run in Google Cloud

After the pods are ready, you can get the public IP address of the load balancer:

kubectl get services polls

Note the EXTERNAL-IP address, and go to http://[EXTERNAL-IP] in your browser to see the Django polls landing page and access the admin console.

Understand the code

Sample application

The Django sample app was created using standard Django tooling. The following commands create the project and the polls app:

django-admin startproject mysite
python manage.py startapp polls

The base views, models, and route configurations were copied from Writing your first Django app (Part 1 and Part 2).

Database configuration

The settings.py contains the configuration for your SQL database:

kubernetes_engine/django_tutorial/mysite/settings.py

View on GitHub

DATABASES = {
    'default': {
        # If you are using Cloud SQL for MySQL rather than PostgreSQL, set
        # 'ENGINE': 'django.db.backends.mysql' instead of the following.
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': os.getenv('DATABASE_NAME'),
        'USER': os.getenv('DATABASE_USER'),
        'PASSWORD': os.getenv('DATABASE_PASSWORD'),
        'HOST': '127.0.0.1',
        'PORT': '5432',
    }
}

Kubernetes pod configurations

The polls.yaml file specifies two Kubernetes resources. The first is the Service, which defines a consistent name and private IP address for the Django web app. The second is an HTTP load balancer with a public-facing external IP address.

kubernetes_engine/django_tutorial/polls.yaml

View on GitHub

# The polls service provides a load-balancing proxy over the polls app
# pods. By specifying the type as a 'LoadBalancer', Kubernetes Engine will
# create an external HTTP load balancer.
# For more information about Services see:
#   https://kubernetes.io/docs/concepts/services-networking/service/
# For more information about external HTTP load balancing see:
#   https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/
apiVersion: v1
kind: Service
metadata:
  name: polls
  labels:
    app: polls
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: polls

The service provides a network name and IP address, and GKE pods run the app's code behind the service. The polls.yaml file specifies a deployment that provides declarative updates for GKE pods. The service directs traffic to the deployment by matching the service's selector to the deployment's label. In this case, the selector polls is matched to the label polls.

kubernetes_engine/django_tutorial/polls.yaml

View on GitHub

apiVersion: apps/v1
kind: Deployment
metadata:
  name: polls
  labels:
    app: polls
spec:
  replicas: 3
  selector:
    matchLabels:
      app: polls
  template:
    metadata:
      labels:
        app: polls
    spec:
      containers:
      - name: polls-app
        # Replace  with your project ID or use `make template`
        image: gcr.io/<your-project-id>/polls
        # This setting makes nodes pull the docker image every time before
        # starting the pod. This is useful when debugging, but should be turned
        # off in production.
        imagePullPolicy: Always
        env:
            - name: DATABASE_NAME
              valueFrom:
                secretKeyRef:
                  name: cloudsql
                  key: database
            - name: DATABASE_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql
                  key: username
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql
                  key: password
        ports:
        - containerPort: 8080

      - image: gcr.io/cloudsql-docker/gce-proxy:1.16
        name: cloudsql-proxy
        command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                  "-instances=<your-cloudsql-connection-string>=tcp:5432",
                  "-credential_file=/secrets/cloudsql/credentials.json"]
        volumeMounts:
          - name: cloudsql-oauth-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
          - name: ssl-certs
            mountPath: /etc/ssl/certs
          - name: cloudsql
            mountPath: /cloudsql
      volumes:
        - name: cloudsql-oauth-credentials
          secret:
            secretName: cloudsql-oauth-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir: {}

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the individual resources

If you don't want to delete the project, delete the individual resources.

1. Delete the Google Kubernetes Engine cluster:

   gcloud container clusters delete polls
   

2. Delete the Docker image that you pushed to Container Registry:

   gcloud container images delete gcr.io/PROJECT_ID/polls
   

3. Delete the Cloud SQL instance:

   gcloud sql instances delete INSTANCE_NAME
   

What's next

• Learn how to configure PostgreSQL for production
• Learn more about Django on Google Cloud