Running Django on the App Engine standard environment

Django apps that run on App Engine standard environment scale dynamically according to traffic.

This tutorial assumes that you're familiar with Django web development. If you're new to Django development, it's a good idea to work through writing your first Django app before continuing.

While this tutorial demonstrates Django specifically, you can use this deployment process with other Django-based frameworks, such as Wagtail and Django CMS.

This tutorial uses Django 3, which requires at least Python 3.7. App Engine standard supports Python 3.7 and higher.

Objectives

In this tutorial, you will:

  • Create and connect a Cloud SQL database.
  • Create and use Secret Manager secret values.
  • Deploy a Django app to App Engine standard.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  4. Enable the Cloud SQL Admin API, Secret Manager, and Cloud Build APIs.

    Enable the APIs

  5. Install and initialize the Cloud SDK.
  6. If you haven't already done so, initialize App Engine and select your preferred region:

    gcloud app create
    

Prepare your environment

Clone a sample app

The code for the Django sample app is in the GoogleCloudPlatform/python-docs-samples repository on GitHub.

  1. You can either download the sample as a ZIP file and extract it or clone the repository to your local machine:

    git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
    
  2. Go to the directory that contains the sample code:

    Linux/macOS

    cd python-docs-samples/appengine/standard_python3/django
    

    Windows

    cd python-docs-samples\appengine\standard_python3\django
    

Confirm your Python setup

This tutorial relies on Python to run the sample application on your machine. The sample code also requires installing dependencies

For more details, refer to the Python development environment guide.

  1. Confirm your Python is at least version 3.7.

     python -V
    

    You should see Python 3.7.3 or higher.

  2. Create a Python virtual environment and install dependencies:

    Linux/macOS

    python -m venv venv
    source venv/bin/activate
    pip install --upgrade pip
    pip install -r requirements.txt
    

    Windows

    python -m venv env
    venv\scripts\activate
    pip install --upgrade pip
    pip install -r requirements.txt
    

Download Cloud SQL Auth proxy to connect to Cloud SQL from your local machine

When deployed, your app uses the Cloud SQL Auth proxy that is built into the App Engine standard environment to communicate with your Cloud SQL instance. However, to test your app locally, you must install and use a local copy of the proxy in your development environment. For more details, refer to the Cloud SQL Auth proxy guide.

The Cloud SQL Auth proxy uses the Cloud SQL API to interact with your SQL instance. To do this, it requires application authenication through the gcloud.

  1. Authenticate and acquire credentials for the API:

    gcloud auth application-default login
    
  2. Download and install the Cloud SQL Auth proxy to your local machine.

    Linux 64-bit

    1. Download the Cloud SQL Auth proxy:
      wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
      
    2. Make the Cloud SQL Auth proxy executable:
      chmod +x cloud_sql_proxy
      

    Linux 32-bit

    1. Download the Cloud SQL Auth proxy:
      wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.386 -O cloud_sql_proxy
      
    2. If the wget command is not found, run sudo apt-get install wget abd repeat the download command.
    3. Make the Cloud SQL Auth proxy executable:
      chmod +x cloud_sql_proxy
      

    macOS 64-bit

    1. Download the Cloud SQL Auth proxy:
      curl -o cloud_sql_proxy https://dl.google.com/cloudsql/cloud_sql_proxy.darwin.amd64
      
    2. Make the Cloud SQL Auth proxy executable:
      chmod +x cloud_sql_proxy
      

    macOS 32-bit

    1. Download the Cloud SQL Auth proxy:
      curl -o cloud_sql_proxy https://dl.google.com/cloudsql/cloud_sql_proxy.darwin.386
      
    2. Make the Cloud SQL Auth proxy executable:
      chmod +x cloud_sql_proxy
      

    Windows 64-bit

    Right-click https://dl.google.com/cloudsql/cloud_sql_proxy_x64.exe and select Save Link As to download the Cloud SQL Auth proxy. Rename the file to cloud_sql_proxy.exe.

    Windows 32-bit

    Right-click https://dl.google.com/cloudsql/cloud_sql_proxy_x86.exe and select Save Link As to download the Cloud SQL Auth proxy. Rename the file to cloud_sql_proxy.exe.

    Cloud SQL Auth proxy Docker image

    For convenience, several container images that contain the Cloud SQL Auth proxy are available on GitHub in the Cloud SQL Auth proxy repo. You can pull the latest image to your local machine using Docker with the following command:
    docker pull gcr.io/cloudsql-docker/gce-proxy:1.19.1
    

    Other OS

    For other operating systems not included here, you can compile the Cloud SQL Auth proxy from source.

    You can choose to move the download to somewhere common, such as a location on your PATH, or your home directory. If you choose to do this, when you start the Cloud SQL Auth proxy later on in the tutorial, remember to reference your chosen location when using cloud_sql_proxy commands.

Create backing services

This tutorial uses several Google Cloud services to provide the database, media storage, and secret storage that support the deployed Django project. These services are deployed in a specific region. For efficiency between services, it is best all services should be deployed in the same region. For more information about the closest region to you, see Products available by region.

This tutorial opts to use the integrated static asset hosting mechanisms in App Engine standard.

Set up a Cloud SQL for PostgreSQL instance

Django officially supports multiple relational databases, but offers the most support for PostgreSQL. PostgreSQL is supported by Cloud SQL, so this tutorial chooses to use that type of database.

The following section describes the creation of a PostgreSQL instance, database, and database user for the app.

  1. Create the PostgreSQL instance

    Console

    1. In the Cloud Console, go to the Cloud SQL Instances page.

      Go to the Cloud SQL Instances page

    2. Click Create Instance.

    3. Click PostgreSQL.

    4. In the Instance ID field, enter INSTANCE_NAME.

    5. Enter a password for the postgres user.

    6. Keep the default values for the other fields.

    7. Click Create.

    It takes a few minutes to create the instance and for it to be ready for use.

    gcloud

    • Create the PostgreSQL instance:

      gcloud sql instances create INSTANCE_NAME \
          --project PROJECT_ID \
          --database-version POSTGRES_13 \
          --tier db-f1-micro \
          --region REGION
      

    Replace the following:

    • INSTANCE_NAME: the Cloud SQL instance name
    • PROJECT_ID: the Google Cloud project ID
    • REGION: the Google Cloud region

    It takes a few minutes to create the instance and for it to be ready for use.

  2. Within the created instance, create a database:

    Console

    1. Within your instance page, go to the Databases tab.
    2. Click Create database.
    3. In the Database name dialog, enter DATABASE_NAME.
    4. Click Create.

    gcloud

    • Create the database within the recently created instance:

      gcloud sql databases create DATABASE_NAME \
          --instance INSTANCE_NAME
      

      Replace DATABASE_NAME with a name for the database inside the instance.

  3. Create a database user

    Console

    1. Within your instance page, go to the Users tab.
    2. Click Add User Account.
    3. In the Add a user account to instance dialog under "Built-in Authentication":
    4. Enter the username DATABASE_USERNAME.
    5. Enter the password DATABASE_PASSWORD
    6. Click Add.

    gcloud

    • Create the user within the recently created instance:

      gcloud sql users create DATABASE_USERNAME \
          --instance INSTANCE_NAME \
          --password DATABASE_PASSWORD
      

      Replace PASSWORD with a secure password.

Store secret values in Secret Manager

Now that the backing services are configured, Django needs information about these services. Instead of putting these values directly into the Django source code, this tutorial uses Secret Manager to store this information securely.

App Engine standard interacts with secrets by using its service account.

Create Django environment file as a Secret Manager secret

You store the settings required to start Django in a secured env file. The sample app uses the Secret Manager API to retrieve the secret value, and the django-environ package to load the values into the Django environment. The secret is configured to be accessible by the App Engine standard.

  1. Create a file called .env, defining the database connection string, the media bucket name, and a new SECRET_KEY value:

    echo DATABASE_URL=postgres://DATABASE_USERNAME:DATABASE_PASSWORD@//cloudsql/PROJECT_ID:REGION:INSTANCE_NAME/DATABASE_NAME > .env
    echo GS_BUCKET_NAME=PROJECT_ID_MEDIA_BUCKET >> .env
    echo SECRET_KEY=$(cat /dev/urandom | LC_ALL=C tr -dc '[:alpha:]'| fold -w 50 | head -n1) >> .env
    
  2. Store the secret in Secret Manager:

    Console

    1. In the Cloud Console, go to the Secret Manager page.

      Go to the Secret Manager page

    2. Click Create secret

    3. In the Name field, enter django_settings.

    4. In the Secret value dialog, paste the contents of your .env file.

    5. Click Create secret.

    6. Delete the local file to prevent local setting overrides.

    gcloud

    1. Create a new secret, django_settings, with the value of the .env file:

      gcloud secrets create django_settings --data-file .env
      
    2. To confirm the creation of the secret, check it:

      gcloud secrets describe django_settings
      
      gcloud secrets versions access latest --secret django_settings
      
    3. Delete the local file to prevent local setting overrides:

      rm .env
      
  3. Configure access to the secret:

    Console

    1. Click on the Permissions tab.
    2. Click Add.
    3. In the New Members field, enter PROJECT_ID@appspot.gserviceaccount.com, and then press Enter.
    4. In the Role drop-down menu, select Secret Manager Secret Accessor.
    5. Click Save.

    gcloud

    1. Grant access to the secret to the App Engine standard service account:

      gcloud secrets add-iam-policy-binding django_settings \
          --member serviceAccount:PROJECT_ID@appspot.gserviceaccount.com \
          --role roles/secretmanager.secretAccessor
      

Run the app on your local computer

With the backing services configured, you can now run the app on your computer. This setup allows for local development, creating a superuser, and applying database migrations.

  1. In a separate terminal, start the Cloud SQL Auth proxy:

    Linux/macOS

    ./cloud_sql_proxy -instances="PROJECT_ID:REGION:INSTANCE_NAME"=tcp:5432
    

    Windows

    cloud_sql_proxy.exe -instances="PROJECT_ID:REGION:INSTANCE_NAME"=tcp:5432
    

    This step establishes a connection from your local computer to your Cloud SQL instance for local testing purposes. Keep the Cloud SQL Auth proxy running the entire time you test your app locally. Running this process in a separate terminal allows you to keep working while this process runs. Ensure the next steps are completed in a separate terminal

  2. In a new terminal, set the Project ID locally :

    Linux/macOS

      export GOOGLE_CLOUD_PROJECT=PROJECT_ID
    

    Windows

      set GOOGLE_CLOUD_PROJECT=PROJECT_ID
    
  3. Set an environment variable to indicate you are using Cloud SQL Auth proxy (this value is recognised in the code):

    Linux/macOS

      export USE_CLOUD_SQL_AUTH_PROXY=true
    

    Windows

      set USE_CLOUD_SQL_AUTH_PROXY=true
    
  4. Run the Django migrations to set up your models and assets:

    python manage.py makemigrations
    python manage.py makemigrations polls
    python manage.py migrate
    python manage.py collectstatic
    
  5. Start the Django web server:

    python manage.py runserver
    
  6. In your browser, go to http://localhost:8000.

    The page displays the following text: "Hello, world. You're at the polls index." The Django web server running on your computer delivers the sample app pages.

  7. Press Control+C to stop the local web server.

Use the Django admin console

In order to log into Django's admin console, you need to create a superuser. Since you have a locally accessible connection to the database, you can run management commands:

  1. Create a superuser. You will be prompted to enter a username, email, and password.

    python manage.py createsuperuser
    
  2. Start a local web server:

    python manage.py runserver
    
  3. In your browser, go to http://localhost:8000/admin.

  4. Log in to the admin site using the username and password you used when you ran createsuperuser.

Deploy the app to the App Engine standard environment

With all the backing services set up and the application tested locally, you can now deploy the app to App Engine standard:

  1. Upload the app by running the following command, which deploys the app as described in app.yaml and sets the newly deployed version as the default version, causing it to serve all new traffic:
    gcloud app deploy
  2. Confirm the settings by typing "yes" when prompted.
  3. Wait for the message that notifies you that the update has completed.

Running the deployed app

The app has been deployed, and now can be accessed:

  • Open the deployed website:

    gcloud app browse
    
  • Alternatively, display the URL and open manually:

    gcloud app describe --format "value(defaultHostname)"
    

Your request is served by a web server running in the App Engine standard environment.

Updating the application

To update your application, make changes to the code, then run the gcloud app deploy command again.

The deployment creates a new version of your app and promotes it to the default version. The earlier versions of your app remain. All of these app versions are billable resources. To reduce costs, delete the non-default versions of your app.

Configuring for production

You now have a working Django deployment, but there are further steps you can take to ensure that your application is production-ready.

Ensure debugging is disabled

Confirm that the DEBUG variable in mysite/settings.py is set to False. This will prevent detailed error pages from being displayed to the user, which can leak information about the configurations.

Limit the database user privileges

Any users that are created by using Cloud SQL have the privileges associated with the cloudsqlsuperuser role: CREATEROLE, CREATEDB, and LOGIN.

To prevent the Django database user from having these permissions, manually create the user in PostgreSQL. You will need to have the psql interactive terminal installed, or use Cloud Shell which has this tool pre-installed.

Console

  1. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

  2. In Cloud Shell, use the built-in terminal to connect to your INSTANCE_NAME instance:

    gcloud sql connect INSTANCE_NAME --user postgres
    
  3. Enter the postgres user password.

    You are now using psql. You should see the postgres=> prompt.

  4. Create a user:

    CREATE USER DATABASE_USERNAME WITH PASSWORD 'DATABASE_PASSWORD';
    

    Replace PASSWORD with a random, unique password.

  5. Grant full rights on the new database to the new user:

    GRANT ALL PRIVILEGES ON DATABASE DATABASE_NAME TO DATABASE_USERNAME;
    
  6. Exit psql:

    \q
    

gcloud

  1. Start a connection to the SQL instance:

    gcloud sql connect INSTANCE_NAME --user postgres
    

    Replace INSTANCE_NAME with the created Cloud SQL instance.

  2. Enter the postgres user password.

    You are now using psql. You should see the postgres=> prompt.

  3. Create a user:

    CREATE USER DATABASE_USERNAME WITH PASSWORD 'DATABASE_PASSWORD';
    
  4. Grant full rights on the new database to the new user:

    GRANT ALL PRIVILEGES ON DATABASE DATABASE_NAME TO DATABASE_USERNAME;
    
  5. Exit psql:

    \q
    

Understand the code

Sample application

The Django sample app was created using standard Django tooling. These following commands create the project and the polls app:

django-admin startproject mysite
python manage.py startapp polls

The base views, models, and route configurations were copied from Writing your first Django app (Part 1 and Part 2).

Secrets from Secret Manager

The settings.py file contains code that uses the Secret Manager Python API to retrieve the latest version of the named secret, and pull it into the environment (using django-environ):

env = environ.Env(DEBUG=(bool, False))
env_file = os.path.join(BASE_DIR, ".env")

if os.path.isfile(env_file):
    # Use a local secret file, if provided

    env.read_env(env_file)
# ...
elif os.environ.get("GOOGLE_CLOUD_PROJECT", None):
    # Pull secrets from Secret Manager
    project_id = os.environ.get("GOOGLE_CLOUD_PROJECT")

    client = secretmanager.SecretManagerServiceClient()
    settings_name = os.environ.get("SETTINGS_NAME", "django_settings")
    name = f"projects/{project_id}/secrets/{settings_name}/versions/latest"
    payload = client.access_secret_version(name=name).payload.data.decode("UTF-8")

    env.read_env(io.StringIO(payload))
else:
    raise Exception("No local .env or GOOGLE_CLOUD_PROJECT detected. No secrets found.")

The secret is used to store multiple secret values to reduce the number of different secrets that needed to be configured.

Local secret overrides

If a .env file is found on the local filesystem, it is used instead of the value from Secret Manager. Creating a .env file locally can help with local testing (e.g. local development against a SQLite database, or other local settings).

Database connection

The settings.py file contains the configuration for your SQL database. If you set the USE_CLOUD_SQL_AUTH_PROXY, the DATABASES setting is changed to infer use of the Cloud SQL Auth proxy.

# Use django-environ to parse the connection string
DATABASES = {"default": env.db()}

# If the flag as been set, configure to use proxy
if os.getenv("USE_CLOUD_SQL_AUTH_PROXY", None):
    DATABASES["default"]["HOST"] = "127.0.0.1"
    DATABASES["default"]["PORT"] = 5432

Hosted static content

The app.yaml file contains configuration information for deployment to App Engine. This app.yaml file specifies that App Engine serves static files from the static/ directory:

runtime: python39

handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /static
  static_dir: static/

# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
  script: auto

When running the app locally with DEBUG enabled, these files are served locally by Django:

from django.conf import settings
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("", include("polls.urls")),
    path("admin/", admin.site.urls),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next