Securing Cloud Run Deployments with Least Privilege Access

With Cloud Run, developers can quickly deploy production web applications and APIs on a serverless environment that runs on top of Google’s scalable infrastructure. While development teams can leverage Cloud Run to improve development agility and iterate quickly, many overlook their infrastructure’s security posture. In particular, one aspect of security that does not get enough attention is access management and the principle of least privilege.

The principle of least privilege states that a resource should only have access to the exact resources it needs in order to function. This principle was developed to address the risk of compromised identities granting an attacker access to a wide range of resources. Fortunately, Google Cloud offers fine-grained access controls and tools that can help you apply it. In this post, we’ll explore the ways you can improve your Cloud Run security posture by applying the principle of least privilege to both inbound (users and services accessing Cloud Run) and outbound (the Cloud Run service itself accessing other services) scenarios. 

How to configure Cloud Run for least privilege access?   

Let’s consider two broad Cloud Run use cases: user-facing (frontend) applications and internal (backend) applications. Both use cases are captured in a public web app where a backend Cloud Run service is invoked either directly by the frontend Cloud Run service, or in response to an event. An example of such a scenario is a simple PDF conversion web app as shown in the following diagram:

In this hypothetical website, users can upload documents they wish to convert to PDF. The website implements signup and login functionalities so that authenticated users can access their conversion history and make sure only they have access to their documents. A Cloud SQL database is used to persist user data, and documents are stored in a Cloud Storage bucket. 

As shown in the preceding diagram, each time a new document is uploaded to Cloud Storage, a message is published to Cloud Pub/Sub, and a push subscription is used to trigger the Backend Worker service to perform the conversion. Once the file is converted, the service will upload the new file to a separate location in the Cloud Storage bucket. In this simple example, the backend service is not updating the Cloud SQL database. (Please note that in a more realistic scenario, the backend service would update the Cloud SQL database.) 

So, we have two Cloud Run services: a user-facing, public frontend app, and an internal backend app triggered by the Cloud Pub/Sub service.

As a public website, the frontend Cloud Run service cannot restrict access to authenticated users only. Any internal app functionality that requires authentication and authorization must be handled using appropriate authentication mechanisms and code libraries. The details of how to approach end user authentication are outside the scope of this article, but if you’re interested in that, there’s a handy tutorial here. What we’re going to discuss is how to enforce least privilege access at three downstream areas of the traffic flow, as highlighted in the preceding diagram:

1. Access from the App Frontend service to the backing services.

2. Access from internal services (Cloud Pub/Sub) to the Backend Worker service.

3. Access from the Backend Worker service to backing services (similar to #1).

What you need to know to address these three areas – and to implement the least privilege principle with Cloud Run apps in general – boils down to two things: limiting access for internal users and services invoking Cloud Run services, and limiting access for Cloud Run services accessing other Google Cloud services.

For internal users and services invoking Cloud Run services

Disallow unauthenticated access

When you have an internal Cloud Run service, whether it’s a web or mobile backend, a private API, or some lightweight automation job, an important action you need to take to align with the least privilege principle is to disallow unauthenticated access. Using gcloud, you can do this by specifying the no-allow-unauthenticated option when deploying a service:

To achieve the same using the console, you must check the “Require authentication” box when creating a service:

Once you’ve applied this setting, your service will now reject unauthenticated requests with an HTTP 403 Forbidden error.

Now, how do we ensure that a legitimate request gets through? In our example, we want only our Cloud Pub/Sub push subscription to invoke the Cloud Run service. No other service or user should be able to do it. Let’s see how we can do that.

Create a custom service account

If you create a Cloud Pub/Sub push subscription without specifying a service account, the requests will be unauthenticated. If the Cloud Run service accepts unauthenticated requests, this would work. However, we’re talking about an internal service we want to protect by applying the principle of least privilege. We already configured Cloud Run to only accept authenticated requests. Therefore, we need our Cloud Pub/Sub push subscription to authenticate all requests to Cloud Run. The first step is to create a custom service account. 

To create a service account using gcloud, you can run:

The service account name must be unique within the project and, ideally, descriptive of how it’s used. For example, we could name our service account pubsub-cloud-run-invoker-sa. You can optionally define a more friendly display name with the display-name option. For example:

Configure IAM policy bindings with minimal permissions

A newly created service account will have no permissions associated with it, so it won't be able to do anything. 

To grant the service account permission to invoke our Cloud Run service, we can assign it a predefined role or create a custom one with the permission set we need. Primitive roles such as Owner, Editor, and Viewer should be avoided, as these will likely include many more permissions than necessary. Predefined roles are service-specific, Google Cloud-curated roles that can apply to an entire project or a specific resource. You should prefer a predefined role if you find one that has the minimal required set of permissions and without any unnecessary ones. Let’s take a look at what options are available for Cloud Run:

From the available options, we can see that the Cloud Run Invoker role has the exact permission we need: invoke services. No more. No less.

To assign this role to our service account, we need to add an IAM policy binding in our Cloud Run service. To do this using gcloud, we run:

The SERVICE_ACCOUNT_ID is the email address used to identify the service account. It uses the following format:

[SERVICE_ACCOUNT_NAME]@[PROJECT_ID].iam.gserviceaccount.com

So, for our service account named pubsub-cloud-run-invoker, and if we suppose our project ID is my-project, the email address will be:

pubsub-cloud-run-invoker@my-project.iam.gserviceaccount.com

Once we’ve added the IAM policy binding, now the service account will have the permission to invoke our backend Cloud Run service. 

Enable the calling service to invoke Cloud Run

An authenticated Cloud Run request must present proof of the calling service’s identity by adding a Google Cloud-signed OpenID Connect token as part of the request. For that, we need the service (in our example, Cloud Pub/Sub) to have permissions to generate these tokens. 

For Google Cloud projects created on or before April 8, 2021, you must grant the Service Account Token Creator role to the Google-managed service account service-[PROJECT_NUMBER]@gcp-sa-pubsub.iam.gserviceaccount.com. This role lets principals create OAuth 2.0 access tokens, create OpenID Connect (OIDC) ID tokens, and sign JSON Web Tokens (JWTs). To add a policy binding that grants the service account this role, you can run the following using gcloud:

For projects created after April 8, 2021, you don’t need to grant this role because the Google Cloud-managed Pub/Sub service account has the role Service Agent with identical permissions.

The final step is to “tell” our Pub/Sub push subscription to use our custom, user-managed service account. To do that, we include the --push-auth-service-account property in the gcloud command that creates a push subscription:

With that, as long as we don’t grant any other user or service permission to invoke this Cloud Run service, we’ll ensure that all processed requests came from Pub/Sub. In addition, we know that the Cloud Pub/Sub subscription can’t perform any action other than invoke the Cloud Run service, because that’s the only action permitted in the Cloud Run Invoker role. This is the principle of least privilege in action.

For Cloud Run services accessing other services

Unless you specify a custom service account, Cloud Run will by default use the default Compute Engine service account. This default service account has the Editor role in the project, which grants read and write permissions on all resources in your Google Cloud project. This allows for a seamless and convenient development experience, especially for those starting out with Google Cloud. But it goes against the principle of least principle and should therefore be avoided in production environments. 

Just like we did for Cloud Pub/Sub, we can create a custom service account for Cloud Run and grant it the minimal set of permissions it needs. And what does it need? In our example, the frontend app needs read and write access to a Cloud SQL database. It also needs access to get, create, delete, and manage objects in a Cloud Storage bucket. The backend app only needs to upload objects to the Cloud Storage bucket.

Create Cloud Run service identities and grant minimal permissions    

In Cloud Run, each service revision is linked to a service account. We refer to this service account as the Cloud Run service identity. We recommend giving each Cloud Run service its own dedicated identity, as opposed to reusing identities across services (even if different services require the same permissions). 

So, we need to create two custom service accounts. Let’s call them frontend-app-sa and backend-app-sa. In gcloud, you can run:

gcloud iam service-accounts create frontend-app-sa

gcloud iam service-accounts create backend-app-sa

Now, we can take a look at the predefined roles available for Cloud SQL and Cloud Storage services. Luckily, we have predefined roles we can use that include only the exact set of permissions we need:

In Google Cloud, an identity can have multiple policy bindings, which means we can assign multiple roles to a service account. In this case, we can assign our frontend app service account the roles Cloud SQL Client and Storage Object Admin. For our backend app service account, we can use the Storage Object Creator role. 

Using gcloud, we can run:

Now we have the user-managed service accounts we need with minimal permissions. 

Optional: Add an IAM policy condition

If you want to take the principle of least privilege to the next level, you can also make sure that access to resources is only granted if certain conditions are met. For example, let’s say our application is an internal corporate application only used on weekdays. If no one (at least no legitimate user) is using the service on weekends, then why not strengthen our security posture further by not allowing the frontend service to access the database at all outside of weekdays?

To do this, let’s use the Google Cloud console which offers an easier experience for creating IAM conditions.

In the IAM & Admin console, navigate to IAM to see the list of principals. Click on the pencil icon next to the app-frontend-sa service account created earlier to edit it. The edit page should look like this:

Then, click on Add IAM Condition. In the Condition Builder tab, you can use the graphical user interface to construct your access condition. In our example, we want access to be granted on weekdays only. It should look like this:  

Once we’re done we can save our changes. Now, the frontend app will no longer be able to access the database on weekends. Ideally, you would update your frontend code to handle that failure gracefully, by for example displaying a friendly message to the weekend worker stating that the service is not available on weekends.

Beyond date and time, other IAM conditions you can apply are:

• The access level

• The destination IP address and port (for IAP TCP tunneling)

• The expected URL host/path (for IAP)

In many cases, however, there aren’t really IAM conditions you can set without disrupting the user experience in unwanted ways. Hence why this is an optional step, to be considered in only some scenarios. 

Deploy a Cloud Run service with the new service identity

To deploy a Cloud Run service with a user-managed service identity, we can simply run (using gcloud):

It’s also possible to update an existing service with:

Note: To deploy a Cloud Run service using a custom service account, you must have permission to impersonate that service account

In our example, assuming the frontend and backend services already exist, we would run:

And now we’re done securing our Cloud Run apps with least privilege access. Or are we?

Auditing Access and Using IAM Recommender

A good security posture requires not only applying the right settings, but also following the right processes. For identity and access management, that means doing regular audits and monitoring access over time. It’s important to not lose visibility of who is accessing what and when in your project. As projects evolve over time, access patterns may change, and you’ll want to periodically review your IAM policies for permissions that are granted but may no longer be required. 

Where do I find who is accessing what and when?

In Google Cloud, all platform activities are logged under Cloud Audit Logs. This is enabled by default in every Google Cloud project and it can’t be disabled. These logs help answer the question of “Who did what, when?”. To find them, go to the Logging console and navigate to Logs Explorer. In the Query pane, you can click on Log name on the top right corner to filter logs by name. Select activity under Cloud Audit to only see activity logs, as shown in the following image:

Click on Apply and you should see all the activity logs listed in the Logs Explorer page. You can expand each entry to inspect the details of the activity.

Alternatively, you can go to the Activity page in the console where you can view abbreviated audit log entries.

Data Access Audit Logs

Data Access audit logs are a special type of audit logs that contains API calls that read the configuration or metadata of resources, as well as user-driven API calls that create, modify, or read user-provided resource data. These logs are disabled by default, except for BigQuery. This is because they can generate very large volumes of logs. 

If you want Data Access audit logs to be written for Google Cloud services other than BigQuery, you must explicitly enable them by following the instructions here.

Alerts

If you wish to be notified when certain messages appear in your audit logs, you can use Cloud Monitoring to create log-based alerts. For example, you may want to trigger an alert when access to some sensitive data is recorded by Data Access audit logs. Or when certain rare and impactful activities are being performed, such as changes to project security settings. 

To create log-based alerts, follow the instructions here.

Another option you have, in addition to inspecting audit logs and setting up alerts, is to use IAM Recommender. 

What is IAM Recommender and how do I use it?

The IAM Recommender generates role recommendations that help you identify and remove excess permissions from your principals. It is one of the suggestions that the Recommendations service offers. Each role recommendation suggests that you remove or replace a role that gives your principals excess permissions, and you should leverage them to ensure you stay on track with the principle of least privilege.

To view recommendations, go to the Recommendations console. If you have role recommendations (which can take up to 90 days from role creation to appear,) you may see something like this:

This recommendation is pointing out that I have an estimated 12,340 excess permissions. This is because I have two unused service accounts with the Editor role. By clicking on View all, I can inspect the principals the recommender is referring to. For each principal, I can also see the list of permissions that are believed to be unnecessary:

As mentioned previously, the Editor role includes a large number of permissions and should be avoided unless strictly required. But the IAM Recommender doesn’t just look at Editor roles, it will inspect any role assignment where there might be an excess of permissions. 

And by the way, if you forget to create a user-managed service account for your Cloud Run services, it will recommend you do so as well:

Conclusion and Call to Action

Developers choose Cloud Run because it’s a scalable and flexible application platform. However, managing access permissions and applying the principle of least privilege are an essential security practice that is often overlooked. Luckily, Google Cloud offers the tools and capabilities that allow you to apply the principle of least privilege across the platform. These include not only IAM controls, but the IAM Recommender service and Cloud Audit Logs which can also help you maintain your security posture over time. 

But remember, in this article, we only discussed security from the perspective of the principle of least privilege. Ideally, you would also think about isolating your databases from the internet, securing your VPC networks, protecting your sensitive data with VPC Service Controls, and leveraging services such as Cloud KMS and Secret Manager, to name just a few examples. For a more comprehensive secure serverless architecture, check out the secured serverless architecture blueprint.

For hands-on labs where you can apply the principles you’ve learned in this article, check out the Serverless Cloud Run Development quest on Google Cloud Skills Boost. For gaining more experience with identity and access management, check out the labs in the quest Ensure Access & Identity in Google Cloud.