Receive a Cloud Audit Logs event

Standard

---

This tutorial shows you how to deploy an authenticated Cloud Run service that receives events from Cloud Storage using Cloud Audit Logs. Use this tutorial to deploy production workloads. The Eventarc trigger filters events based on Cloud Audit Logs entries. For more information, see Determine event filters for Cloud Audit Logs.

Note: All Cloud Run services are deployed privately by default, which means that they can't be accessed without providing authentication credentials in the request. For projects with domain restricted sharing organization policies, use an authenticated Cloud Run target service rather than one that allows public (unauthenticated) access.

You can complete this tutorial using either the Google Cloud console or the Google Cloud CLI.

Objectives

In this tutorial, you will:

1. Create a Cloud Storage bucket to be the event source.

2. Deploy an event receiver service to Cloud Run.

3. Create an Eventarc trigger.

4. Generate an event by uploading a file to the Cloud Storage bucket, and view it in the Cloud Run logs.

Costs

In this document, you use the following billable components of Google Cloud:

• Artifact Registry
• Cloud Build
• Cloud Logging
• Cloud Run
• Cloud Storage
• Eventarc
• Pub/Sub

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

Security constraints defined by your organization might prevent you from completing the following steps. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

If you are the project creator, you are granted the basic Owner role (roles/owner). By default, this Identity and Access Management (IAM) role includes the permissions necessary for full access to most Google Cloud resources and you can skip this step.

If you are not the project creator, required permissions must be granted on the project to the appropriate principal. For example, a principal can be a Google Account (for end users) or a service account (for applications and compute workloads). For more information, see the Roles and permissions page for your event destination.

Note that by default, Cloud Build permissions include permissions to upload and download Artifact Registry artifacts.

Required permissions

To get the permissions that you need to complete this tutorial, ask your administrator to grant you the following IAM roles on your project:

• Cloud Build Editor (roles/cloudbuild.builds.editor)
• Cloud Run Admin (roles/run.admin)
• Eventarc Admin (roles/eventarc.admin)
• Logs View Accessor (roles/logging.viewAccessor)
• Project IAM Admin (roles/resourcemanager.projectIamAdmin)
• Service Account Admin (roles/iam.serviceAccountAdmin)
• Service Account User (roles/iam.serviceAccountUser)
• Service Usage Admin (roles/serviceusage.serviceUsageAdmin)
• Storage Admin (roles/storage.admin)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Complete the following steps using the Google Cloud console or the gcloud CLI:

Console

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Artifact Registry, Cloud Build, Logging, Pub/Sub, Cloud Run, Cloud Storage, and Eventarc APIs.

   Enable the APIs

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the Artifact Registry, Cloud Build, Logging, Pub/Sub, Cloud Run, Cloud Storage, and Eventarc APIs.

   Enable the APIs

8. To configure what data access is recorded in your audit logs, enable the Admin Read, Data Read, and Data Write log types for Google Cloud Storage:

   Go to Audit Logs

9. In the Google Cloud console, go to the Service accounts page.

   Go to Service Accounts

10. Click Create service account.

11. Enter a service account name to display in the Google Cloud console.

    The Google Cloud console generates a service account ID based on this name. Edit the ID if necessary. You cannot change the ID later.

12. Optional: Enter a description of the service account.

13. If you don't want to set access controls now, click Done to finish creating the service account.

    To set access controls now, click Create and continue and continue to the next step.

14. Select the Cloud Run Invoker and Eventarc Event Receiver roles to grant on the project to the service account associated with your Eventarc trigger.

    For testing purposes, this grants the Cloud Run Invoker role on all Cloud Run services and jobs in the project; however, you can grant the role on the service. For more information, see Grant Cloud Run service permissions.

    Note that if you create a trigger for an authenticated Cloud Run service without granting the Cloud Run Invoker role, the trigger is created successfully and is active. However, the trigger won't work as expected and a message similar to the following appears in the logs:

    The request was not authenticated. Either allow unauthenticated invocations or set the proper Authorization header.
15. When you are done adding roles, click Continue and Done to finish creating the service account.

gcloud

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

5. Make sure that billing is enabled for your Google Cloud project.

6. Enable the Artifact Registry, Cloud Build, Logging, Pub/Sub, Cloud Run, Cloud Storage, and Eventarc APIs:

   gcloud services enable artifactregistry.googleapis.com cloudbuild.googleapis.com 
         logging.googleapis.com  pubsub.googleapis.com  run.googleapis.com 
         storage.googleapis.com  and eventarc.googleapis.com

7. Install the Google Cloud CLI.

8. To initialize the gcloud CLI, run the following command:

   gcloud init

9. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

10. Make sure that billing is enabled for your Google Cloud project.

11. Enable the Artifact Registry, Cloud Build, Logging, Pub/Sub, Cloud Run, Cloud Storage, and Eventarc APIs:

    gcloud services enable artifactregistry.googleapis.com cloudbuild.googleapis.com 
          logging.googleapis.com  pubsub.googleapis.com  run.googleapis.com 
          storage.googleapis.com  and eventarc.googleapis.com

12. Update gcloud components:

    gcloud components update

13. Sign in using your account:

    gcloud auth login
    

14. Set the configuration variables used in this tutorial:

    export REGION=us-central1
    gcloud config set run/region ${REGION}
    gcloud config set run/platform managed
    gcloud config set eventarc/location ${REGION}
    export SERVICE_NAME=helloworld-events
    

15. To configure what data access is recorded in your audit logs, enable the ADMIN_READ, DATA_READ, and DATA_WRITE log types for the storage.googleapis.com service:
    1. Read your project's IAM policy and store it in a file:

       gcloud projects get-iam-policy PROJECT_ID > /tmp/policy.yaml

    2. Edit your policy in /tmp/policy.yaml, adding or changing only the Data Access audit logs configuration.
       Caution: You must preserve the bindings: and etag: sections without changes. Failure to do so might cause your project to become unusable.

       auditConfigs:
       - auditLogConfigs:
       - logType: ADMIN_READ
       - logType: DATA_WRITE
       - logType: DATA_READ
       service: storage.googleapis.com

    3. Write your new IAM policy:

       gcloud projects set-iam-policy PROJECT_ID /tmp/policy.yaml

       If the preceding command reports a conflict with another change, then repeat these steps, starting with reading the project's IAM policy.
16. Create a service account for the project:

    gcloud iam service-accounts create sample-service-account \
        --description="A sample service account" \
        --display-name="Sample service account"

    After you create a service account, it can take up to seven minutes before you can use the service account. If you try to use a service account immediately after you create it, and you receive an error, wait at least 60 seconds and try again.
17. To confirm that sample-service-account has been created, run:

    gcloud iam service-accounts list

    The output should be similar to the following:

    DISPLAY NAME                     EMAIL                                                               DISABLED
    Default compute service account  [e-mail]                False
    Sample service account           sample-service-account@PROJECT_ID.iam.gserviceaccount.com           False

18. Grant the Cloud Run Invoker (run.invoker) and the Eventarc Event Receiver (roles/eventarc.eventReceiver) roles on the project to the service account associated with your Eventarc trigger:

    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member="serviceAccount:sample-service-account@PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/run.invoker"

    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member="serviceAccount:sample-service-account@PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/eventarc.eventReceiver"

    Replace PROJECT_ID with your Google Cloud project ID.

    For testing purposes, this grants the Cloud Run Invoker role on all Cloud Run services and jobs in the project; however, you can grant the role on the service. For more information, see Grant Cloud Run service permissions.

    Note that if you create a trigger for an authenticated Cloud Run service without granting the Cloud Run Invoker role, the trigger is created successfully and is active. However, the trigger won't work as expected and a message similar to the following appears in the logs:

    The request was not authenticated. Either allow unauthenticated invocations or set the proper Authorization header.
Note: If you enabled the Pub/Sub service account on or before April 8, 2021, grant the iam.serviceAccountTokenCreator role to the Pub/Sub service account.

Create an Artifact Registry standard repository

Create an Artifact Registry standard repository to store your container image.

Console

1. In the Google Cloud console, go to the Repositories page.
   

   Go to Repositories

2. Click Create repository.
3. Configure your repository:
   1. Enter a unique Name.
   2. For Format, choose Docker.
   3. For Mode, choose Standard.
   4. For Location type, choose Region.
   5. In the Region list, select us-central1 (Iowa).
4. Accept the other defaults and click Create.

gcloud

gcloud artifacts repositories create REPOSITORY \
    --repository-format=docker \
    --location=$REGION

Replace REPOSITORY with a unique name for the Artifact Registry repository.

Create a Cloud Storage bucket

This tutorial uses Cloud Storage as the event source. To create a storage bucket:

Console

1. In the Google Cloud console, go to the Buckets page.
   

   Go to Buckets

2. Click Create bucket.
3. Enter your bucket information and click Continue to complete each step:
   1. Enter a unique Name. For example, eventarcbucket.
   2. Select Region as the Location Type.
   3. Select us-central1 (Iowa) as the Location.
   4. Select Standard for default storage class.
   5. Select Uniform for Access control.
4. Click Create.

gcloud

gsutil mb -l $REGION gs://events-tutorial-PROJECT_ID/

After the event source is created, you can deploy the event receiver service on Cloud Run.

Deploy the event receiver service to Cloud Run

Deploy a Cloud Run service that receives and logs events. To deploy the sample event receiver service:

Console

1. Clone the sample repository to your GitHub account:

Go

1. On GitHub, navigate to GoogleCloudPlatform/golang-samples
2. Click Fork.

3. If prompted, select the location where you want to fork the repository.

Java

1. On GitHub, navigate to GoogleCloudPlatform/java-docs-samples.
2. Click Fork.

3. If prompted, select the location where you want to fork the repository.

.NET

1. On GitHub, navigate to GoogleCloudPlatform/dotnet-docs-samples.
2. Click Fork.

3. If prompted, select the location where you want to fork the repository.

Node.js

1. On GitHub, navigate to GoogleCloudPlatform/nodejs-docs-samples.
2. Click Fork.

3. If prompted, select the location where you want to fork the repository.

Python

1. On GitHub, navigate to GoogleCloudPlatform/python-docs-samples.
2. Click Fork.

3. If prompted, select the location where you want to fork the repository.
2. In the Google Cloud console, go to the Services page.

Go to Services

3. Click Create service to display the Create service form.
4. Select Continuously deploy from a repository.

   Changes to your GitHub repository are automatically built into container images in Artifact Registry and deployed to Cloud Run.

5. Click Set up with Cloud Build to open the Set up with Cloud Build form.
1. If prompted, enable the Cloud Build API and Artifact Analysis API.
2. Select GitHub as the Repository Provider.
   Note:If you are not yet authenticated, click Authenticate and follow the instructions to connect to your repository using the Cloud Build GitHub app.
3. If prompted, click Install Google Cloud Build.
4. Select the GitHub repository you forked as the Repository.
5. Click Next.
6. In the Branch field, enter ^main$.
7. Select Dockerfile as the Build Type and provide the source location of the Dockerfile:
   • eventarc/audit-storage/Dockerfile

     or

   • eventarc/audit_storage/Dockerfile(Go)
8. Click Save.
6. In the Create service form, enter a service name. For example, helloworld-events.
7. Select us-central1(Iowa) as the Region where you want your service located.
8. Select any of the Ingress options based on the ingress traffic you want to allow on the Cloud Run service.
9. Select Require authentication
10. Click Create.

gcloud

1. Clone the GitHub repository:

   Go

   git clone https://github.com/GoogleCloudPlatform/golang-samples.git
   cd golang-samples/eventarc/audit_storage
   

   Java

   git clone https://github.com/GoogleCloudPlatform/java-docs-samples.git
   cd java-docs-samples/eventarc/audit-storage

   .NET

   git clone https://github.com/GoogleCloudPlatform/dotnet-docs-samples.git
   cd dotnet-docs-samples/eventarc/audit-storage

   Node.js

   git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git
   cd nodejs-docs-samples/eventarc/audit-storage

   Python

   git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
   cd python-docs-samples/eventarc/audit-storage

2. Build the container and upload it to Cloud Build:

   gcloud builds submit --tag $REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/$SERVICE_NAME:v1
   

3. Deploy the container image to Cloud Run:

   gcloud run deploy $SERVICE_NAME \
       --image $REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/$SERVICE_NAME:v1
   

   Note:Based on the required level of network access to the Cloud Run service, you can also set the ingress flag.

   At the prompt to allow unauthenticated invocations, enter n.

   When the deployment succeeds, the command line displays the service URL.

Now that you have deployed your event receiver service called helloworld-events to Cloud Run, you can set up your trigger.

Create an Eventarc trigger

The Eventarc trigger will send events from the Cloud Storage bucket to the helloworld-events Cloud Run service.

Console

1. In the Google Cloud console, go to the Triggers page.

   Go to Triggers

2. Click Create trigger.
3. Type a Trigger name.

   This is the ID of the trigger and it must start with a letter. It can contain up to 63 lowercase letters, numbers, or hyphens.

4. Select a Trigger type:
   • First-party: Filters events sent from Google Cloud providers (directly or through Cloud Audit Logs entries) or providers using Pub/Sub messages.
   • Third-party: Filters events sent from third-party providers.
5. In the Event provider list, select Cloud Storage as the source of events.
6. In the Event type list, select storage.objects.create.
7. In the Region list, select us-central1 as the region to receive events from.
8. Select the Service account you created. For example, SERVICE_ACCOUNT_ID@PROJECT_ID.iam.gserviceaccount.com.
9. In the Event destination list, select Cloud Run.
10. In the Select a Cloud Run service list, select helloworld-events.
11. Click Create.
Note that you can also create an Eventarc trigger using the Cloud Run console page.

gcloud

1. Create a trigger that filters Cloud Storage events and that uses the service account you created:

   gcloud eventarc triggers create events-tutorial-trigger \
       --destination-run-service=$SERVICE_NAME \
       --destination-run-region=$REGION \
       --event-filters="type=google.cloud.audit.log.v1.written" \
       --event-filters="serviceName=storage.googleapis.com" \
       --event-filters="methodName=storage.objects.create" \
       --service-account=sample-service-account@PROJECT_ID.iam.gserviceaccount.com

   Replace PROJECT_ID with the Google Cloud project ID.

   Where:

   • type: specifies that audit logs are created when the trigger's filter criteria is met.
   • serviceName: the service that writes the audit log, which is Cloud Storage.
   • methodName: the operation that is being audited, which is storage.objects.create.
   This creates a trigger called events-tutorial-trigger.
2. To confirm events-tutorial-trigger was successfully created, run:

   gcloud eventarc triggers list --location=$REGION

The trigger, events-tutorial-trigger, is listed with a target of helloworld-events.

Note: Although your trigger is created immediately, it can take up to two minutes for triggers to be fully functional.

Generate and view an event

1. To generate an event:

   Console

   1. Create a text file with the filename random.txt and the text "Hello World".
   2. In the Google Cloud console, go to the Buckets page.

      Go to Buckets

   3. Select the storage bucket you created.
   4. In the Objects tab, click Upload files and upload the random.txt file.

   gcloud

   Upload a text file to Cloud Storage:

   echo "Hello World" > random.txt
   gsutil cp random.txt gs://events-tutorial-PROJECT_ID/random.txt

2. To view the log entry:

Console

1. In the Google Cloud console, go to the Services page.

   Go to Services

2. From the list of services, click the name of the service you created to go to its Service details page.
3. Click the Logs tab, to get the request and container logs for all revisions of this service. You can filter by log severity level.
4. Look for a log entry similar to:

   Detected change in Cloud Storage bucket: storage.googleapis.com/projects/_/buckets/BUCKET_NAME/objects/random.txt

   where BUCKET_NAME is the name of the Cloud Storage bucket.

gcloud

1. gcloud logging read "resource.labels.service_name=helloworld-events AND textPayload:random.txt" --format=json

2. Look for a log entry similar to:

   Detected change in Cloud Storage bucket: storage.googleapis.com/projects/_/buckets/BUCKET_NAME/objects/random.txt

   where BUCKET_NAME is the name of the Cloud Storage bucket.
   Note:If you don't see a log entry, wait a few minutes and retry the gcloud logging read command. The trigger can take up to two minutes after creation to start filtering events.

You have successfully deployed an event receiver service to Cloud Run, created an Eventarc trigger, generated an event from Cloud Storage, and viewed it in the Cloud Run logs.

Clean up

While Cloud Run does not charge when the service is not in use, you might still be charged for storing the container image in Artifact Registry, storing files in your Cloud Storage bucket, and Eventarc resources.

You can:

1. Delete your container image.

2. Delete your storage bucket.

3. Delete the Eventarc trigger.

Alternatively, you can delete your Google Cloud project to avoid incurring charges. Deleting your Google Cloud project stops billing for all the resources used within that project.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

What's next

• Build containers
• Create a trigger to route Cloud Audit Logs events to Cloud Run
• Develop Cloud Run services
• Receive Pub/Sub events using an authenticated Cloud Run service
• Tutorial: Debug a Cloud Run service using Eventarc