Scenarios for exporting Cloud Logging data: Datadog

This scenario shows how to export selected logs from Cloud Logging to Pub/Sub for ingestion into Datadog. Datadog is a monitoring and analytics platform for dynamic, cloud-scale application infrastructure. This scenario is intended for platform administrators who implement log management and log retention policies. Some previous experience with Cloud Logging and Datadog can help you follow the scenario, although previous experience is not necessary.

The scenario explains how to export Cloud Audit Logs, Google Kubernetes Engine logs, Cloud Storage logs, and BigQuery logs to Datadog. In Datadog, you can correlate these logs with metrics and distributed traces to get a view of your entire Google Cloud infrastructure.

You can access Datadog through two different domains, depending on your location. EU users use the datadoghq.eu domain. Non-EU users use the datadoghq.com domain. We provide both links in this scenario.

This setup uses Pub/Sub push subscriptions, which are incompatible with VPC Service Controls.

This scenario is part of the series Design patterns for exporting Cloud Logging.

Before you begin

1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to the project selector page

2. If you don't already have a Datadog account, create one by using the Datadog free trial.

3. Enable the Datadog integration with Google Cloud by following the installation instructions.

Setting up the logging export

The following diagram shows the components used to export logs to Datadog through Pub/Sub. Various Google Cloud products produce logs, which are sent to Cloud Logging. These logs are filtered, and the remaining logs are sent to Datadog by using a Pub/Sub sink.

Set up a Pub/Sub topic and subscription

1. In the Cloud Console, activate Cloud Shell.

   Activate Cloud Shell

2. Configure Cloud Shell to use the project that you selected. Replace PROJECT_ID with your project ID:

   export PROJECT_ID=PROJECT_ID
   gcloud config set project ${PROJECT_ID}
   

3. Set up a Pub/Sub topic to export logs:

   gcloud pubsub topics create datadog-logs-export-topic
   

4. Copy your Datadog API key from the Datadog API settings page. If you're an EU user, use the EU page. If you're non-EU user, use the default US page.

5. In Cloud Shell, create a subscription to send logs from the Pub/Sub topic to Datadog:

   gcloud pubsub subscriptions create \
      datadog-logs-export-subscription \
      --topic=datadog-logs-export-topic \
      --push-endpoint=https://gcp-intake.logs.datadoghq.TLD/v1/input/API_KEY/
   

   Replace the following:

   • API_KEY: your Datadog API key
   • TLD: eu if you're an EU user, com if you're non-EU user

Turn on audit logging for all services

In Cloud Audit Logs, Admin Activity audit logs are always written and can't be disabled. However, Data Access audit logs are disabled by default (except for BigQuery, where Data Access audit logs are enabled by default and don't count against your logs allotment). In this section, you enable all audit logs for all Cloud services and users at the project level.

1. In Cloud Shell, read your project's Identity and Access Management (IAM) policy and save it as a file:

   gcloud projects get-iam-policy ${PROJECT_ID} | tee /tmp/policy.yaml
   

   The output is similar to the following:

   bindings:
   - members:
     - user:[e-mail]
     role: roles/editor
   - members:
     - user:[e-mail]
     role: roles/owner
   etag: :BwVM-FDzeYM=
   version: 1
   

2. Add an auditConfigs section to the IAM policy file that you downloaded in order to enable Data Access logs for all services and users:

   cat << EOF >> /tmp/policy.yaml
   auditConfigs:
   - auditLogConfigs:
     - logType: ADMIN_READ
     - logType: DATA_WRITE
     - logType: DATA_READ
     service: allServices
   EOF
   

   For more configuration options, see the AuditConfig documentation.

3. In Cloud Shell, apply the new IAM policy to your Cloud project:

   gcloud projects set-iam-policy ${PROJECT_ID} /tmp/policy.yaml
   

Configure the logging export

In this section, you create a sink that exports the Cloud Audit Logs, Google Kubernetes Engine (GKE) logs, Cloud Storage logs, and BigQuery logs to the Pub/Sub topic that you previously created.

1. In Cloud Shell, create a log sink with a custom logging filter that matches the Cloud Audit Logs, GKE logs, Cloud Storage logs, and BigQuery logs:

   gcloud logging sinks create export-logs-datadog \
       pubsub.googleapis.com/projects/${PROJECT_ID}/topics/datadog-logs-export-topic \
       --log-filter='logName:"/logs/cloudaudit.googleapis.com" OR
   resource.type:gke_cluster OR
   resource.type=gcs_bucket OR
   resource.type=bigquery_resource'
   

   The output is similar to the following:

   Created [https://logging.googleapis.com/v2/projects/PROJECT_ID/sinks/export-logs-datadog].
   Please remember to grant `serviceAccount:LOGS_SINK_SERVICE_ACCOUNT` \
       the Pub/Sub Publisher role on the topic.
   More information about sinks can be found at /logging/docs/export/configure_export
   

   Make a note of the LOGS_SINK_SERVICE_ACCOUNT service account name from the preceding command output.

2. Grant the LOGS_SINK_SERVICE_ACCOUNT service account permission to publish to the datadog-logs-export-topic topic:

   gcloud pubsub topics add-iam-policy-binding datadog-logs-export-topic \
       --member serviceAccount:LOGS_SINK_SERVICE_ACCOUNT \
       --role roles/pubsub.publisher
   

Now that the Google Cloud service account has the necessary permissions, the logs are shipped to Datadog. You can view these logs in the Datadog Log Explorer. If you're an EU user, view the logs at the EU link.

Stay within quotas and limits

Pub/Sub is subject to quotas and resource limits. If your log volume is at risk of exceeding the limit for a single subscription, Datadog recommends splitting logs across multiple topics by using different filters. In Datadog, you can set up a threshold alert so that you are automatically notified when the number of outstanding messages in a subscription (gcp.pubsub.subscription.num_outstanding_messages) exceeds 1000, so that you can make the necessary changes to your configuration. For more information, see Monitoring push subscriptions.

Using the exported logs

Datadog automatically parses useful attributes from your logs, such as cluster name and project ID. You can use these attributes to drill down to subsets of logs, identify common patterns in your logs, and troubleshoot issues. You can also add filtered log streams and log analytics graphs to your dashboards. These elements help you to visualize and correlate log data with metrics and distributed traces from the rest of your Google Cloud environment.

Cleaning up

The easiest way to eliminate billing is to delete the Cloud project that you created for the tutorial. Alternatively, you can delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:
• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.
1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the individual resources

After you complete this scenario, follow these steps to disable the Datadog integration with Google Cloud and revert any changes to your Cloud Logging configuration.

1. In your Datadog account, go to the Google Cloud integration tile and select Uninstall Integration. If you're an EU user, use the EU page. If you're a non-EU user, use the default US page.

2. In Cloud Shell, save your project's IAM policy as a file:

   gcloud projects get-iam-policy ${PROJECT_ID} | tee /tmp/policy.yaml
   

3. Edit the /tmp/policy.yaml file to empty the auditConfigs section. Do not remove this section entirely. Leave it empty. If you remove it completely, the Resource Manager setIamPolicy can't apply any changes to your Data Access audit logs configuration:

   auditConfigs:
   

4. Apply the new IAM policy to your project:

   gcloud projects set-iam-policy ${PROJECT_ID} /tmp/policy.yaml
   

5. Delete the Cloud Logging sink:

   gcloud logging sinks delete export-logs-datadog
   

6. Delete the Pub/Sub topic and subscription:

   gcloud pubsub topics delete datadog-logs-export-topic
   gcloud pubsub subscriptions delete datadog-logs-export-subscription
   

7. Delete the service account that you created for the Datadog integration with Google Cloud:

   1. In the Cloud Console, go to the Service Accounts page:

      Go to Service Accounts

   2. Select the service account that you created for the Datadog integration with Google Cloud.

   3. Click Delete, and then click Delete again to confirm.

What's next

• Learn how to get started with Datadog Log Management.
• Learn how to configure Datadog to monitor your Anthos-managed infrastructure.
• See the other export scenarios:
  • Scenario – Export for compliance requirements.
  • Scenario – Export for security and access analytics.
  • Scenario – Export to Elasticsearch.
  • Scenario – Export to Splunk.
• Try out other Google Cloud features for yourself. Have a look at our tutorials.