Logging and monitoring

Stay organized with collections Save and categorize content based on your preferences.

Logging provides important functionality to development organizations, audit organizations, and security organizations, as well as helping to satisfy regulatory compliance. As shown in Figure 2.9.1, there are a number logging sources in the example.com organization that are aggregated by Google Cloud Logging.

Figure 2.9.1 Logging structure for example.com

Figure 2.9.1 Logging structure for example.com

Within the reference example.com organization, logging functions are split into two projects:

  • A standalone logging project named Logging. This project contains Pub/Sub topics, Cloud Functions and a BigQuery instance used for collecting and processing logs generated by the example.com organization.
  • A SCC Alert Center project named SCC. This project contains the notification Pub/Sub topics from the Security Command Center. It's separate from the logging project so that you have a clear separation of duties between operations teams that might need general log access and the security team that needs access to specific security information and events.

In Google Cloud, logs are sent to the Cloud Logging API, where they pass through the Logs Router. The Logs Router checks each log entry against existing rules to determine which log entries to discard, which log entries to ingest (store) in Cloud Logging, and which log entries to export through log sinks. Table 2.9.1 shows the logs that are collected as part of example.com foundation and how those logs are enabled.

Log source Description Management
Audit Logs Google Cloud services write audit log entries to these logs to answer the question of "who did what, where, and when?" with Google Cloud resources. Enabled at an organization level and configured by the pipeline during the initial organization setup.
VPC Flow Logs VPC Flow Logs records a sample of network flows that are sent from and received by VM instances, including instances that are used as GKE nodes. The sample is typically 50% or less of the VPC network flows. Enabled for each VPC subnet and configured by the pipeline during project creation.
Firewall Rules Logging Firewall logging creates a record each time a firewall rule allows or denies traffic. Enabled for each firewall rule and configured by the pipeline during firewall creation.
Google Workspace audit logging Google Workspace allows its logs to be shared with the Google Cloud logging service. Google Workspace collects Login logs, Admin logs, and Group logs. Enabled at an organization level and configured through the Cloud Identity Admin Console.
Data Access audit logs Data Access audit logs record API calls that read the configuration or metadata of resources, as well as user-driven API calls that create, modify, or read user-provided resource data. Enabled at an organization level and configured by the pipeline during the initial organization setup.
Access Transparency logs Access Transparency provides logs that capture the actions Google personnel take when accessing your content. Enabled at an organization level and configured by raising a support case with Google Cloud.

Table 2.9.1 Log sources in example.com

Exporting to a sink involves writing a query that selects the log entries that you want to export and choosing a destination in Cloud Storage, BigQuery, or Pub/Sub. Logging information can be excluded by changing the query. Table 2.9.2 describes the sinks that are used in the example.com architecture.

Sink Description Purpose
sk-c-logging-bq Sends the aggregate logs from the organization to a BigQuery table in the logging project. The aggregated logs can be analyzed in BigQuery and used as part of the detective control architecture that's described in Detective controls.
sk-c-logging-bkt Sends the aggregate logs from the organization to a Cloud Storage bucket in the logging project. The aggregated logs can be used for compliance, audit, and incident-tracking purposes. For regulatory purposes, you can apply Cloud Storage Bucket Lock.
sk-c-logging-pub Sends the aggregated logs from the organization to a Pub/Sub topic in the logging project. The aggregated logs are sent from Pub/Sub to a Dataflow job and from there to an external SIEM, as described in Detective controls.

Table 2.9.2 Log sinks in example.com

The sinks and sink destinations are created through the deployment pipeline when the organization objects are first created and configured. When a sink is created, a service account identity is returned; that service account must have permission to write to the specified destination. Those permissions are also configured during setup.

Detective controls

Detective controls use platform telemetry to detect misconfigurations, vulnerabilities, and potentially malicious activity in the cloud environment. As shown in Figure 2.10.1, the example.com reference architecture has extensive detective controls to support its overall security posture.

Figure 2.10.1 Detective control architecture in example.com

Figure 2.10.1 Detective control architecture in example.com

The example.com architecture includes the following detective controls:

  • Google security sources through Security Command Center, including security misconfiguration information and vulnerability changes. (For details, see the next section.)
  • Connection of Google Cloud logs to Chronicle.
  • Asset changes and policy compliance notifications from Cloud Asset Inventory.
  • Logs from Google Cloud Logging (including Google Workspace Audit logs) that are sent to the SIEM tool.
  • Custom findings from a serverless BigQuery-based tool.

In addition, if you have existing event sources from any of our third-party providers or have created your own custom sources and custom findings, you can integrate those findings into Security Command Center.

Security Command Center

Security Command Center in Google Cloud provides a single user interface and data platform to aggregate and manage security findings. The example.com architecture uses Security Command Center to aggregate native findings from Google security sources as listed in Security sources and custom findings from the BigQuery analysis solution. It then sends Security Command Center notifications to a Pub/Sub topic for the SIEM to consume.

Premium and Standard

Google Security Command Center is available in two versions:

  • Security Command Center Premium. This is focused on customers with strong security requirements, especially enterprise customers. It includes a full suite of vulnerability and threat detection capabilities as well as integration options that simplify both the connection to third-party SIEMs and the automation of response and remediation actions. Security Command Center Premium is a paid service.
  • Security Command Center Standard. This provides a basic subset of the Security Command Center Premium features for all customers. Its focus is on enabling customers to discover, identify, alert on, manage, and respond to the critical vulnerabilities in their estate. Security Command Center Standard is available to customers free of charge.

Whether you choose Premium or Standard, you can enable the selected Security Command Center capabilities and manage the setup and configuration of the native detectors in the Security Command Center in a unified, centralized way.

The built-in security sources in both Security Command Center Premium and Standard are set by default to apply to all existing and future folders and projects in your Google Cloud organization. Unless there is a specific special security situation, we strongly recommend that you keep this default setting.

Security sources

Security Command Center aggregates findings from a wide variety of security sources, consisting of native Google provided findings, third-party findings, and customer-generated custom findings. Table 2.10.1 lists the security event sources that are enabled for the example.com reference architecture.

Security event source Description
Security Health Analytics Detects common vulnerabilities, misconfigurations, and drift from secure configurations.
Anomaly Detection Uses behavior signals from outside your system to display granular information that helps you detect cloud abuse behavior for your projects and VM instances.
Event Threat Detection Automatically scans various types of logs for suspicious activity in your Google Cloud environment.
Web Security Scanner Scans public web applications for common application vulnerabilities.
Container Threat Detection Provides runtime security for GKE containers to help detect reverse shells, suspicious binaries, and libraries.

Table 2.10.1 Security event sources for example.com

Setting up basic security alerting

Beyond detection, it's important for you to be able to take action to respond and remediate detected vulnerabilities and threats. You should leverage the built-in Security Command Center Pub/Sub topic to connect Security Command Center findings to your downstream alerting systems, ticketing, workflow, SOAR systems, and SIEMs. Security Command Center Pub/Sub notifications are managed through a set of notification configs. You can create your notification configs using either the gcloud command-line tool or client libraries and the API. To start, you should create a dedicated SCC Alerts project and restrict IAM access to just the users and services that are allowed to change and manage notification configurations. Within the project, you can then set up the Pub/Sub topics to which you will be publishing events. For each topic, you should further restrict access to that topic to only the necessary individuals and services whose duties require access to the security findings information in the topic.

Configuring notifications

After the security-alerting project is created and the topics have been created, you should define your notification configs. For detailed instructions, see Setting up finding notifications in the Security Command Center documentation. For information about managing notification configs, see Creating and managing Notification Configs, and for information about filtering notifications, see Filtering notifications.

The default quota on notification configuration counts is 500 per organization. If you need more, you should submit a request to increase your Security Command Center API quota using the Google Cloud console, as shown in Figure 2.10.2.

Figure 2.10.2 Request in Google Cloud console for additional notification
configs quota

Figure 2.10.2 Request in Google Cloud console for additional notification configs quota

The first notification config you should create is one that sends all Security Command Center findings to a Pub/Sub topic that you can then connect to your SIEM. You need the following roles:

  • Security Center Admin (broad permissions)

    or

    Security Center Notification Config Editor (specific permissions)

  • Pub/Sub Admin on the Pub/Sub topic

After you've set up the roles, follow these instructions:

  1. In the Google Cloud console, open Cloud Shell, or open a terminal at a computer where you have the Google Cloud CLI installed and configured.
  2. Set up the Pub/Sub topic:

    # topic-id = siem-export
    gcloud pubsub topics create topic-id
    export TOPIC_ID=topic-id
    
    # subscription-id = siem-subscription
    gcloud pubsub subscriptions create subscription-id --topic topic-id
    
  3. Set up the notification config with the filter:

    export ORGANIZATION_ID=organization-id
    export PUBSUB_PROJECT=project-id
    export GCLOUD_ACCOUNT=your-username@email.com
    
  4. Grant the gcloud account the required roles and permissions:

    gcloud pubsub topics add-iam-policy-binding
      "projects/$PUBSUB_PROJECT/topics/$TOPIC_ID"
      --member="user:$GCLOUD_ACCOUNT"
      --role="roles/pubsub.admin"
    
    gcloud organizations add-iam-policy-binding $ORGANIZATION_ID
      --member="user:$GCLOUD_ACCOUNT"
      --role="roles/securitycenter.notificationConfigEditor"
    
    # The topic to which the notifications are published
    PUBSUB_TOPIC="projects/project-id/topics/topic-id"
    
    # The description for the NotificationConfig
    DESCRIPTION="Notifies for active findings"
    
    # Filters for all active findings - both vulnerabilities and threats
    FILTER="state=ACTIVE"
    gcloud alpha scc notifications create notification-name
      --organization "$ORGANIZATION_ID"
      --description "$DESCRIPTION"
      --pubsub-topic $PUBSUB_TOPIC
      --filter $FILTER
    

For the full set of instructions, including information about how to set up notifications configs using the client libraries and API, see Setting up finding notifications.

Matching the notification configurations to your organization's hierarchy

You can use the definition and organization of notification topics to configure the routing of security alerts to the appropriate teams within your organization. Security Command Center enables you to create and store a set of notification configurations, each of which is a unique combination of selection filter and Pub/Sub topic. You match your specific requirements and enforce IAM access policies to security findings by controlling the following:

  • The number of notification configs that you create.
  • The selection filter in each config.
  • The name of the topic where the selected and filtered findings events are published.

The following sections describe four examples of successful notification configuration and topic patterns.

Example Description
One security queue The simplest pattern is just one notification config, with the selection filter set for all projects and all findings categories and the target set to just one topic (for example, gcp-security-alerts). This pattern is for customers who want all Google cloud security vulnerability and threat findings to be routed to a single centralized Security Operations Center (SOC).
By line of businessYou should create multiple separate notification configs when you've adopted the approach where each line of business has its own SOC and therefore owns the end-to-end security responsibilities for your infrastructure, workloads, and applications. Then you should set up a unique Security Command Center notification config and a dedicated topic for each line-of-business SOC. The selection filter in the notification config should select all projects within the scope of the line of business and select all findings categories*. *This configuration enables IAM control and separation on the dedicated topics between each line-of-business team so that each team has access to and receives only the vulnerability and threat signals relevant to them.
Cloud-native DevSecOpsSimilarly, you should create multiple separate notification configs when you've adopted the approach where each application team owns the end-to-end security responsibilities for their infrastructure, workloads, and applications. Then you should set up a unique Security Command Center notification config along with a dedicated topic for each application team. The selection filter in the notification config should select all projects within the scope of the application team and all findings categories. This configuration enables IAM control and separation on the dedicated topics between each application team so each application team has access to and receives only the vulnerability and threat signals relevant to them.
By Security finding categoryYou should create multiple separate notification configs when you've adopted the approach where different types of security findings are handled by different customer teams. The most common example of this is when you've chosen to separate the response and remediation of vulnerabilities and misconfigurations from those of live, active threats. We've seen many customers route the first to their cloud security team and the second to their SOC. In this example case, you can set up a unique Security Command Center notification config and a dedicated Pub/Sub topic for the two teams. The selection filter in the notification config for misconfigurations and vulnerabilities should select all projects and all the finding sources for vulnerabilities (for example, Security Health Analytics and Web Security Scanner). The selection filter in the notification config for threats should select all projects and all the finding sources for threats (for example, Event Threat Detection, Container Threat Detection, and Abuse Anomaly Detection).

Vulnerability and drift detection

Configuration drift refers to changes in configuration away from a predefined standard or baseline. Over time, as a solution evolves, a certain amount of drift is necessary, but for security or stability reasons, some configuration details should not be violated. Managing drift can be a manual process where an administrator is notified and then takes action where needed. But it can also be automated for specific changes, reverting a change that violates a policy.

Built-in drift detection using Security Command Center Premium

Security Command Center Premium includes Security Health Analytics, which automatically runs over 100 different misconfiguration and vulnerability checks daily against all the Security Command Center-supported resources in your organization. The resulting assessments are listed in the Vulnerabilities tab of the Security Command Center dashboard. You can view and explore the data either at the organization level or for specific sets of projects. You can also further filter the data by status, category, recommendation, severity, active or inactive status, and compliance benchmark. Then in the Compliance tab dashboards, the assessment results are specifically matched against CIS 1.0, NIST-800-53, PCI-DSS, and ISO 27001 benchmarks and summarized for either the full organization or for a specific set of customer-selected projects. Examples of both tabs are shown in Figure 2.10.3 and Figure 2.10.4.

Figure 2.10.3 The Vulnerability tab in Security Command Center Premium

Figure 2.10.3 The Vulnerability tab in Security Command Center Premium

Figure 2.10.4 The Compliance tab in Security Command Center Premium

Figure 2.10.4 The Compliance tab in Security Command Center Premium

With the Security Command Center, you can set up different notification configs to generate alerts on changes in the Security Health Analytics findings to help you identify drift in the configurations of the resources that you care about. In addition, in the Security Command Center dashboard, you can use the Asset Changed view and Findings Changed view to discover and explore changes in user-selected time ranges (for example, last hour, last day, last 7 days, last 30 days, and so on). Examples of both tabs are shown in Figure 2.10.5 and Figure 2.10.6.

Figure 2.10.5 The Asset Changed tab in the Security Command Center dashboard

Figure 2.10.5 The Asset Changed tab in the Security Command Center dashboard

Figure 2.10.6 The Findings Changed tab in the Security Command Center dashboard

Figure 2.10.6 The Findings Changed tab in the Security Command Center dashboard

In the dashboards, you can explore the time ranges relative to now—that is, to the current time. If instead you want the reference point to be a different time, you can use either the Security Command Center command-line interface or the API or client libraries to set your own reference point.

Managed web vulnerability scans

If you deploy web applications, you should also leverage Security Command Center Premium's built-in managed web vulnerability scans. Web Security Scanner currently provides 13 built-in web vulnerability detections that cover 4 of the OWASP top 10. Managed scans automatically run once each week to detect and scan public web endpoints.

Active threat detection

In addition to enabling built-in vulnerability and misconfiguration assessments and threat prevention, you should also leverage Security Command Center Premium's built-in active threat detection capabilities. The initial set that's built in to Security Command Center Premium includes Event Threat Detection and Container Threat Detection.

Event Threat Detection

Event Threat Detection is a built-in service with Security Command Center Premium that detects threats that target your Google Cloud assets. It provides near real-time detection of threats from platform logs, network logs, and compute logs, and it leverages native Google threat intelligence and detection algorithms. Findings are automatically written to the Security Command Center and can also be exported to Cloud Logging. Event Threat Detection performs basic deduplication on findings for users and is enabled by default when you're using Security Command Center Premium.

Container Threat Detection

Container Threat Detection is a built-in service with Security Command Center Premium that detects threats that target Google Kubernetes Engine (GKE) containers. It provides near real-time detection of reverse shell execution, suspicious binary execution, and the linking of suspicious libraries. Container Threat Detection is uniquely able to make the connection between containers and underlying physical nodes at the time of threat detection.

To use Container Threat Detection, you must run the latest Container-Optimized OS image for your GKE environment. Container Threat Detection automatically deploys and manages a daemonset container on every node to transmit data plus additional information that identifies the container. Findings are automatically written to Security Command Center and Cloud Logging.

Real-time compliance monitoring of custom policies

To monitor compliance of custom policies in real time, the example.com architecture uses Cloud Asset Inventory real-time notifications. Cloud Asset Inventory can send a Pub/Sub message for each configuration change to a specific asset name or to an asset type. The message triggers a Cloud Function to examine the change. If that change represents a policy violation, the Cloud Function can take action such as reverting the change or notifying an administrator.

One policy that's monitored in example.com is if external Gmail accounts are being granted any permissions to example.com projects. A Cloud Function is used to check when an IAM policy is created or modified that has a Gmail address as a member. If the Cloud Function detects that the policy has been violated, the Cloud Function reverts the change and sends a custom finding to the Security Command Center. Figure 2.10.7 shows this configuration.

Figure 2.10.7 Automatically reverting an IAM policy change and sending a
notification

Figure 2.10.7 Automatically reverting an IAM policy change and sending a notification

Cloud Asset Inventory also enables you to export configuration metadata to a BigQuery table for in-depth analysis. When the data is in BigQuery, you can use SQL queries to explore the configuration and build reports. Listing 2.10.1 shows an example to detect IAM policies that grant access to a Gmail account.

SELECT name, asset_type, bindings.role
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
JOIN UNNEST(iam_policy.bindings) AS bindings
JOIN UNNEST(bindings.members) AS members
WHERE members like "%@gmail.com"

Listing 2.10.1 SQL query to identify IAM bindings with @gmail.com members

Integration with Chronicle

Cloud Logging and Security Command Center events can be exported to Chronicle, which is purpose-built for security threat detection and investigation. Chronicle is built on the Google infrastructure, which lets you take advantage of Google's scale to reduce your time to investigate and triage potential threats. It correlates data from multiple security sources and threat feeds, providing a single timeline-based aggregated view of events and detections. You can leverage Chronicle's detection rules or use real-time search and custom rules to create your own detections.

SIEM solutions integrations

An enterprise SIEM product can be useful for the overall aggregation and visibility of security events, but sometimes the SIEM can be inefficient when dealing with cloud-scale logging. Both Security Command Center and Cloud Logging can output configurable built-in Pub/Sub event streams. You can therefore choose the option that best fits your needs.

As shown earlier in Figure 2.10.1, the example.com reference architecture includes a mechanism for a SIEM tool to ingest Google Cloud logs through a Pub/Sub subscription. A log sink in Cloud Logging is used to put all required logs into a raw logs Pub/Sub topic. A Dataflow job subscribes to the raw logs topic and aggregates the logs before putting them into a processed-logs Pub/Sub topic to which the SIEM can subscribe.

Integrations with Splunk

Both Cloud Logging and Security Command Center events can be directly exported to Splunk and can leverage the Splunk Add-on for Google Cloud for integration. For Cloud Logging, you can set up a logging export to Splunk. For Security Command Center, you can set up a notification config. In both cases, after the Pub/Sub event stream is set up, you can leverage the approved Pub/Sub Splunk Dataflow template to complete the integration.

Analyzing your security data using BigQuery

In cases where exporting to an enterprise SIEM is inefficient, an option is to build your own security analysis solution that's made up of cloud-native tools, including BigQuery, Pub/Sub, Cloud Functions, and Security Command Center. This solution is capable of working with massive log volumes. It can identify security events and forward the events to an enterprise SIEM as findings, rather than as raw logs.

Building your own analysis solution

The example.com reference architecture has some cloud-specific security events that are monitored through the BigQuery-based solution. These security event logs are aggregated and surfaced as custom findings in Security Command Center, and are forwarded to the enterprise SIEM tool through the notifications Pub/Sub topic. This BigQuery-based architecture can handle log sources that have a high log volume, such as data-access logs and VPC Flow Logs. For details, see Figure 2.10.8.

Figure 2.10.8 BigQuery-based analysis architecture

Figure 2.10.8 BigQuery-based analysis architecture

The BigQuery-based solution works as follows:

  1. Logs from Cloud Logging are stored to a raw-data dataset in BigQuery using a log sink. This dataset can be configured to expire on a regular basis in order to help manage costs.
  2. Views are added to a BigQuery dataset that represent the security events being monitored. A view is like a stored query, and in this case, it should produce one entry for each incident of the security event being monitored. Monitored use cases are described in Examples of alerting use cases.
  3. Cloud Scheduler pushes an event to a scheduling Pub/Sub topic every 15 minutes. A Cloud Function is configured to trigger on every scheduling event. The Cloud Function queries the views for new events in the last 20 minutes to ensure no missed events; if it finds events, it pushes them to the Security Command Center API as custom findings.
  4. The Security Command Center notifications feature pushes all new findings, including custom and Google or third-party security sources, to a Pub/Sub findings topic.
  5. The enterprise SIEM subscribes to this findings topic to receive the security events.

Examples of alerting use cases

Alerting use cases are implemented as SQL queries that are saved as views in BigQuery. The following list specifies conditions that generate alerts using the BigQuery-based architecture.

  • A login occurs from a high-privilege account login (Super Admin, Organization Admin, and so on).
  • IAM permissions are granted to a user who is from a non-allowed domain.
  • Changes are made to logging settings.
  • VPC Flow Logs with IP addresses that are outside of expected ranges are detected.
  • Changes are made to permissions on critical encryption keys.
  • Non-approved Google services are used.

You can extend these use cases by writing a query and saving it as a view in the BigQuery dataset so that it gets queried regularly for new results with views for other use cases. As an example, the following query (Listing 2.10.2) addresses the case where logging settings are changed.

SELECT
  receiveTimestamp,
  timestamp AS eventTimestamp,
  protopayload_auditlog.requestMetadata.callerIp,
  protopayload_auditlog.authenticationInfo.principalEmail,
  protopayload_auditlog.resourceName,
  protopayload_auditlog.methodName
FROM
  `${project}.${dataset}.cloudaudit_googleapis_com_activity_*`
WHERE
  protopayload_auditlog.serviceName = "logging.googleapis.com"

Listing 2.10.2 SQL query to detect logging setting changes