Edit on GitHub
Report issue
Page history

Exporting Stackdriver logs to Elastic Cloud

Author(s): @twenny ,   Published: 2019-03-20

Google Cloud Community tutorials submitted from the community do not represent official Google Cloud product documentation.


Overview

This tutorial explains how to export Stackdriver logs to the Elastic Cloud Elasticsearch SaaS platform to perform log analytics. Elastic Cloud is a SaaS offering, which saves time by not needing to build and manage the Elasticsearch infrastructure.

Stackdriver to Elastic Cloud architecture

Costs

This tutorial uses billable components of Google Cloud Platform (GCP), including Compute Engine.

New GCP users might be eligible for a free trial.

Configure GCP resources

The high-level steps in this section:

  1. Create a user-managed service account
  2. Create a VM for Logstash
  3. Create a Cloud Pub/Sub topic
  4. Create a Stackdriver log sink and subscribe it to the Cloud Pub/Sub topic

Enable APIs

Log in or sign up for Google Cloud Platform, then open the Cloud Console.

The examples in this document use the gcloud command-line inteface. GCP APIs must be enabled via the Services and APIs page in the console before they can be used with gcloud. To perform the steps in this tutorial, enable the following APIs:

  • Compute Engine
  • Cloud Pub/Sub
  • Identity and Access Management (IAM)
  • Stackdriver

Enable Cloud APIs

Activate Google Cloud Shell

The GCP Console provides an interactive shell that includes the gcloud command-line interface. At the top right corner of the page, click the Activate Google Cloud Shell button.

alt text

Create a service account

GCP best practices suggest using a service account to configure security controls to a VM. A service account is useful for a VM to determine which other GCP resources can be accessed by the VM and its applications, and which firewall rules should be applied to the VM.

While credentials can be created to be used by a service account, this step is not necessary when the service account is attached to a VM running on Google Compute Engine. Google manages the keys, and applications can retrieve the credentials securely with the metadata service.

  1. Create a service account to attach to the VM:

    gcloud iam service-accounts create logstash \
        --display-name="Logstash to Stackdriver"
    

    Expected response:

    Created service account [logstash].
    
  2. Provide IAM permissions allowing the new service account to access Cloud Pub/Sub using the pubsub.subscriber role.

    gcloud projects add-iam-policy-binding scalesec-dev \
    --member serviceAccount:logstash@scalesec-dev.iam.gserviceaccount.com \
    --role roles/pubsub.subscriber
    

    Excerpt of expected response:

    Updated IAM policy for project [scalesec-dev].
    [...]
    - members:
      - serviceAccount:logstash@scalesec-dev.iam.gserviceaccount.com
      role: roles/pubsub.subscriber
    [...]
    etag: BwWEjM0909E=
    version: 1
    

Create a Cloud Pub/Sub topic and subscription

  1. Create a Cloud Pub/Sub topic where Stackdriver will send events to be picked up by Logstash:

    gcloud pubsub topics create stackdriver-topic
    

    Expected response:

    Created topic [projects/scalesec-dev/topics/stackdriver-topic].
    

    Next, create a subscription:

    gcloud pubsub subscriptions create logstash-sub --topic=stackdriver-topic --topic-project=scalesec-dev
    

    Expected response:

    Created subscription [projects/scalesec-dev/subscriptions/logstash-sub].
    

Create a Stackdriver log sink

  1. Create a log sink to be used to export Stackdriver logs to the new Cloud Pub/Sub topic.

    gcloud logging sinks create logstash-sink pubsub.googleapis.com/projects/scalesec-dev/topics/stackdriver-topic \
    --log-filter='resource.type="project"'
    

    Expected response:

    Created [https://logging.googleapis.com/v2/projects/scalesec-dev/sinks/logstash-sink].
    Please remember to grant `serviceAccount:p352005273005-058743@gcp-sa-logging.iam.gserviceaccount.com` Pub/Sub
    Publisher role to the topic.
    More information about sinks can be found at /logging/docs/export/ 
    

    The filter specified above will produce events associated with changes to IAM, which is a typical area to be monitored closely. Stackdriver supports monitoring activities for vpn_gateway and other resource types. See the documentation for more filter ideas.

    The second part of the output is a reminder to verify that the service account used by Stackdriver has permissions to publish events to the Cloud Pub/Sub topic. The beta version of gcloud CLI supports permissions management for Cloud Pub/Sub.

    gcloud beta pubsub topics add-iam-policy-binding stackdriver-topic \
    --member serviceAccount:p352005273005-776084@gcp-sa-logging.iam.gserviceaccount.com \
    --role roles/pubsub.publisher
    

    Expected response:

    Updated IAM policy for topic [stackdriver-topic].
    bindings:
    - members:
      - serviceAccount:p352005273005-776084@gcp-sa-logging.iam.gserviceaccount.com
      role: roles/pubsub.publisher
    etag: BwWEi9uEM1A=
    

Create the Logstash VM

Note: Some system responses are omitted in this section for brevity.

  1. Create a VM to run logstash to pull logs from the Cloud Pub/Sub logging sink and send them to ElasticSearch:

    gcloud compute --project=scalesec-dev instances create logstash \
    --zone=us-west1-a \
    --machine-type=n1-standard-1 \
    --subnet=default \
    --service-account=logstash@scalesec-dev.iam.gserviceaccount.com \
    --scopes="https://www.googleapis.com/auth/cloud-platform" \
    --image-family=ubuntu-1804-lts \
    --image-project=ubuntu-os-cloud \
    --boot-disk-size=10GB \
    --boot-disk-type=pd-ssd \
    --boot-disk-device-name=logstash
    

    Expected response:

    Created [https://www.googleapis.com/compute/beta/projects/scalesec-dev/zones/us-west1-a/instances/logstash].
    NAME      ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
    logstash  us-west1-a  n1-standard-1               10.138.0.3   35.233.166.234  RUNNING
    

Create Elastic Cloud deployment

  1. Go to https://cloud.elastic.co/login. A trial account provides suitable service to complete this tutorial.

    Sign up for Elastic Cloud

  2. Create an Elasticsearch deployment. This example is deployed on GCP in us-west1.

    Create an Elastic Cloud deployment

  3. While the deployment is finishing up, make sure to capture the credentials and store them in a safe place. While the Cloud ID can be viewed from the deployment page, this is the only time the password for the elastic user is available. Visit the Security page to reset the password if needed. When considering production environments, create new Elasticsearch credentials with tighter permissions and avoid using the elastic user. As documented: "On a production system, you should adapt these examples by creating a user that can write to and access only the minimally required indices."

    Launching an Elastic Cloud deployment

  4. Obtain the URI of the Elasticsearch endpoint that has been provisioned. A link to this endpoint can be copied from the Deployments page. This value will be needed to configure Logstash output plugin configuration.

    Copy the Elasticsearch URI

    The next section provides steps to complete the setup to send events to the new Elasticsearch deployment.

Configure the Logstash VM

  1. Compute Engine supports several ways to access your VM. You can use the gcloud command in Cloud Shell to leverage oslogin to connect to the logstash VM via SSH, noting the zone from the VM creation step above.

    gcloud compute ssh logstash --zone us-west1-a
    
  2. Perform typical system updates and install OpenJDK:

    sudo apt-get update
    sudo apt-get -y upgrade
    sudo apt -y install openjdk-8-jre-headless
    echo "export JAVA_HOME=\"/usr/lib/jvm/java-8-openjdk-amd64\"" >> ~/.profile
    sudo reboot
    

    After a few moments, the VM will complete its reboot and can be accessed again via gcloud.

    gcloud compute ssh logstash --zone us-west1-a
    

Install Logstash

  1. Install logstash from Elastic.

    wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
    echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
    sudo apt-get update
    sudo apt-get install logstash
    
  2. Install the Logstash Plugin for Cloud Pub/Sub.

    cd /usr/share/logstash
    sudo -u root sudo -u logstash bin/logstash-plugin install logstash-input-google_pubsub
    

    Expected response:

    Validating logstash-input-google_pubsub  
    Installing logstash-input-google_pubsub  
    Installation successful  
    

Configure Logstash

Logstash comes with no default configuration.

  1. Create a new file /etc/logstash/conf.d/logstash.conf with these contents, modifying values as needed:

    input
    {
        google_pubsub {
            project_id => "scalesec-dev"
            topic => "stackdriver-topic"
            subscription => "logstash-sub"
            include_metadata => true
            codec => "json"
        }
        # optional, but helpful to generate the ES index and test the plumbing
        heartbeat {
            interval => 10
            type => "heartbeat"
        }
    }
    filter {
        # don't modify logstash heartbeat events
        if [type] != "heartbeat" {
            mutate {
                add_field => { "messageId" => "%{[@metadata][pubsub_message][messageId]}" }
            }
        }
    }
    output
    {
        stdout { codec => rubydebug }
        elasticsearch
        {
            hosts => ["https://c36297ebbc024cd4b29c98319dc8c38d.us-west1.gcp.cloud.es.io:9243"]
            user => "elastic"
            password => "NTmWdNJXkzMWL4kkIcIzY8O6"
            index => "logstash-%{+YYYY.MM.dd}"
        }
    }
    

Start Logstash

  1. Start Logstash:

    sudo service logstash start
    
  2. Monitor the startup logs closely for issues:

    sudo tail -f /var/log/syslog
    
  3. Review log messages. It may take a few moments for events to begin flowing.

    Log messages like these indicate that Logstash is working internally:

    Jul 15 20:43:09 logstash logstash[2537]: {
    Jul 15 20:43:09 logstash logstash[2537]:           "type" => "heartbeat",
    Jul 15 20:43:09 logstash logstash[2537]:      "messageId" => "%{[@metadata][pubsub_message][messageId]}",
    Jul 15 20:43:09 logstash logstash[2537]:        "message" => "ok",
    Jul 15 20:43:09 logstash logstash[2537]:     "@timestamp" => 2018-07-15T20:43:08.367Z,
    Jul 15 20:43:09 logstash logstash[2537]:       "@version" => "1",
    Jul 15 20:43:09 logstash logstash[2537]:           "host" => "logstash"
    Jul 15 20:43:09 logstash logstash[2537]: }
    

    Log messages like these indicate that Logstash is pulling events from Cloud Pub/Sub. Actual message content will differ.

    Jul 17 20:58:13 logstash logstash[15198]:              "logName" => "projects/scalesec-dev/logs/cloud.googleapis.com%2Fipsec_events",
    Jul 17 20:58:13 logstash logstash[15198]:             "resource" => {
    Jul 17 20:58:13 logstash logstash[15198]:         "labels" => {
    Jul 17 20:58:13 logstash logstash[15198]:             "project_id" => "scalesec-dev",
    Jul 17 20:58:13 logstash logstash[15198]:                 "region" => "us-west1",
    Jul 17 20:58:13 logstash logstash[15198]:             "gateway_id" => "1810546051445508503"
    Jul 17 20:58:13 logstash logstash[15198]:         },
    Jul 17 20:58:13 logstash logstash[15198]:           "type" => "vpn_gateway"
    Jul 17 20:58:13 logstash logstash[15198]:     },
    Jul 17 20:58:13 logstash logstash[15198]:             "severity" => "DEBUG",
    Jul 17 20:58:13 logstash logstash[15198]:           "@timestamp" => 2018-07-17T20:58:12.918Z,
    Jul 17 20:58:13 logstash logstash[15198]:          "textPayload" => "sending packet: from 35.233.211.219[500] to 35.231.4.41[500] (49 bytes)",
    Jul 17 20:58:13 logstash logstash[15198]:             "insertId" => "1e8d5s7f6uc4ap",
    Jul 17 20:58:13 logstash logstash[15198]:            "timestamp" => "2018-07-17T20:58:08.401562594Z",
    Jul 17 20:58:13 logstash logstash[15198]:             "@version" => "1",
    Jul 17 20:58:13 logstash logstash[15198]:               "labels" => {
    Jul 17 20:58:13 logstash logstash[15198]:         "tunnel_id" => "1091689068647389715"
    Jul 17 20:58:13 logstash logstash[15198]:     },
    Jul 17 20:58:13 logstash logstash[15198]:            "messageId" => "146817684320772",
    Jul 17 20:58:13 logstash logstash[15198]:     "receiveTimestamp" => "2018-07-17T20:58:08.65636792Z"
    

Configure Kibana

Kibana is a powerful graphical user interface that uses the underlying Elasticsearch data. This is the main console to monitor and triage security events and perform searches and investigations.

  1. Return to the Elasticsearch deployment page and click the link to Kibana.

    Click to access the Kibana UI

  2. Log in as the elastic user.

    Kibana login screen

  3. Navigate to the Management page to set up index patterns for Kibana.

    Manage Kibana index patterns

  4. Enter logstash-* for the index pattern.

    Configure logstash index pattern

  5. Use @timestamp for the time field.

    Specify index timestamp

Verify log flow

Return to the main Kibana dashboard (shown as Discover in the navigation menu). The Kibana dashboard should display Stackdriver events similar to those shown below:

log flow

Submit a tutorial

Share step-by-step guides

Submit a tutorial

Request a tutorial

Ask for community help

Submit a request

View tutorials

Search Google Cloud tutorials

View tutorials

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.