Collect logs on Anthos with Splunk Connect

This tutorial describes how to install and use Splunk Connect for Kubernetes to collect logs from Kubernetes applications running on Anthos in Google Cloud. Anthos lets you take advantage of Kubernetes and cloud technology in your existing on-premises investments or in the public cloud. You get the Google Kubernetes Engine (GKE) experience with quick, managed, simple installations and upgrades validated by Google. Splunk Connect can index, store, and analyze your logs on-premises, as shown in the following diagram:

Architecture for logging Anthos deployed on VMware with Splunk.

The tutorial is intended for administrators setting up the integration of logs and metrics from Anthos into Splunk. In this tutorial, you configure an HTTP Event Collector for Splunk Enterprise, deploy Splunk Connect for Kubernetes in your Anthos cluster, and deploy a sample application.

This tutorial assumes that you are familiar with Kubernetes and also has the following technical requirements:

  • You are an Anthos customer or participate in the Anthos Free Trial program.
  • You have a previously configured Anthos installation with a running user cluster.
  • You are currently running Splunk Enterprise (7.0 or later) on-premises in your organization.

If you're running Anthos on Google Cloud or other public clouds and want to connect to a Splunk instance running on-premises, you need to have hybrid connectivity between the two environments in place, as shown in the following diagram:

Architecture for logging Anthos on Google Cloud or other public clouds with Splunk

Objectives

  • Install and use Splunk Connect for Kubernetes to collect logs from Kubernetes applications running on Anthos on Google Cloud.
  • Configure an HTTP Event Collector for Splunk Enterprise.
  • Deploy Splunk Connect for Kubernetes in your Anthos cluster.
  • Deploy a sample application.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. In the Cloud Console, on the project selector page, select or create a Cloud project.

    Go to the project selector page

  3. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  4. Make sure that you have Splunk deployed and configured. See Getting started with Splunk and deploy Splunk Enterprise.
  5. Make sure that you have an Anthos user cluster deployed and registered in Google Cloud Console. This tutorial assumes there are three nodes in the cluster, but this is not mandatory. Follow Anthos documentation to deploy this Anthos component.
  6. Ensure that there is network connectivity between your Anthos cluster and Splunk.

Configuring Splunk Enterprise

Set up indexes

Splunk Connect requires a minimum of two indexes for exporting data—one for events and one for metrics. In this tutorial, you also create a separate events index for objects, for a total of three indexes.

  1. Log in to Splunk as an administrator.
  2. Go to Settings > Indexes.
  3. Click New Index.
    1. In the Index Name text box, enter events.
    2. Set Index Data Type to Events.
  4. Click Save.
  5. Click New Index.
    1. In the Index Name text box, enter objects.
    2. Set Index Data Type to Events.
  6. Click Save.
  7. Click New Index.
    1. In the Index Name text box, enter metrics.
    2. Set Index Data Type to Metrics.
  8. Click Save.

Set up an HTTP Event Collector (HEC)

To export logs and metrics from Anthos to Splunk, you need to set up the Splunk HTTP Event Collector (HEC).

  1. Go to Settings > Data inputs.
  2. Click Global Settings.
    1. Set All Tokens to Enabled.
    2. Select Enable SSL.
    3. Click Save.
  3. In the HTTP Event Collector dialog, select +Add New.
  4. In the Name text box, enter anthos-hec, and then select Next.
  5. In the Input Settings dialog, do the following:
    1. Set App Context to Search & Reporting (search).
    2. In the Select Allowed Indexes menu, select events, metrics, and objects.
    3. Click Next.
  6. The Review page appears. Check that the page looks as follows:

    Review page for HEC Collector.

  7. Click Submit to create the HEC. Copy the resulting token value for use later.

  8. To check connectivity, log in to a node in your Anthos cluster and send a test event to Splunk. Replace hec-token with the token value created earlier, and replace splunk-ip with the IP of your Splunk Enterprise server, as shown in the following code:

    curl -k http://splunk-ip:8088/services/collector/event \
    -H "Authorization: Splunk hec-token" -d '{"event": "hello splunk"}'
    

    If the request is successful, the server returns a response like the following:

    {"text":"Success","code":0}
    

Deploying Splunk Connect for Kubernetes

To deploy Splunk Connect for Kubernetes, you need to create a Splunk service account, install Helm, and deploy Splunk Connect.

Log in to the Anthos user cluster as an administrator

  • From a shell in your Anthos administrator workstation, ensure that your kubectl tool is configured to point your Anthos cluster by default:
  • kubectl config current-context
  • The preceding command returns your Anthos cluster. If it doesn't, link the master kubeconfig file to a local config file and try again:
  • mkdir .kube
    ln -s /kubeconfig ~/.kube/config
    

Install Helm

You install Splunk Connect with the Helm package manager.

  1. Install the Helm client and ensure that Helm is in your path:

    helm version
    

    The output is similar to the following, though the version and commit numbers might differ:

    version.BuildInfo{Version:"v3.2.3", GitCommit:"8f832046e258e2cb800894579b1b3b50c2d83492", GitTreeState:"clean", GoVersion:"go1.13.12"}
    

Deploy Splunk Connect

  1. Create a Splunk namespace. This namespace is used to isolate Splunk Connect from the rest of your cluster:

    kubectl create namespace splunk
    
  1. Create a YAML file containing the Splunk Connect configuration for reading logs, objects, and metrics. Replace hec-token with the token value created earlier and replace splunk-ip with the IP of your Splunk Enterprise server:

    cat <<EOF >values.yaml
    global:
      splunk:
        hec:
          protocol: https
          insecureSSL: true
          token: hec-token
          host: splunk-ip
          port: 8088
          indexName: events
    
    splunk-kubernetes-logging:
      journalLogPath: /run/log/journal
      splunk:
        hec:
          indexName: events
    
    splunk-kubernetes-objects:
      objects:
        core:
          v1:
            - name: pods
            - name: namespaces
            - name: nodes
            - name: services
            - name: config_maps
            - name: persistent_volumes
            - name: service_accounts
            - name: persistent_volume_claims
            - name: resource_quotas
            - name: component_statuses
            - name: events
              mode: watch
        apps:
          v1:
            - name: deployments
            - name: daemon_sets
            - name: replica_sets
            - name: stateful_sets
      splunk:
        hec:
          indexName: objects
    
    splunk-kubernetes-metrics:
      kubernetes:
        insecureSSL: true
      splunk:
        hec:
          indexName: metrics
    EOF
    
  2. Install Splunk Connect for Kubernetes into the splunk namespace:

    helm install anthos-splunk -f values.yaml --namespace splunk \
    https://github.com/splunk/splunk-connect-for-kubernetes/releases/downlo
    ad/1.4.1/splunk-connect-for-kubernetes-1.4.1.tgz
    
  3. Check that Splunk Connect has been successfully started:

    kubectl -n splunk get pods
    

    You should see 3 logging pods, 3 metrics pods, 1 metrics aggregator pod, and 1 objects pod running, as follows:

    NAME                                                          READY   STATUS    RESTARTS   AGE
    anthos-splunk-splunk-kubernetes-logging-46srr                  1/1     Running   0          2m
    anthos-splunk-splunk-kubernetes-logging-gkgkm                  1/1     Running   0          2m
    anthos-splunk-splunk-kubernetes-logging-r52nz                  1/1     Running   0          2m
    anthos-splunk-splunk-kubernetes-metrics-agg-595946f874-vjzmb   1/1     Running   0          2m
    anthos-splunk-splunk-kubernetes-metrics-gbkb8                  1/1     Running   0          2m
    anthos-splunk-splunk-kubernetes-metrics-k6ksn                  1/1     Running   0          2m
    anthos-splunk-splunk-kubernetes-metrics-vg5wn                  1/1     Running   0          2m
    anthos-splunk-splunk-kubernetes-objects-7c5cd8cf7d-csvw6       1/1     Running   0          2m
    
  4. Go to Settings > Indexes in the Splunk UI. If the connection is successful, the Latest Event column shows a recent time for the events, metrics, and objects indexes.

Troubleshooting and making changes

  • If a pod isn't starting or data isn't coming through, inspect the pod's logs for errors:

    kubectl -n splunk logs pod-name
    
  • If you need to update your config.yaml file and try again, run the following commands to redeploy Splunk Connect and recreate its pods:

    helm upgrade --namespace splunk -f values.yaml anthos-splunk \
    https://github.com/splunk/splunk-connect-for-kubernetes/releases/downlo
    ad/1.4.1/splunk-connect-for-kubernetes-1.4.1.tgz
    kubectl delete pods -n splunk --all
    

Deploying and testing a sample app

Next, you deploy a sample app and view its logs, metadata, and metrics in Splunk.

Deploy a sample app

  1. Deploy a simple "Hello world" app on Anthos:

    kubectl create deployment hello-kubernetes --image=paulbouwer/hello-kubernetes:1.5
    
  2. Expose the app on port 8080 and port-forward it so that it can be accessed locally:

    kubectl expose deployment hello-kubernetes --port=8080 --type=ClusterIP
    kubectl port-forward deployment/hello-kubernetes 8080:8080
    
  3. Test the app by using the curl command-line tool:

    curl http://localhost:8080/
    

    If the request is successful, the server returns a response like the following:

      <!DOCTYPE html>
      <html>
      <head>
          <title>Hello Kubernetes!</title>
      [...]
    

View app logs

  • Browse to Search in the Splunk Web UI and search for the sample app's logs in the events index:

    index=events sourcetype="kube:container:hello-kubernetes"
    

    This search returns log events from the app in reverse chronological order (newest first), starting with the test request and the entry indicating that the app is listening and ready for service, as shown in the following screenshot:

    Log event entries.

View app metadata

  • In the Search pane, search for metadata objects for the sample app:

    index="objects" "hello-kubernetes"
    

    This search returns a list of metadata snapshots for various Kubernetes objects. For example, by selecting sourcetype="kube:objects:deployments" and expanding metadata and status, you can check the status of the hello-kubernetes deployment at a specific point in time, including the number of available replicas, as shown in the following screenshot:

    Metadata snapshots for Kubernetes objects.

    By default, metadata is polled every 15 minutes. This can be adjusted with the interval setting in the values.yaml file.

    The metadata also includes a stream of events from the Kubernetes API. The following search returns all events for the sample app:

    index="objects" sourcetype="kube:objects:events*" "hello-kubernetes"
    

View system and app metrics

  1. Go to Metrics in the Splunk UI and enter cpu in the search box.

  2. In the drop-down menu, select Metrics > kube > namespace > cpu > usage.

  3. In the Analysis pane on the right, select Split by, and then select name. You'll see a graph of CPU usage categorized by namespace, as shown in the following screenshot:

    CPU usage graph

Deploying Splunk App for Infrastructure

The Splunk App for Infrastructure (SAI) provides insight into the performance of Linux servers, Microsoft Windows servers, Kubernetes clusters, OpenShift containers, Docker containers, and Amazon EC2, ELB, and EBS instances. SAI uses metrics for performance monitoring, and it uses log data for deep understanding and troubleshooting of your server infrastructure.

For a simplified setup installation that is fully compatible with the features provided in the Splunk App for Infrastructure, install both the Splunk Add-on for Infrastructure and the Splunk App for Infrastructure in your Splunk environment. Then follow these steps for generating an installation script for generating an installation script that configures data delivery to Splunk.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Delete the Cloud project

The easiest way to eliminate billing is to delete the Cloud project you created for the tutorial.

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project that you want to delete and then click Delete .
  3. In the dialog, type the project ID and then click Shut down to delete the project.

Delete the resources

After completing this tutorial, run these commands to remove the created resources:

   helm delete --namespace splunk anthos-splunk
   kubectl delete deployment hello-kubernetes

What's next