Collect logs on Anthos with Splunk Connect

This tutorial describes how to install and use Splunk Connect for Kubernetes to collect logs from Kubernetes applications running on Anthos in Google Cloud. Anthos lets you take advantage of Kubernetes and cloud technology in your existing on-premises investments or in the public cloud. You get the Google Kubernetes Engine (GKE) experience with quick, managed, simple installations and upgrades validated by Google. Splunk Connect can index, store, and analyze your logs on-premises, as shown in the following diagram:

The tutorial is intended for administrators setting up the integration of logs and metrics from Anthos into Splunk. In this tutorial, you configure an HTTP Event Collector for Splunk Enterprise, deploy Splunk Connect for Kubernetes in your Anthos cluster, and deploy a sample application.

This tutorial assumes that you are familiar with Kubernetes and also has the following technical requirements:

• You are an Anthos customer or participate in the Anthos Free Trial program.
• You have a previously configured Anthos installation with a running user cluster.
• You are currently running Splunk Enterprise (7.0 or later) on-premises in your organization.

If you're running Anthos on Google Cloud or other public clouds and want to connect to a Splunk instance running on-premises, you need to have hybrid connectivity between the two environments in place, as shown in the following diagram:

Note: If you want to deploy Splunk Enterprise on Anthos itself, see this blog series for instructions on deploying Splunk Enterprise on Kubernetes and Splunk Connect for Kubernetes.

Objectives

• Install and use Splunk Connect for Kubernetes to collect logs from Kubernetes applications running on Anthos on Google Cloud.
• Configure an HTTP Event Collector for Splunk Enterprise.
• Deploy Splunk Connect for Kubernetes in your Anthos cluster.
• Deploy a sample application.

Costs

This tutorial uses the following billable components of Google Cloud:

• Anthos

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

1. Sign in to your Google Account.

   If you don't already have one, sign up for a new account.

2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to the project selector page

3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

4. Make sure that you have Splunk deployed and configured. See Getting started with Splunk and deploy Splunk Enterprise.
5. Make sure that you have an Anthos user cluster deployed and registered in Google Cloud Console. This tutorial assumes there are three nodes in the cluster, but this is not mandatory. Follow Anthos documentation to deploy this Anthos component.
6. Ensure that there is network connectivity between your Anthos cluster and Splunk.

Configuring Splunk Enterprise

Set up indexes

Splunk Connect requires a minimum of two indexes for exporting data—one for events and one for metrics. In this tutorial, you also create a separate events index for objects, for a total of three indexes.

1. Log in to Splunk as an administrator.
2. Go to Settings > Indexes.
3. Click New Index.
   1. In the Index Name text box, enter events.
   2. Set Index Data Type to Events.
4. Click Save.
5. Click New Index.
   1. In the Index Name text box, enter objects.
   2. Set Index Data Type to Events.
6. Click Save.
7. Click New Index.
   1. In the Index Name text box, enter metrics.
   2. Set Index Data Type to Metrics.
8. Click Save.

Set up an HTTP Event Collector (HEC)

To export logs and metrics from Anthos to Splunk, you need to set up the Splunk HTTP Event Collector (HEC).

1. Go to Settings > Data inputs.
2. Click Global Settings.
   1. Set All Tokens to Enabled.
   2. Select Enable SSL.
   3. Click Save.
3. In the HTTP Event Collector dialog, select +Add New.
4. In the Name text box, enter anthos-hec, and then select Next.
5. In the Input Settings dialog, do the following:
   1. Set App Context to Search & Reporting (search).
   2. In the Select Allowed Indexes menu, select events, metrics, and objects.
   3. Click Next.

6. The Review page appears. Check that the page looks as follows:

   

7. Click Submit to create the HEC. Copy the resulting token value for use later.

8. To check connectivity, log in to a node in your Anthos cluster and send a test event to Splunk. Replace hec-token with the token value created earlier, and replace splunk-ip with the IP of your Splunk Enterprise server, as shown in the following code:

   curl -k http://splunk-ip:8088/services/collector/event \
   -H "Authorization: Splunk hec-token" -d '{"event": "hello splunk"}'
   

   If the request is successful, the server returns a response like the following:

   {"text":"Success","code":0}
   

Deploying Splunk Connect for Kubernetes

To deploy Splunk Connect for Kubernetes, you need to create a Splunk service account, install Helm, and deploy Splunk Connect.

Log in to the Anthos user cluster as an administrator

• From a shell in your Anthos administrator workstation, ensure that your kubectl tool is configured to point your Anthos cluster by default:

kubectl config current-context

• The preceding command returns your Anthos cluster. If it doesn't, link the master kubeconfig file to a local config file and try again:

mkdir .kube
ln -s /kubeconfig ~/.kube/config

Install Helm

You install Splunk Connect with the Helm package manager.

1. Install the Helm client and ensure that Helm is in your path:

   helm version
   

   The output is similar to the following, though the version and commit numbers might differ:

   version.BuildInfo{Version:"v3.2.3", GitCommit:"8f832046e258e2cb800894579b1b3b50c2d83492", GitTreeState:"clean", GoVersion:"go1.13.12"}
   

Deploy Splunk Connect

1. Create a Splunk namespace. This namespace is used to isolate Splunk Connect from the rest of your cluster:

   kubectl create namespace splunk
   

Note: If you intend to deploy Splunk App for Infrastructure (SAI), you can skip the rest of this section and follow the steps outlined later in this document, Deploying Splunk App for Infrastructure, to install Splunk Connect and SAI simultaneously.

1. Create a YAML file containing the Splunk Connect configuration for reading logs, objects, and metrics. Replace hec-token with the token value created earlier and replace splunk-ip with the IP of your Splunk Enterprise server:

   cat <<EOF >values.yaml
   global:
     splunk:
       hec:
         protocol: https
         insecureSSL: true
         token: hec-token
         host: splunk-ip
         port: 8088
         indexName: events
   
   splunk-kubernetes-logging:
     journalLogPath: /run/log/journal
     splunk:
       hec:
         indexName: events
   
   splunk-kubernetes-objects:
     objects:
       core:
         v1:
           - name: pods
           - name: namespaces
           - name: nodes
           - name: services
           - name: config_maps
           - name: persistent_volumes
           - name: service_accounts
           - name: persistent_volume_claims
           - name: resource_quotas
           - name: component_statuses
           - name: events
             mode: watch
       apps:
         v1:
           - name: deployments
           - name: daemon_sets
           - name: replica_sets
           - name: stateful_sets
     splunk:
       hec:
         indexName: objects
   
   splunk-kubernetes-metrics:
     kubernetes:
       insecureSSL: true
     splunk:
       hec:
         indexName: metrics
   EOF
   

   Note: In the preceding configuration, insecureSSL: true allows Splunk Connect to connect to Splunk Enterprise by using Splunk's built-in, self-signed certificates. If your Splunk Enterprise instance is already configured to use custom or CA-signed certificates for SSL, remove this line.

2. Install Splunk Connect for Kubernetes into the splunk namespace:

   helm install anthos-splunk -f values.yaml --namespace splunk \
   https://github.com/splunk/splunk-connect-for-kubernetes/releases/downlo
   ad/1.4.1/splunk-connect-for-kubernetes-1.4.1.tgz
   

   Note: This tutorial assumes Splunk Connect for Kubernetes version 1.4.1. For a different version, change the path above accordingly.

3. Check that Splunk Connect has been successfully started:

   kubectl -n splunk get pods
   

   You should see 3 logging pods, 3 metrics pods, 1 metrics aggregator pod, and 1 objects pod running, as follows:

   NAME                                                          READY   STATUS    RESTARTS   AGE
   anthos-splunk-splunk-kubernetes-logging-46srr                  1/1     Running   0          2m
   anthos-splunk-splunk-kubernetes-logging-gkgkm                  1/1     Running   0          2m
   anthos-splunk-splunk-kubernetes-logging-r52nz                  1/1     Running   0          2m
   anthos-splunk-splunk-kubernetes-metrics-agg-595946f874-vjzmb   1/1     Running   0          2m
   anthos-splunk-splunk-kubernetes-metrics-gbkb8                  1/1     Running   0          2m
   anthos-splunk-splunk-kubernetes-metrics-k6ksn                  1/1     Running   0          2m
   anthos-splunk-splunk-kubernetes-metrics-vg5wn                  1/1     Running   0          2m
   anthos-splunk-splunk-kubernetes-objects-7c5cd8cf7d-csvw6       1/1     Running   0          2m
   

4. Go to Settings > Indexes in the Splunk UI. If the connection is successful, the Latest Event column shows a recent time for the events, metrics, and objects indexes.

Troubleshooting and making changes

• If a pod isn't starting or data isn't coming through, inspect the pod's logs for errors:

  kubectl -n splunk logs pod-name
  

• If you need to update your config.yaml file and try again, run the following commands to redeploy Splunk Connect and recreate its pods:

  helm upgrade --namespace splunk -f values.yaml anthos-splunk \
  https://github.com/splunk/splunk-connect-for-kubernetes/releases/downlo
  ad/1.4.1/splunk-connect-for-kubernetes-1.4.1.tgz
  kubectl delete pods -n splunk --all
  

Deploying and testing a sample app

Next, you deploy a sample app and view its logs, metadata, and metrics in Splunk.

Deploy a sample app

1. Deploy a simple "Hello world" app on Anthos:

   kubectl create deployment hello-kubernetes --image=paulbouwer/hello-kubernetes:1.5
   

2. Expose the app on port 8080 and port-forward it so that it can be accessed locally:

   kubectl expose deployment hello-kubernetes --port=8080 --type=ClusterIP
   kubectl port-forward deployment/hello-kubernetes 8080:8080
   

3. Test the app by using the curl command-line tool:

   curl http://localhost:8080/
   

   If the request is successful, the server returns a response like the following:

     <!DOCTYPE html>
     <html>
     <head>
         <title>Hello Kubernetes!</title>
     [...]
   

View app logs

• Browse to Search in the Splunk Web UI and search for the sample app's logs in the events index:

  index=events sourcetype="kube:container:hello-kubernetes"
  

  This search returns log events from the app in reverse chronological order (newest first), starting with the test request and the entry indicating that the app is listening and ready for service, as shown in the following screenshot:

  

View app metadata

• In the Search pane, search for metadata objects for the sample app:

  index="objects" "hello-kubernetes"
  

  This search returns a list of metadata snapshots for various Kubernetes objects. For example, by selecting sourcetype="kube:objects:deployments" and expanding metadata and status, you can check the status of the hello-kubernetes deployment at a specific point in time, including the number of available replicas, as shown in the following screenshot:

  

  By default, metadata is polled every 15 minutes. This can be adjusted with the interval setting in the values.yaml file.

  The metadata also includes a stream of events from the Kubernetes API. The following search returns all events for the sample app:

  index="objects" sourcetype="kube:objects:events*" "hello-kubernetes"
  

View system and app metrics

1. Go to Metrics in the Splunk UI and enter cpu in the search box.

2. In the drop-down menu, select Metrics > kube > namespace > cpu > usage.

3. In the Analysis pane on the right, select Split by, and then select name. You'll see a graph of CPU usage categorized by namespace, as shown in the following screenshot:

   

Deploying Splunk App for Infrastructure

The Splunk App for Infrastructure (SAI) provides insight into the performance of Linux servers, Microsoft Windows servers, Kubernetes clusters, OpenShift containers, Docker containers, and Amazon EC2, ELB, and EBS instances. SAI uses metrics for performance monitoring, and it uses log data for deep understanding and troubleshooting of your server infrastructure.

For a simplified setup installation that is fully compatible with the features provided in the Splunk App for Infrastructure, install both the Splunk Add-on for Infrastructure and the Splunk App for Infrastructure in your Splunk environment. Then follow these steps for generating an installation script for generating an installation script that configures data delivery to Splunk.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the Cloud project

The easiest way to eliminate billing is to delete the Cloud project you created for the tutorial.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the resources

After completing this tutorial, run these commands to remove the created resources:

   helm delete --namespace splunk anthos-splunk
   kubectl delete deployment hello-kubernetes

What's next

• Learn more about Anthos:
  • Anthos documentation
  • Anthos clusters deployed on-premises documentation
• Learn more about Splunk products and solutions:
  • Splunk for containers
  • Deploying Splunk Enterprise on Google Cloud
  • Splunk App for Infrastructure (SAI)
  • Splunk Enterprise on Kubernetes
• Try out other Google Cloud features for yourself. Have a look at our tutorials.