Kafka Plugin

Kafka is a distributed, partitioned, replicated commit log service. For more information visit kafka.apache.org.

Discovery

Even if the Kafka plugin is not configured, Stackdriver Monitoring will discover Kafka services running in your Cloud Platform project by:

  • searching instance names for kafka
  • checking for ports opened to 9092 via firewall rules

The services discovered are displayed on the Kafka Services page in the Resources menu. After you install the monitoring agent and configure the Kafka plugin on your instances, Stackdriver Monitoring populates the Kafka Services page with inventory and metrics.

Configuring Stackdriver Monitoring to monitor Kafka

After you install the monitoring agent, use the following instructions to configure the Kafka plugin on your instances. For instructions on upgrading from the legacy jmxtrans monitoring, see Upgrading from jmxtrans below.

Prerequisites

The Kafka plugin requires version 5.5.0-257 or later of the monitoring agent. To update the agent, see Upgrading the agent.

Enabling the Kafka monitoring plugin

Kafka is monitored via JMX. Stackdriver Monitoring supports monitoring Kafka version 0.8.2 and higher.

On your VM instance, download kafka-082.conf from the GitHub configuration repository and place it in the directory /opt/stackdriver/collectd/etc/collectd.d/:

(cd /opt/stackdriver/collectd/etc/collectd.d/ && curl -O https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/kafka-082.conf)

The downloaded plugin configuration file assumes that your Kafka server is configured to accept JMX connections on port 9999. If you have configured Kafka with a different JMX port, edit the file and follow the instructions to change the JMX port settings.

After adding the configuration file, restart the monitoring agent by running the following command:

sudo service stackdriver-agent restart

Information on other plugin configuration options can be found at collectd.org.

What is monitored

The Kafka plugin monitors the following metrics for Kafka server/broker:

  • Message In Rate
  • Bytes In/Out Rate
  • Request Rate
  • Log Flush Rate
  • ISR Expansion and Shrink Rate
  • Request times

Stackdriver Monitoring also monitors the following basic JVM metrics:

  • Active JVM Threads
  • JVM Heap memory usage
  • JVM Non-Heap memory usage
  • JVM Open File Descriptors
  • JVM Garbage Collection Count

Upgrading from jmxtrans

If you are currently running a VM with the legacy jmxtrans monitoring, you need to upgrade all of the Java-based plugins, namely Cassandra, HBase, Kafka, or Tomcat. After you've upgraded those plugins and restarted the stackdriver-agent service, you can shut down the jmxtrans process. You can then uninstall jmxtrans by removing the /mnt/jmxtrans tree and the /etc/init.d/jmxtrans script (if the script is present).

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Stackdriver Monitoring