Kafka plugin

Kafka is a distributed, partitioned, replicated commit log service. For more information visit kafka.apache.org.

This functionality is only available for agents running on Linux. It is not available on Windows.


Even if the Kafka plugin is not configured, Cloud Monitoring discovers Kafka services running in your Google Cloud project by:

  • Searching instance names for kafka
  • Checking for ports opened to 9092 via firewall rules

The services discovered are displayed on the Kafka Services page in the Resources menu. After you install the Monitoring agent and configure the Kafka plugin on your instances, Monitoring populates the Kafka Services page with inventory and metrics.

Configuring Monitoring to monitor Kafka

After you install the Monitoring agent, use the following instructions to configure the Kafka plugin on your instances.


The Kafka plugin requires version 5.5.0-257 or later of the Monitoring agent. To update the agent, see Upgrading the agent.

Enabling the Kafka monitoring plugin

Kafka is monitored via JMX. Monitoring supports monitoring Kafka version 0.8.2 and higher.

On your VM instance, download kafka-082.conf from the GitHub configuration repository and place it in the directory /etc/stackdriver/collectd.d/:

(cd /etc/stackdriver/collectd.d/ && sudo curl -O https://raw.githubusercontent.com/Stackdriver/stackdriver-agent-service-configs/master/etc/collectd.d/kafka-082.conf)

The downloaded plugin configuration file assumes that your Kafka server is configured to accept JMX connections on port 9999. If you have configured Kafka with a different JMX port, as root, edit the file and follow the instructions to change the JMX port settings.

After adding the configuration file, restart the Monitoring agent by running the following command:

sudo service stackdriver-agent restart

For more information on the collectd GenericJMX plugin used to monitor Kafka, see Plugin:GenericJMX.

What is monitored

The full lists of metrics the Kafka plugin monitors are at see Agent metrics: kafka and Agent metrics: jvm.