How to reduce costs with Managed Service for Apache Kafka: CUDs, compression and more

Running and managing Apache Kafka clusters can be costly as your business and streaming workloads grow. Today, we’re announcing the general availability of Managed Service for Apache Kafka committed use discounts (CUDs), which let you save up to 40% on compute costs. Read more to learn how to save money with CUDs and other cost saving features.

Google Cloud Managed Service for Apache Kafka is a fully managed, secure and highly scalable service for running open-source Apache Kafka on Google Cloud. This managed Kafka service reduces the cost of manual broker management by handling cluster creation with automatic broker sizing and rebalancing. It also comes with Cloud Monitoring, Cloud Logging, and Identity and Access Management (IAM), and creates all deployments to be highly available by default. 

Here are some tips on how to optimize your Managed Service for Apache Kafka costs through committed use discounts and other cost saving features.

Use committed use discounts to save on compute

With committed use discounts (CUDs) for Managed Service for Apache Kafka, you can save up to 40% on vCPU and RAM  by committing to using these resources in advance. You can apply the spend-based CUDs to save on steady-state compute usage across all Kafka clusters and projects that are under the same billing account. 

For a one-year commitment, you save 20% over on-demand pricing, and 40% for a three-year commitment. We recommend purchasing Managed Service for Apache Kafka CUDs when you have a predictable minimum compute cost that you can commit to for at least a year. 

You can buy CUDs now by going to the committed use discounts page in the Google Cloud console. Check out the CUD documentation and pricing page for more details.

Optimize inter-zone transfer costs

Data transfer between Kafka brokers and consumers in different zones can incur costs. Inter-zone data incurs transfer charges from Private Service Connect (PSC), while data transfer between brokers and consumers in the same zone do not incur this charge.

By default, consumers retrieve data directly from the leader replica. This setup incurs inter-zone data transfer costs from reading data across different zones. Starting from Apache Kafka version 2.4 (KIP-392), you can configure Kafka consumers to read messages from the replica in the same zone as your consumer. This feature is called “rack awareness”, where a rack refers to a cloud zone. This configuration helps to reduce data transfer costs as well as latency. 

To set up consumer rack awareness, you must configure consumers so that their client rack IDs match the cloud zone of where they are deployed. Each broker’s rack ID is set to its cloud zone. To find where the brokers are deployed, search the cluster logs for "broker.rack =". If one or more replicas exists with a matching rack ID, the algorithm chooses the most caught-up replica. The replica selector falls back to the replica leader if client rack ID is null or if there is no replica in the same zone as the consumer. You need to deploy at least one consumer in each zone that contains a broker and have enough replicas configured in your topic.

Turn on compression 

Enabling compression can reduce both your networking and storage costs, by reducing the size of your messages, which in turn reduces network traffic and latency while requiring less storage. You can enable compression at both the producer-level and broker-level. Compression is also available at the topic-level, though it is recommended to enable compression at the producer-level and broker-level so that the compression applies to all topics. 

Having a larger message batch size can help improve the effectiveness of compression. You can set compression to either gzip, lz4, or snappy (the zstd compression type is not yet available). If you enable compression for producers, by default, compression will be used for producer-to-broker communication, for storage on broker, and broker-to-consumer communication.

Finetune batch size

Experimenting with different batch sizes can help you save on CPU and network bandwidth while increasing throughput. Our internal benchmarks show that producer throughput can vary by a factor of 3 depending on the batch size used. With some finetuning, you can figure out which batch size works best for a given cluster configuration. When you are looking for the ideal batch size, you are trading batch size for latency. We recommend first setting the maximum tolerable wait time with linger.ms in the producer setting, then experimenting with different batch sizes to find the ideal one for your specific cluster configuration.

Reduce retention and storage in test environments

If you don’t need to retain your messages for an extended period of time, you can limit the retention period of your Kafka messages and hence lower the amount of data stored and its associated storage costs. For instance, if you do not need the message backlog, you can lower the retention time and size with log.retention.ms and log.retention.bytes. This causes messages to expire after the retention period or after hitting the maximum size of a partition. In test environments, you can also create topics with a replication factor of 2 and set the minimum number of insync replicas to 1, further reducing storage. However, this comes at the cost of reliability and availability. We don’t recommend doing this for production workloads, only for test environments.

Avoid message conversions 

If Kafka clients and brokers are using different versions of Kafka, they may not be aligned on the underlying binary message format. This misalignment requires message conversion, which introduces processing overhead and results in increased CPU and memory utilization for the cluster. To prevent message conversions and save costs, make sure that all Kafka clients (both producers and consumers) are using the same version of Kafka as the Kafka brokers. 

Get started today

There are lots of things you can do to keep your Kafka costs under control. To purchase CUDs, go to the committed use discounts page in the Google Cloud console or find more information in the CUDs documentation. Then, be sure to explore the best practices we’ve outlined above. To learn more about Managed Service for Apache Kafka, follow this quick start.