Google Cloud Platform and Confluent partner to deliver a managed Apache Kafka service

One of the most common requests we get from customers migrating to GCP is whether we’ll offer a managed version of Apache Kafka. This comes as no surprise, as Kafka has become a leading open-source solution for event streaming and is increasingly the primary messaging platform for event-driven organizations. This prominent role brings with it the operational and engineering design challenges of high availability and replication, which in turn drives the aforementioned demand for a managed Kafka solution that removes these difficulties.

When it comes to stream ingestion, Google Cloud Pub/Sub is the answer for many GCP users. Yet, for many others, Kafka can make more sense. Some customers don’t want to rewrite their on-premises systems to GCP-native services such as Pub/Sub. Others have developed expertise, processes, and standards for running Kafka themselves. Still others have adopted an event sourcing model that is uniquely dependent on Kafka’s log data structure. Finally, many users want to maintain hybrid deployments, consistent across GCP and public/private clouds where Cloud Pub/Sub is not an option.

For these Kafka fans, we’re happy to announce a simple solution: Confluent Cloud on GCP. Confluent Cloud is a fully-managed streaming service based on Apache Kafka. Led by the creators of Kafka—Jay Kreps, Neha Narkhede and Jun Rao—Confluent provides enterprises with a real-time streaming platform built on a reliable, scalable ecosystem of products that place Kafka at their core. Further, Kafka benefits from a large community of developers working with and contributing to the open source project, and offers a broad range of connectors, plug-ins, monitoring tools, and configuration tools. Confluent on GCP links this Kafka ecosystem with GCP’s big data and machine learning services, and removes the burden of managing Kafka for developers looking to build streaming applications.

Confluent Cloud is particularly valuable to the designers of hybrid systems. It provides a persistent bridge between on-premises and public cloud data streams. Once on-premises data is streamed into Confluent Cloud, it becomes immediately accessible on GCP. And because Confluent Cloud is available across public clouds, it can also be used to build multi-cloud data pipelines. Confluent Cloud handles the burden of balancing and replicating data across clouds.

Confluent Cloud complements GCP’s unique stream analytics and data warehousing services. GCP’s stream data processing service, Cloud Dataflow service for Apache Beam, integrates natively with Kafka. Data from Kafka can flow into Google BigQuery, GCP’s cloud-scale data warehouse, as well as Google’s other analytics, machine learning, and serverless compute services. And Kafka developers have the option of using Kafka Streams and KSQL for real-time stream processing while staying within the Kafka ecosystem.

Confluent Cloud is available in two editions. Confluent Cloud Professional gives developers of new and pilot projects a self-serve tool to provision their Kafka environment. It enables users to start building apps in minutes through a self-service provisioning model. It offers the low latency of Kafka, automatic upgrades, and features that accelerate Kafka application development. Confluent Cloud Enterprise offers 99.95% uptime SLAs, 24x7 support with 1 hour response times, VPC peering, and unlimited scalability to meet the needs of all enterprise applications, from development to production. To learn more, read the Confluent blog or visit the Confluent Cloud website.