Aiven: Building an open source database-as-a-service cloud where developers can thrive
About Aiven
Founded by developers, for developers, Aiven helps organizations to drive the continuous innovation needed to create data-intensive applications. Using open source technologies, Aiven’s all-in-one cloud data platform combines all the tools developers need to connect to the data services they use, on all major cloud providers.
Tell us your challenge. We're here to help.
Contact usOn a mission to empower developers with cloud databases that are frictionless and easy to set up, Aiven built a fully-managed, open source cloud data platform on Google Cloud.
Google Cloud results
- Addresses database requirements of 1,000 customers across more than 100 regions
- Handles any amount of data, enabling 40 million unique metric data points to be processed twice every minute
- Seamless, infinite scalability provides bandwidth and no bottlenecks
Supports 100,000 VMS hosted on Compute Engine
Managing data infrastructure steals time away from skilled developers, who would rather be focusing on building new features and updating code. When open source developers Oskari Saarenmaa, Hannu Valtonen, Mika Eloranta, and Heikki Nousiainen found themselves in this situation back in 2016, they decided to build their own solution. They called it Aiven.
"We run our business entirely on software, so having developer-friendly SDKs and APIs that make technical sense was a big draw for us. Besides, we had no doubts that we’d be able to scale as a business on Google Cloud."
—Heikki Nousiainen, Co-Founder and CTO, AivenAiven is a Database-as-a-Service (DaaS) platform that brings multiple popular open source services and tools, like Apache Kafka and OpenSearch, into one place where developers can deploy them across multiple cloud providers in minutes. With Aiven, companies can grow from prototyping to worldwide scale at greater velocity. To bring this vision to life as a SaaS offering, Nousiainen and the Aiven founding team reached out to Google Cloud.
"We noticed that cloud adoption was increasing but infrastructure solutions were either proprietary or difficult to translate into business results," explains Nousiainen, now Co-Founder and field CTO at Aiven. Knowing first-hand the problems developers face each day, the Aiven team combined their experience, knowledge, and belief in open source best practices to create an easy-to-use platform that gives developers everything they need to build their applications. When looking for a cloud platform that would help Aiven achieve its ambitions, Google Cloud was the obvious choice. "We run our business entirely on software, so having developer-friendly SDKs and APIs that make technical sense was a big draw for us," says Nousiainen. "We had no doubts that we’d be able to scale as a business on Google Cloud."
Selecting a scalable, reliable home for developers’ applications
To get the solution off the ground, Aiven decided to create and run virtual machines on Compute Engine. "Since day one, Compute Engine has been the brains of our operations," explains Nousiainen. "We’ve grown rapidly to serve more than 1,000 customers worldwide, and scaling was seamless on Compute Engine with pre-built and ready-to-go configurations. That’s where we implement all customer services and handle all of our resources running on Google Cloud to this day."
"With Compute Engine, we can divide virtual machines into availability zones to prevent failures, giving us the confidence to promise customers that we’ll keep their services running healthily at a 99.99% availability rate, at all times."
—Heikki Nousiainen, Co-Founder and CTO, AivenAiven uses Compute Engine to create virtual machines that support the fully-managed services run by customers on open source toolkits such as Apache Kafka, Apache OpenSearch, and Apache Cassandra. Each node required by the customers translates into a virtual machine managed by Aiven on Google Cloud. "This allows us to guarantee that resources we allocate to different customers can operate safely and in isolation from one another," explains Nousiainen. "With Compute Engine, we can divide virtual machines into availability zones to prevent failures, giving us the confidence to promise customers that we’ll keep their services running healthily at a 99.99% availability rate, at all times."
Making data infrastructure simple for developers
Aiven’s system generates a huge amount of data. With the amount of unaggregated data currently amounting to about 1TB in just one week. “We have tens of thousands of nodes running, and we collect thousands of metrics for each node. At any one time, we currently process 40 million unique metric data points, most of them twice every minute,” explains Nousiainen. Aiven needed a storage solution that could handle any amount of data, and that’s what it found with Cloud Storage.
Additionally, with more than 1,000 customers plus independent, self-service developers leveraging Aiven’s services, the company implements 100,000 service instances and delivers services across five different clouds. To manage and orchestrate all of this, Aiven runs its own control plane built on Google Cloud using open source. This control plane monitors services in real-time to ensure they are running smoothly across all the different clouds on which Aiven customers operate, while collecting metrics centrally so that Aiven can watch for anomalies, advise customers, and scale instances as needed.
"Our fleet of services includes managing, monitoring, calculating invoices, and using open source internally for storing logs, sending messages, configuring, and informing our services used by customers. All this, combined, keeps customer services up and running and this is all implemented on Compute Engine and Cloud Storage," explains Nousiainen. "That’s why Google Cloud is central to our business."
Based on their own expertise as developers working in high-scale systems, it was important for Nousiainen and his colleagues that the solution was built on a platform that also offered managed services and automation to support a smooth workflow. "The technical implementation always aimed at preventing bottlenecks, but being able to seamlessly scale wouldn’t have been possible without cloud resources. This goes from managing Apache Kafka and maintaining them in separate streams across clusters, to adding more compute resources to be able to serve all of our customers on-the-fly," says Nousiainen. "The strength of Google Cloud is that it gives us bandwidth and no bottlenecks."
Empowering developers with autonomy, productivity, and the ability to innovate
Aiven now has more than 500 employees, of which half are in technical roles, and the company is growing. The mission driving this growth is to help developers by tackling their common pain points through open source technologies. To that end, Aiven launched an OpenSource Program Office in 2021 to help foster open source adoption globally.
"We want to give back to the community," says Nousiainen, "so we’re continuously on the lookout for interesting open source technologies. We just added Apache Flink and ClickHouse to cater for developers’ streaming processing and analytics needs. It’s an interesting and motivating place to be in because we get to see a lot of success from our customers as they build on top of these technologies, using Apache Flink, ClickHouse, and Looker together to better understand their data."
Contributing towards open source projects and a more sustainable future
One of the ways in which Aiven measures success is by looking at developer productivity, both its own and that of its customers. "A productive developer is happier, and a happy developer is productive," says Nousiainen. Some of the developers Aiven works with support hundreds of teams within their own organizations, using Aiven to provide a standard and build the foundation for those teams to work efficiently.
"Google Cloud is very transparent on the carbon intensity of its different data centers, so not only are we a natural fit but also we have a shared mission."
—Heikki Nousiainen, Co-Founder and CTO, AivenNousiainen credits this accomplishment to standard APIs and tools that make it easy for people to share data within their organization and leverage that data for trying out new things. "In that sense, developer success is measured in productivity, but not only in that we must move quickly, but also in that we need the ability to experiment and do more with autonomy. Because that’s how we get to innovate," he explains. With this in mind, Aiven aims to make open source increasingly accessible for developers worldwide.
Aiven has recently launched its sustainability impact programme aiming to bring visibility to customers of the environmental impact of their services so they can select computing locations that reduce their carbon footprint. "Google Cloud is very transparent on the carbon intensity of its different data centers, so not only are we a natural fit but also we have a shared mission," says Nousiainen. "We’re excited to continue working together on this journey."
Tell us your challenge. We're here to help.
Contact us
About Aiven
Founded by developers, for developers, Aiven helps organizations to drive the continuous innovation needed to create data-intensive applications. Using open source technologies, Aiven’s all-in-one cloud data platform combines all the tools developers need to connect to the data services they use, on all major cloud providers.