mabl: Setting a new pace for intelligent QA testing of web apps
About mabl
Launched in 2017, by the founders of Stackdriver (later acquired by Google), mabl is the only machine-learning-driven test automation service that automatically maintains tests and identifies regressions for faster completion of the software QA process and more reliable UI testing in software delivery pipelines.
Tell us your challenge. We're here to help.
Contact usmabl is reinventing the software test automation space using Google Cloud and machine learning solutions for a QA service that vastly accelerates software release cycles for its clients.
Google Cloud results
- Speeds QA process for customers by running thousands of tests in parallel on demand
- Shortens time gap between user feedback and machine learning model re-training, for faster product evolution
- Enables engineering team to focus 95% of time on development and spend just 5% on underlying systems
Validating 100 million pages per month with Kubernetes
mabl solves the problem of QA as a bottleneck in the software development process. Web app development teams can subscribe to the service, create functional UI tests without any coding, run them incredibly fast, and quickly generate reports identifying potential issues to be addressed by the team. With mabl helping to accelerate the continuous delivery pipeline of production-ready feature updates, any software company can innovate more rapidly for its customers for higher customer satisfaction and additional revenue.
The founders of mabl, familiar with the power of cloud computing through their previous experience as creators of Stackdriver (later acquired by Google), realized the potential in applying machine learning and automation technologies to the software QA process. As the founders built out mabl's early engineering team and infrastructure, they empowered the team to complete an objective comprehensive evaluation of cloud and technology service providers.
The team wanted a fast time to launch for its new service, which promised to maximize development velocity for its users and minimize operational overhead. The team also sought to enable differentiating features for mabl that would give it a competitive advantage.
"Our costs completely correlate to the value we deliver to customers. Our growth in capacity and spend are directly in line with our revenue. The on-demand capacity of Google Cloud reduces risk and strain on our resources."
—Dan Belcher, Co-founder, mablAlthough some of mabl's engineers had previous experience and preference for other cloud providers prior to mabl, after a detailed analysis, the team concluded that Google Cloud was the best choice in a broad array of cloud and machine learning technologies.
For a rapidly growing company like mabl, flexible Google Cloud pricing, coupled with dynamic scaling, are important. "Our costs completely correlate to the value we deliver to customers," says Dan Belcher, Co-founder of mabl. "Our growth in capacity and spend are directly in line with our revenue. The on-demand capacity of Google Cloud reduces risk and strain on our resources."
Accelerating the QA process
mabl uses Google Kubernetes Engine (GKE) to run tests, spinning up a new GKE container instance in seconds. The containers execute comprehensive tests in minutes, with mabl capturing screenshots, DOM snapshots, logs, network data, and timing information along the way.
mabl allows thousands of such tests to run in parallel when a customer wants to test its latest application build with a comprehensive QA service on demand, leveraging the dynamic bursting capabilities of Google Cloud to support massive changes in workload volumes.
mabl's customer test scheduler runs in App Engine, sending work to GKE. mabl's users have the ability to schedule tests based on a time of day schedule, whenever new application code is deployed, or on demand via mabl's user interface.
"Cloud Functions is the glue between all our subsystems," says Dan. "By not needing to deal with managing the application tier and offloading everything except core functions, we've saved months or years of engineering effort."
Once a test is scheduled, GKE provisions a cluster and the job creates a new container to receive a script with instructions for the test. Once the test is completed, the container is destroyed and the data is funneled to Cloud Pub/Sub for processing of test outputs.
"We're provisioning and de-provisioning new containers every second," says Dan. "It's less than ten seconds from making a call to scheduling and running a new journey. Executing in seconds is a big deal for us. This has allowed us to run millions of containerized tests in the last year."
"Google Kubernetes Engine manages a huge part of everything we do, but its operational burden is minimal. Our engineering team can spend 95 percent of its time on development, and only 5 percent managing the underlying systems."
—Dan Belcher, Co-founder, mablMore dev, less ops
Outside of GKE, mabl doesn't use any compute services. "We don't have to deal with managing hosts or provisioning VMs," says Dan. "We can scale horizontally and get very good container density. It's a lot more efficient for us than running on raw VMs on compute."
"Google Kubernetes Engine manages a huge part of everything we do, but its operational burden is minimal," says Dan. "Our engineering team can spend 95 percent of its time on development, and only 5 percent managing the underlying systems." As a result, mabl's team of a dozen engineers can easily release five major features in just a week's time.
Dan relates a comparison to his work at another company five years ago, running on a different cloud services provider, when more than 50 percent of engineering effort was spent on infrastructure operations management. "The difference we see now is amazing. It means our team can focus on solving customer problems, not dealing with the overhead of managing VMs," says Dan. "The velocity at which we can build and ship new capabilities for our users is incredible. Some examples include building out support for a wider variety of devices and browsers, and a constant stream of new functionalities to our Chrome extension, which is fundamental to the mabl framework."
Making data more accessible
At mabl, the team of data scientists doesn't want to deal with provisioning clusters of compute nodes and configuring software to build a data warehouse. "Because we can take advantage of all these serverless products, such as APIs, it makes so many capabilities much more accessible to our ML engineers and data scientists." says Dan.
Cloud Datalab is used by mabl's data scientists to test different statistical models and visualizations based on data stored in BigQuery. "Our data scientists can write data to BigQuery, run experiments, and use Cloud Datalab to test out different statistical models — without dedicated support from sys admins or engineers," says Dan. "The ease of use of Google Cloud means more team members can focus on the value they're building for our customers, be it shipping new features or attending customer calls for direct feedback on the product."
Cloud Dataflow processes the disparate data types created by the testing so that machine learning models can be trained with this information. Without such a tool to handle the massive pipeline of data, a test that takes minutes with mabl could easily take days.
"The Google vision around machine learning is appealing because it's not based on black boxes. The flexibility and openness engenders trust with our engineers."
—Dan Belcher, Co-founder, mablDan's team uses Cloud Machine Learning Engine (Cloud ML Engine) to build distribution models that look for anomalies, using a package that builds on TensorFlow to model distributions. The team leverages both the training and the online prediction capabilities to manage all the training of its distribution models, without needing to spin up VMs. The systems call on information from Dataflow and uses a prediction API to determine what the models say about new, incoming data. This allows for better optimization of model training.
Prior to the release of Cloud ML Engine, mabl had to do model training using TensorFlow on its own machines. Now it is glad to have Cloud ML Engine as another managed service of Google Cloud.
The mabl team appreciates the Google promotion of open source technologies. "The Google vision around machine learning is appealing because it's not based on black boxes," says Dan. "The flexibility and openness engenders trust with our engineers."
Effortless scaling with APIs
Cloud Datastore is another critical component of mabl's infrastructure, providing the primary persistent data store for nearly all its data, including raw test results, insights generated based on the tests, schedules, and any other data the mabl user interface may need to quickly retrieve for someone.
"At my last company, the discussions around capacity for our Apache Cassandra cluster were an everyday event," Dan says. "Concerns about how to scale the instance types and add more nodes to the ring were ongoing. We had 10 percent of our engineering capacity dedicated full time to managing that one subsystem. Today, Cloud Datastore gives us an API that takes care of all that, so scaling is effortless."
App Engine handles the cloud endpoints for mabl's APIs, taking the burden off mabl to host these itself. mabl uses Firebase hosting as the front end for static assets on its web app and Firebase Realtime Database for syncing data across all clients in real time. "The Firebase database is amazing," says Dan. "It allows mabl to provide a real-time presentation of data and status of tests throughout the system, which really differentiates our service. Our customers can see logs and screenshots from their tests as they're being executed in the cloud. It's a real wow-factor for our users."
The company relies heavily on Stackdriver for logging, monitoring, and error reporting. The error alerts are its first line of defense in understanding anomalies that occur in production, particularly after rolling out a new deployment.
More secure web apps
mabl's customers, along with mabl, value the inherent security features of Google Cloud. "Our customers are much more confident in the ability of Google to secure the infrastructure and network compared to running servers in a co-location data center," says Dan.
Dan points to Cloud Key Management Service (KMS) as a particularly useful feature, allowing mabl users to store credentials with an encryption key specific to their accounts and the ability to rotate credentials and passwords easily. "It would have taken months of work for us to try to build that capability natively, but it took us just a couple of days with Cloud KMS," says Dan. "It's a powerful feature that improves security for our customers."
From Chrome a shiny future unfolds
While mabl tests on multiple browsers, its customers are focused on testing in Chrome, so mabl was designed as a Chrome first service. mabl users define test journeys via a Chrome extension. "Chrome is key to the framework for our product," says Dan. "It's a huge benefit that we can add features to our Chrome extension, and Chrome handles all the automatic updates."
In the future, mabl would like to integrate more closely with developer tools on Google Cloud, such as making it easy for customers using App Engine as their primary application front end to run tests on their apps within the context of Google Cloud. In addition, mabl anticipates leveraging the vast data generated from running tests and integrating with Google Cloud for analysis and reporting on the test data.
mabl's plans include broadening its efforts in the spirit of making QA simpler for its clients. For example, it wants to use artificial intelligence to help customers more easily ask higher-level questions in natural language, such as "Is this deployment risky?" or "Why did this set of tests fail?"
Reflecting on the company's success, Dan marvels at how quickly and easily the company has ramped up. "I can't overstate the impact of Google Cloud on our velocity and ability to focus on strategic priorities," Dan says. "The lack of stress over infrastructure issues is a really beautiful thing."
Tell us your challenge. We're here to help.
Contact us
About mabl
Launched in 2017, by the founders of Stackdriver (later acquired by Google), mabl is the only machine-learning-driven test automation service that automatically maintains tests and identifies regressions for faster completion of the software QA process and more reliable UI testing in software delivery pipelines.