mabl: Setting a new pace for intelligent QA testing of web apps

About mabl

Launched in 2017, by the founders of Stackdriver (later acquired by Google), mabl is the only machine-learning-driven test automation service that automatically maintains tests and identifies regressions for faster completion of the software QA process and more reliable UI testing in software delivery pipelines.

Industries: Technology
Location: United States

mabl is reinventing the software test automation space using Google Cloud Platform and machine learning solutions for a QA service that vastly accelerates software release cycles for its clients.

Google Cloud Results

  • Speeds QA process for customers by running thousands of tests in parallel on demand
  • Shortens time gap between user feedback and machine learning model re-training, for faster product evolution
  • Enables engineering team to focus 95% of time on development and spend just 5% on underlying systems

Validating 100 million pages per month with Kubernetes

mabl solves the problem of QA as a bottleneck in the software development process. Web app development teams can subscribe to the service, create functional UI tests without any coding, run them incredibly fast, and quickly generate reports identifying potential issues to be addressed by the team. With mabl helping to accelerate the continuous delivery pipeline of production-ready feature updates, any software company can innovate more rapidly for its customers for higher customer satisfaction and additional revenue.

The founders of mabl, familiar with the power of cloud computing through their previous experience as creators of Stackdriver (later acquired by Google), realized the potential in applying machine learning and automation technologies to the software QA process. As the founders built out mabl's early engineering team and infrastructure, they empowered the team to complete an objective comprehensive evaluation of cloud and technology service providers.

The team wanted a fast time to launch for its new service, which promised to maximize development velocity for its users and minimize operational overhead. The team also sought to enable differentiating features for mabl that would give it a competitive advantage.

"Our costs completely correlate to the value we deliver to customers. Our growth in capacity and spend are directly in line with our revenue. The on-demand capacity of Google Cloud Platform reduces risk and strain on our resources."

Dan Belcher, Co-founder, mabl

Although some of mabl's engineers had previous experience and preference for other cloud providers prior to mabl, after a detailed analysis, the team concluded that Google Cloud Platform (GCP) was the best choice in a broad array of cloud and machine learning technologies.

For a rapidly growing company like mabl, flexible GCP pricing, coupled with dynamic scaling, are important. "Our costs completely correlate to the value we deliver to customers," says Dan Belcher, Co-founder of mabl. "Our growth in capacity and spend are directly in line with our revenue. The on-demand capacity of Google Cloud Platform reduces risk and strain on our resources."

Accelerating the QA process

mabl uses Google Kubernetes Engine (GKE) to run tests, spinning up a new GKE container instance in seconds. The containers execute comprehensive tests in minutes, with mabl capturing screenshots, DOM snapshots, logs, network data, and timing information along the way.

mabl allows thousands of such tests to run in parallel when a customer wants to test its latest application build with a comprehensive QA service on demand, leveraging the dynamic bursting capabilities of GCP to support massive changes in workload volumes.

mabl's customer test scheduler runs in App Engine, sending work to GKE. mabl's users have the ability to schedule tests based on a time of day schedule, whenever new application code is deployed, or on demand via mabl's user interface.

"Cloud Functions is the glue between all our subsystems," says Dan. "By not needing to deal with managing the application tier and offloading everything except core functions, we've saved months or years of engineering effort."

Once a test is scheduled, GKE provisions a cluster and the job creates a new container to receive a script with instructions for the test. Once the test is completed, the container is destroyed and the data is funneled to Cloud Pub/Sub for processing of test outputs.

"We're provisioning and de-provisioning new containers every second," says Dan. "It's less than ten seconds from making a call to scheduling and running a new journey. Executing in seconds is a big deal for us. This has allowed us to run millions of containerized tests in the last year."

"Google Kubernetes Engine manages a huge part of everything we do, but its operational burden is minimal. Our engineering team can spend 95 percent of its time on development, and only 5 percent managing the underlying systems."

Dan Belcher, Co-founder, mabl

More dev, less ops

Outside of GKE, mabl doesn't use any compute services. "We don't have to deal with managing hosts or provisioning VMs," says Dan. "We can scale horizontally and get very good container density. It's a lot more efficient for us than running on raw VMs on compute."

"Google Kubernetes Engine manages a huge part of everything we do, but its operational burden is minimal," says Dan. "Our engineering team can spend 95 percent of its time on development, and only 5 percent managing the underlying systems." As a result, mabl's team of a dozen engineers can easily release five major features in just a week's time.

Dan relates a comparison to his work at another company five years ago, running on a different cloud services provider, when more than 50 percent of engineering effort was spent on infrastructure operations management. "The difference we see now is amazing. It means our team can focus on solving customer problems, not dealing with the overhead of managing VMs," says Dan. "The velocity at which we can build and ship new capabilities for our users is incredible. Some examples include building out support for a wider variety of devices and browsers, and a constant stream of new functionalities to our Chrome extension, which is fundamental to the mabl framework."

Making data more accessible

At mabl, the team of data scientists doesn't want to deal with provisioning clusters of compute nodes and configuring software to build a data warehouse. "Because we can take advantage of all these serverless products, such as APIs, it makes so many capabilities much more accessible to our ML engineers and data scientists." says Dan.

Cloud Datalab is used by mabl's data scientists to test different statistical models and visualizations based on data stored in BigQuery. "Our data scientists can write data to BigQuery, run experiments, and use Cloud Datalab to test out different statistical models — without dedicated support from sys admins or engineers," says Dan. "The ease of use of Google Cloud Platform means more team members can focus on the value they're building for our customers, be it shipping new features or attending customer calls for direct feedback on the product."

Cloud Dataflow processes the disparate data types created by the testing so that machine learning models can be trained with this information. Without such a tool to handle the massive pipeline of data, a test that takes minutes with mabl could easily take days.

"The Google vision around machine learning is appealing because it's not based on black boxes. The flexibility and openness engenders trust with our engineers."

Dan Belcher, Co-founder, mabl

Dan's team uses Cloud Machine Learning Engine (Cloud ML Engine) to build distribution models that look for anomalies, using a package that builds on TensorFlow to model distributions. The team leverages both the training and the online prediction capabilities to manage all the training of its distribution models, without needing to spin up VMs. The systems call on information from Dataflow and uses a prediction API to determine what the models say about new, incoming data. This allows for better optimization of model training.

Prior to the release of Cloud ML Engine, mabl had to do model training using TensorFlow on its own machines. Now it is glad to have Cloud ML Engine as another managed service of Google Cloud Platform.

The mabl team appreciates the Google promotion of open source technologies. "The Google vision around machine learning is appealing because it's not based on black boxes," says Dan. "The flexibility and openness engenders trust with our engineers."

Effortless scaling with APIs

Cloud Datastore is another critical component of mabl's infrastructure, providing the primary persistent data store for nearly all its data, including raw test results, insights generated based on the tests, schedules, and any other data the mabl user interface may need to quickly retrieve for someone.

"At my last company, the discussions around capacity for our Apache Cassandra cluster were an everyday event," Dan says. "Concerns about how to scale the instance types and add more nodes to the ring were ongoing. We had 10 percent of our engineering capacity dedicated full time to managing that one subsystem. Today, Cloud Datastore gives us an API that takes care of all that, so scaling is effortless."

App Engine handles the cloud endpoints for mabl's APIs, taking the burden off mabl to host these itself. mabl uses Firebase hosting as the front end for static assets on its web app and Firebase Realtime Database for syncing data across all clients in real time. "The Firebase database is amazing," says Dan. "It allows mabl to provide a real-time presentation of data and status of tests throughout the system, which really differentiates our service. Our customers can see logs and screenshots from their tests as they're being executed in the cloud. It's a real wow-factor for our users."

The company relies heavily on Stackdriver for logging, monitoring, and error reporting. The error alerts are its first line of defense in understanding anomalies that occur in production, particularly after rolling out a new deployment.

More secure web apps

mabl's customers, along with mabl, value the inherent security features of GCP. "Our customers are much more confident in the ability of Google to secure the infrastructure and network compared to running servers in a co-location data center," says Dan.

Dan points to Cloud Key Management Service (KMS) as a particularly useful feature, allowing mabl users to store credentials with an encryption key specific to their accounts and the ability to rotate credentials and passwords easily. "It would have taken months of work for us to try to build that capability natively, but it took us just a couple of days with Cloud KMS," says Dan. "It's a powerful feature that improves security for our customers."

From Chrome a shiny future unfolds

While mabl tests on multiple browsers, its customers are focused on testing in Chrome, so mabl was designed as a Chrome first service. mabl users define test journeys via a Chrome extension. "Chrome is key to the framework for our product," says Dan. "It's a huge benefit that we can add features to our Chrome extension, and Chrome handles all the automatic updates."

In the future, mabl would like to integrate more closely with developer tools on GCP, such as making it easy for customers using App Engine as their primary application front end to run tests on their apps within the context of GCP. In addition, mabl anticipates leveraging the vast data generated from running tests and integrating with GCP for analysis and reporting on the test data.

mabl's plans include broadening its efforts in the spirit of making QA simpler for its clients. For example, it wants to use artificial intelligence to help customers more easily ask higher-level questions in natural language, such as "Is this deployment risky?" or "Why did this set of tests fail?"

Reflecting on the company's success, Dan marvels at how quickly and easily the company has ramped up. "I can't overstate the impact of GCP on our velocity and ability to focus on strategic priorities," Dan says. "The lack of stress over infrastructure issues is a really beautiful thing."

About mabl

Launched in 2017, by the founders of Stackdriver (later acquired by Google), mabl is the only machine-learning-driven test automation service that automatically maintains tests and identifies regressions for faster completion of the software QA process and more reliable UI testing in software delivery pipelines.

Industries: Technology
Location: United States