DevOps tech: Test automation

The key to building quality into software is getting fast feedback on the impact of changes. Traditionally, teams used manual code inspection and testing to verify systems' correctness. These inspections and tests occurred after "dev complete," and had the following drawbacks:

  • Manual regression testing is time-consuming to execute and expensive to perform, which makes it a bottleneck in the process. Software can't be released frequently and developers can't get quick feedback.
  • Manual tests and inspections are not reliable, because people are poor at repetitive tasks like manual regression tests, and it is hard to predict the impact of changes on a complex software system through inspection.
  • For systems that evolve over time, keeping test documentation up to date requires a considerable effort.

To speed up feedback, the agile development community proposed a set of test automation techniques in the early 2000s. These techniques evolved and are now used in continuous delivery pipelines to provide quick developer feedback, reduce lead time for changes, reduce failure rate, and more.

How to implement test automation

To build quality into the software, you must continually run both automated and manual tests throughout the delivery process.

Automated tests include the following:

  • Unit tests. These typically test a single method, class, or function in isolation, providing assurance to developers that their code operates as designed. To ensure that the code is testable and tests are maintainable, write tests before the code, a technique also known as test-driven development (TDD).
  • Acceptance tests: These typically test the app as a whole to provide assurance that a higher level of functionality operates as designed and that regression errors have not been introduced. Example acceptance tests might check for the business acceptance criteria for a user story, the correctness of an API, and broken functionality that was previously operating correctly. Write these tests as part of the development process. No one should be able to declare their work "dev complete" unless automated acceptance tests are passing.

The following diagram, initially created by Brian Marick and later used as the heart of the book Agile Testing: A Practical Guide for Testers and Agile Teams, shows the types of automated and manual tests to run.


The automated tests highlighted in the preceding diagram fit in a continuous delivery deployment pipeline. In such pipelines, every change runs a build that creates software packages, executes unit tests, and possibly performs other checks, such as static analysis. After these packages pass the first stage, more comprehensive automated acceptance tests, and likely some nonfunctional tests, run against automatically deployed running software. Any build that passes the acceptance stage is then typically made available for manual exploration and usability testing. Finally, if no errors are found in these manual steps, the app is considered releasable. That is, it's a business decision to deploy it, or not deploy it, in production.

Continuous deployment pipeline execution ensures quick feedback for developers, a short lead time from check-in to release, and a low error rate in production environments. Developers have most of their work validated in a matter of minutes, instead of days or weeks, so they can fix bugs as soon as they are introduced.

The following diagram shows an example of a simple linear deployment pipeline. In this example, red represents negative feedback, either tests or checks that have failed. Green represents positive feedback, meaning all checks have passed.


In the deployment pipeline pattern, every change creates a release candidate and the quick feedback loop helps to catch problems as early in the process as possible. When a package reaches the end of the pipeline and the team still doesn't feel comfortable with releasing it, or if they discover defects in production, the pipeline must be improved, perhaps by adding or updating tests.

Common pitfalls

  • Not having developers involved in testing. Developers must be involved in creating and maintaining suites of automated tests. When other groups own the test automation, two problems often arise:

    • Test suites are frequently in a broken state. Code changes might require tests to be updated. If developers are not responsible for test automation, the build pipeline stays broken until the responsible team fixes the tests.
    • Developers often generate code that is hard to test. Developers tend to solve the problem they are given without thinking about how it will be tested. This can lead to either poorly designed test suites or to the test automation team refactoring a lot of code.

    This doesn't mean that you should get rid of testers or QA teams. Testers have a unique perspective on the system because they understand how users interact with it. It's a best practice to pair testers with developers to help testers create and evolve the suites of automated tests. This way, developers learn the testers' perspective, and testers can learn about automation. Testers should also perform exploratory testing as part of the deployment pipeline.

  • Having the wrong proportion of unit and acceptance tests. A specific design goal of an automated test suite is to find errors as early as possible. This is why faster-running unit tests run before slower-running acceptance tests, and both are run before any manual testing.

    You should find errors with the fastest test category as possible. When you find an error in an acceptance test or during exploratory testing, add a unit test to make sure this error is caught faster, earlier, and cheaper next time. Mike Cohn described the ideal test automation pyramid, shown in the following diagram, where most of the errors are caught using unit testing.


  • Failing to curate your test suites. For example:

    • If every time you change your code you must also change multiple unit tests, you're probably over-relying on mocking, or failing to prune your unit test suite.
    • Acceptance test suites should typically represent real end-to-end user journeys through the system, rather than being collections of automated acceptance criteria. For more information on this, see the video Setting a Foundation For Successful Test Automation by Angie Jones.
    • Keep your test suites well-factored. If every change to your UI causes multiple acceptance tests to fail, use the page object pattern to decouple your tests from the system under test.

Ways to improve test automation

If you have don't have enough test automation, get started by building a skeleton deployment pipeline. For example, create a single unit test, a single acceptance test, and an automated deployment script that stands up an exploratory testing environment, and thread them together. Then incrementally increase test coverage and extend your deployment pipeline as your product or service evolves.

If you're already working on a brownfield system, follow the guidance in this article, but don't stop to retrofit a comprehensive suite of automated tests. Instead, write a small number of acceptance tests for the high-value functionality. Then, make sure you require developers to write unit and acceptance tests for any new functionality, and any functionality you are changing. Consider using TDD to improve the quality and maintainability of both main and test code, and finally, ensure that when your acceptance tests break, you write unit tests to discover the defect faster in the future.

If you have a test suite that is expensive to maintain and unreliable, don't be afraid to prune it down. A test suite of ten tests that is reliable, fast, and trustworthy is much better than a test suite of hundreds of tests that is hard to maintain and that nobody trusts.

Finally, ensure that testers and developers work together as part of the development process, rather than starting testing only after "dev complete". Physically pairing developers and testers for programming on building, triaging, and maintaining acceptance tests is ideal. If you can't do that, try using screen sharing tools.

Ways to measure test automation

You can measure the results of test automation in your environment by doing the following:

Factor to test What to measure Goal
Writers of acceptance and unit tests. Percentage of tests written by developers, testers, and any other group in your company. Primary authors and maintainers of acceptance tests are developers.
Number of bugs found in exploratory testing and in production. Change in proportion of bugs found over time. More bugs are found in "cheaper" test phases, and you add automated tests for the bugs you find during exploratory testing and production.
Time spent fixing acceptance test failures. Change in time spent fixing test failures over time. (It should reduce.) Developers can easily fix acceptance test failures.
Automated tests are meaningful. Track the quantity of automated test failures that represent a real defect and the quantity which were poorly coded. Test failures always indicate a real defect in the product.
Automated tests run on delivery pipeline. Check (yes/no) whether all test suites run in every pipeline trigger. Automated tests are run as part of the main pipeline and workflow.

What's next

  • For links to other articles and resources, see the DevOps page.
  • Learn how to continuously build, test, and deploy your system using Cloud Build.
  • Learn how to monitor your system and tests using Cloud Monitoring.