2022 State of DevOps Report data deep dive: Documentation is like sunshine

Editor's note: Every year, Google Cloud's DevOps Research and Assessment (DORA) team seeks to gain a better understanding of the software delivery and organizational performance through the annual Accelerate State of DevOps reports. In this series, we’re taking a deeper dive into the research and exploring some interesting findings that we didn’t cover in the 2022 State of DevOps report.   

Internal documentation is a fundamental part of software development. For the past two years, the DORA team has looked at the quality of internal documentation. From our analysis, we’ve found a clear link between documentation quality and organizational performance — the organization’s ability to meet their performance and profitability goals. Just like a plant needs water and sunshine to thrive, an organization with good technical practices and documentation is significantly more likely to meet or exceed their performance goals.

Documentation is highly technical work, and our findings show just how much this work pays off.  In this post we’ll discuss:

• How we define and measure documentation quality

• Documentation’s impact on technical capabilities and organizational performance

• Practices that you can implement on your team to increase your documentation quality

How do we measure documentation quality?

We don’t have units like lumens or watts to work with, so we had to design a way to define and measure documentation quality.

For our research, we used a set of eight metrics that assess documentation attributes like clarity, findability, and reliability. You can review the questions from 2022 for the statements we used to assess these attributes.

For each respondent, we aggregated the responses and then expressed them as a number between zero and one to give a standardized score. Once we had this score, we added it to our model to find out how it interacts with the rest of the system.

Documentation is like sunshine

Similar to our research from 2021, we see documentation quality driving the implementation of every single  technical practice we studied. Our model from 2022 shows documentation underpinning these technical capabilities.

What may be more exciting, however, is how documentation quality interacts with these capabilities to impact organizational performance.

Let's take a look at this effect. In the following graph, we visualize how continuous integration and documentation quality interact to impact organizational performance.

Here we are looking at three sets of simulated predictions based on our data. Each set has 4000 simulated predictions to help convey the degree of uncertainty (or certainty) present in our data.

• The top set, shown in green, is above average documentation quality (one standard deviation above average)

• The middle set, shown in yellow, is average documentation quality

• The bottom set, shown in red, is below average documentation quality (one standard deviation below average)

Along the x-axis is the implementation of continuous integration practices. As this technical practice is implemented, it has a positive impact on organizational performance. This impact is amplified as the quality of documentation increases.

In the presence of sunshine, a watered plant grows and thrives. And in the presence of quality documentation, technical practices help elevate an organization’s performance, from profits to customer satisfaction. 

This interaction and impact applies to every single technical capability we studied in 2022. In the table below, check out the lift to organizational performance as various technical capabilities are implemented. The lift is significantly amplified for teams with above-average documentation quality.

How does documentation have this impact?

We can speculate about different things that may be at play. For instance, there is an argument that the act of writing is transformative, and so writing documentation increases the technical quality of the work.

Studies, including the 2019 State of DevOps Report, have found a link between productivity and access to internal knowledge sources such as documentation. Teams with poor quality documentation who are successfully implementing technical practices may be doing so with a lot of effort and less overall productivity.

Maybe what we’re seeing is that teams who have implemented a capability while working with poor quality documentation are isolated in this success. They may be surrounded by others who are struggling. In contrast, technical success in the presence of quality documentation may mean a similar level of technical success both throughout the organization and through time.

A combination of all of these factors and others may be what is causing such large effects from documentation. What we can say for sure is that documentation quality has a significant impact on the successful implementation of technical practices. And quality documentation significantly amplifies the positive impact these technical practices have on organizational performance.

How do you create quality documentation?

All plants need sunlight to grow; luckily, the sun is always there, up in the sky. But while documentation is like sunlight, it isn’t like the sun. Documentation needs to be actively created and maintained, which takes work.

For guidance from our research, you can review the 2021 State of DevOps Report for practices that you can implement to improve documentation quality.

This research doesn’t investigate individual resources, but there are plenty out there to help you create useful documentation. In your own organization, you may have documentation guidelines and training. You may also have technical writers or other documentation champions to work with. Here at Google, our documentation resources include: 

• Technical Writing Courses for software engineers

• The Google developer documentation style guide

• The documentation chapter of Software Engineering at Google

If you’re lucky enough to work in an environment with plenty of quality documentation, it’s because you or people in your organization create that content. But if your organization has poor quality documentation, there’s good news. By implementing the practices linked above, you can improve your documentation quality and make it shine.

Tell us all about your documentation quality and other DevOps practices by taking the 2023 Accelerate State of DevOps Survey! The survey takes just about 15 minutes and will remain open until midnight PDT on June 30, 2023.