Metrics are the Drivers of CTI Value

Written by: Gina Hill, John Doyle, Jason Atwell

Mandiant Threat Intelligence customers often ask how they can measure their cyber threat intelligence (CTI) capability to ensure they are delivering business value that is aligned to the organizations vision and strategy. Not much has been published by the CTI community on developing metrics to measure key performance indicators (KPIs) success rate and the implications thereof for making informed programmatic decisions for the future direction of a CTI program. Well-crafted KPIs should be centered around targets and tie-into business objectives, whereas metrics focus on supporting the KPI process. In this blog post, we will explore how metrics can be used to measure a CTI team's effectiveness using an approach identified by industry as “the balanced scorecard”. This will be the first blog post in a series aimed to help organizations develop CTI metrics, track metrics over time, and understand the actual costs required to run a CTI program.

Balanced scorecards are widely used in various organizations to benchmark, identify, improve, and control various functions and resulting outcomes. By adopting the balance scorecard approach for use with CTI programs, the goal is to provide achievable program milestones with associated metrics to ensure lasting program success, drive business value and ensure better alignment to business strategy and vision. We recommend that balanced scorecards are created and revised on an annual basis to align with annual resource planning and forecasting exercises. Mandiant has created a balanced scorecard approach for a CTI team aligned to three levels of effort (LoEs) and defines the level of work needed for each. Each LoE has underlining objectives, metrics, and target values a CTI program should focus on to drive value.

• Innovation & Learning
• Internal Processes
• Financial Requirements

Innovation & Learning

To grow and mature a CTI program, innovation and learning are critical components for success. A CTI capability that can tie-in innovation and learning into daily workflows can show value to the organization. Building or refining process and adding technology to streamline the flow of intelligence are examples of innovation. From there, the CTI program can produce quality intelligence in an adaptable fashion across multiple technology solutions to inform stakeholders. To successfully achieve this, the team needs to have the right composition of skills ranging from strategic understanding of the cyber threat landscape to operational tracking of intrusion sets to the ability to extract insights from malicious software, network traffic and host-based artifacts and curation of intelligence through technology (i.e., threat intelligence platform). To ensure members of the team embody the requisite skills to achieve mission goals, the organization should provide training opportunities on a regular basis.

We would expect to see the CTI program earmark a fixed number of hours per year per team member and corresponding costs to support innovation initiatives, training, conference attendance, and associated travel expenses mapped to how the opportunities will improve overall effectiveness in completing job-related tasks defined by the KPIs.

Internal Processes

The core competencies of a CTI team is to provide decision support to all stakeholders through the incorporation of four key goals: proactively identifying cyber threats across the organizations operating environment; increasing the efficiency of security resource allocation and implementation (e.g., personnel, technology & funding); dissemination of relevant, actionable, and timely threat intelligence to reduce cyber risk; and, facilitating high-fidelity, tailored communication of threats with heightened business context and impact to senior leaders.

Measuring program success is critical to the CTI team and helps qualify the program’s internal processes and evolving maturity. Crafting KPIs that focus on intelligence products that tie back to business requirements, afford greater actionability to stakeholders and reduce cyber risk will provide the greatest impact to the organization.

Financial Requirements

The financial dimension attempts to capture all costs associated with a CTI program beyond that of personnel related costs. Often this takes the form of tools, technology, and data sets to support developing, communicating, and distributing finished analytic products. This dimension also accounts for the cost of contracted vendor support and staff augmentation to fill key gaps in skills shortage or to keep pace with organizational demand for threat intelligence support. The specific component-level inputs for this dimension will vary from organization to organization.

The Balanced Scorecard

Pulling these various levels of efforts LoEs together into a unified table provides an impactful visual for executive leadership that represents the resource requirements necessary to operate a CTI program and its alignment with the organization’s broader cyber defense mission. We can add additional columns into this table for resource tracking throughout a given year or create more robust, independent tables to track progress over time. However, at its core, the balanced scorecard will examine the three aforementioned LoEs, corresponding objectives for each in the form of KPIs, related metrics, and the goal rate defined as a target value as illustrated here. The figures chosen are not representative and used to convey completeness in a balanced scorecard. It is important to note that target values for each LoE should be set by the organization based on realistic and achievable goals. Information derived from previous program performance metrics should be used as a baseline to set target values. After one year, the program should evaluate their achieved values and make adjustments based on the previous year’s outcomes.

Metrics Tracking

Tracking metrics over time provides a gauge for CTI and executive leadership to understand how they are progressing against intended outcomes at a particular interval and determine whether any additional resource requests or course corrections are required based on periodic reconciliation of forecasted metrics.

Stay tuned as Mandiant will be providing additional blog posts on recommended CTI program KPIs, sources of information and tools needed to capture metrics, as well as the monetary costs associated with the build-out of a program.

For more information on how Mandiant can assist in building our a CTI program, please visit Mandiant’s Threat Intelligence Transformation Services page.