Most organizations I deal with think the more metrics they have, the better off they are. To the contrary, collecting metrics data takes time.  If the metrics you're reporting on are not valuable, then that time you spent collecting data is all wasted time.  So, the following is not intended to be an exhaustible list of indicators or instructions. Rather, the intent is to provide context on how anyone can glean information from metrics to demonstrate results toward better business outcomes.

This post focuses on an organization's desired to have its Scrum Delivery Team predictably meeting its release or milestone commitments.

I will lay out the process in 5 step.

1. Know what outcomes look like.

2. Know how to demonstrate key results toward an outcome.

3. Know what metrics should be used to measure key results.

4. Know where metrics data will originate from.

5. Know collection methods and frequency of the metrics data.

Theory: Monitoring Goals

A journal article* from 2016 supports the suggestion that monitoring goal progress is a crucial process between setting and attaining the goal.  Additionally, the article* stated that progress monitoring had a larger effects on goal attainment when the outcomes were reported or made publicly available, and when the information was physically recorded.

I believe leading indicators are the mechanism by which we monitor goal progress and reviewing that progress frequently by way of dashboards provides the larger effects of goal attainment.

*Psychological Bulletin, Vol 142(2), Feb 2016, 198-229

Assumptions

For this post, I will assume you are already familiar with LeadingAgile's Compass. Comparing the planning characteristics of a company with the planning expectations and needs of the client allows you to put your company, or your division, or even just your product area into one of four quadrants. The four quadrants will help you understand more clearly some of the challenges you might experience meeting customer expectations, why you are struggling or might struggle to adopt agile, and how to talk about the steps necessary to safely and pragmatically move your organization forward.

For this blog post, I am focusing on organizations that are actually in alignment with their customers expectations, but may struggle making and meeting commitments due to lack of clarity around the requirements, poor estimation, and extreme variability in the rate in which individuals can actually complete work against the estimates. These organizations often have highly structured PMO organizations, very disciplined governance, long-term planning and tracking processes, but still struggle to make and meet commitments on a regular basis.

Understand Goals and Outcomes of Going Agile

Your journey toward greater business agility starts by identifying what outcomes are most important to your company’s success. This knowledge helps you lay a foundation for making decisions about how to tailor your approach to measurably show progress toward your critical business objectives.  For this blog post, our goal is Greater Predictability.

Step One: Define the Desired Outcome

In our example, our desired outcome is to have teams plan, coordinate, and deliver predictably enough to meet a release level commitment.  

But what do we need for this outcome?

Backlogs

Backlog items need to be appropriately sized. They need to be ordered and prioritized to capture work needed to develop product or deliverables.

Stable Team

A stable team has everyone and everything (skill sets, tools, etc.) needed to deliver working, tested, documented, deployable product.

Working Tested Product Increment

This means deliverables meet defined acceptance criteria, have been reviewed and approved by product owner/ stakeholders, have been tested, and are shippable.

Step Two: Align Key Results to Desired Outcomes

For our example, I will list two key results:

1. The team delivers the committed functionality each sprint.

2. There is nothing preventing the team from meeting its commitments.

Step Three: Identify Metrics to Measure the Key Results

1. The team delivers the committed functionality each sprint.

   1. Story Point Completion Percentage

   2. User Story Completion Percentage

   3. Velocity Variance

2. There is nothing preventing the team from meeting its commitments.

   1. Team Stability

   2. Depth of Ready Backlog

   3. Outstanding Blockers

The key results are also known as lagging Indicators.  The metrics to measure them are known as leading indicators.  You can read more about leading and lagging indicators here.  The important distinction is that a lagging indicator is one that usually follows an event. You can use the leading indicators to make changes to your behavior or environment, resulting in more positive key results and better outcomes.

Step Four: Know Where to Get The Data

Any of the major Agile ALM tools will provide leading indicators.  You just need to know what is a meaningful indicator. Atlassian JIRA, CA Agile Central, VersionOne, VSTS... They all work. I also don't discount the value of physical boards and spreadsheets.  But since most of the people I'm working with are struggling with many teams working at once, an ALM tool can pay for itself through less duplication of effort and higher quality of data.

Step Five: Collection Method and Frequency 

Let's use Story Point Completion Percentage as an example leading indicator.  Capture key information about the metric, to ensure there is a shared understanding.  Metric description, how its calculated, when to measure, acceptable thresholds, and the source location are all critical for us to have a share understanding.

An example would looks like this:

Description: 

Story Point Completion Percentage measures Delivery Team predictability.  It is strongly correlated to the team’s stability and governance ability to manage dependencies.  

Metrics Calculation

Total Story Points (Accepted) / Total Story Points (Committed)

When to Measure

Committed Story points are calculated at Sprint Planning; Accepted at Sprint end.

Thresholds

>=90% (Green) | 80-89% (Yellow) | < 80% (Red)

ALM tool (CA Agile Central) Source

I have documented collection methods in all of the major ALM tools. I'm just going to use CA Agile Central for our example.  Within an CA Agile Central active project, navigate to the Iteration Status page.  One of the helpful elements of this page is the ability to see if the team has committed to an appropriate level at Sprint Planning (Planned Velocity graph) and how they are progressing toward the end of their Sprint (Accepted Velocity graph).  Point Completion Ratio is calculated by using the Accepted Velocity graph. (N of N points)

Summary

As noted early, leading indicators are the mechanism by which we monitor goal progress and by reviewing that progress frequently via dashboards, we increase the odds of reaching the goal. Use dashboards (with meaningful metrics) to monitor your progress.