Setting Product Performance Metrics in Web SaaS Product: Guidelines and Frameworks

Companies left and right are using the Agile approach for their business. In this regard, Agile metrics play an important role in tracking progress and ensuring that the implemented principles bring positive changes to the software development life cycle (SDLC).

It is a set of standardized measurements that help evaluate the performance of a team at various stages of the SDLC. In addition, metrics in Agile software development are essential for product quality control. A well-coordinated efficient team releases high-quality software that fully meets the requirements of the client, and does it on time. This is the essence of software development that brings satisfaction to customers and generates profit.

Product performance metrics and measurements help keep under control:

• Team performance.
• Product quality.
• Delivery time.
• Dependencies and more.

Metrics measurement in Agile is equally important to help the team hold accountability for the work done, keep the client informed, streamline processes, and better manage blockers and dependencies, with the ultimate goal of delivering a high-quality product and maximizing the client satisfaction.

Setting Product Performance Metrics

Net Promoter Score - measuring customer satisfaction

Net Promoter Score (NPS) is one of the most popular and trusted metrics for measuring customer satisfaction. Using these specific metrics, you can get a score between -100 and +100.

The lowest score of -100 indicates that none of your clients are referring to you in their circle. On the other hand, the highest amount of +100 means that all your customers are ready to recommend you to others.

For example, you get up to 100 responses on a Net Promoter Score survey where 20 responses range from 0 to 6, 20 responses range from 7 to 8, and the remaining 60 responses range from 9 to 10.

So, Net Promoter Score 60 (number of promoters) - 20 (number of detractors) = 40, ignoring passive responses.

Team Velocity - the value a team delivers to customers

Team velocity is a widely popular tool that measures story scores and quantifies the size and number of product features implemented by a development team in their previous sprints.

It offers a rare insight into the value a team delivers to their client over a period of time.

For example, if a team scored 20, 21, 20, and 19 points over the last four sprints, the average speed would be 20, which is the average of all those sprints.

The number 20 indicates that the command is expected to produce 20 results in the next iterations. Anything over 20 will be considered an improvement.

Release Burndown - track software progress

The Release Burndown Chart offers a wealth of information covering various aspects of software development.

The chart will help you understand the progress of the software development, the remaining features that are yet to be implemented, and the time period when you can expect completion.

The graph tracks the progress of development by indicating the speed, and by comparing the previous speed, calculates the time required to complete the project.

For example, the JIRA software shown in the diagram below indicates the capabilities of the release development diagram. It shows that in 4 sprints the team reduced the task from 43 to 26. So, according to these data, the program will be released in another 7 sprints.

Escaped Defects - software quality analysis

The Escaped Defects metric, as the name suggests, indicates the quality of the software. This indicator shows the number of problems and errors that the end-user encountered during the software production stage. The Escaped Defects metric shows these errors in a graph.

If the graph rises in ascending order, this indicates a poor quality process of this software. These days, most Product Managers use this metric to evaluate software quality.

For example, defects for the months of January, February, March, and April are 3, 6, 7, and 8, respectively, which means there is a software quality defect. However, if the graphs are in descending order, you are on the right track.

True Test Coverage - determines if test cases are covered by the application

Even today, most software development agencies use conventional test coverage product performance metrics that are not reliable enough to be trusted.

On the other hand, the evolution of true test coverage is significant, and gradually more and more people are accepting it. True test coverage is efficient and trustworthy, and measures unit tests exclusively.

This specific metric helps you know the exact number of codebases that have passed all test suites such as UI automation, end-to-end acceptance tests, manual tests, etc.

It demonstrates gaps in quality and available features that did not pass all sets of test coverage.

For example, there are 20 feature requirements and 100 test cases have been created for this. If 80 test cases are executed perfectly, test coverage is 80%.

Bug Rates - the average number of bugs that occur when features are deployed

The Bug Rate metric evaluates the efficiency and quality of your development team by measuring the bugs available in the software or when a particular feature is added. Typically, this metric calculates the average number of errors that occur when a feature is deployed.

If you want to achieve long-term success and increase brand value for your startup’s software product, you should use this metric.

The presence of a number of errors is enough to create a bad impression about your firm. This shows your carelessness and casts doubt on your software developers.

Also, with Bug Rates product performance metrics, you can get application health statistics and get insight into some of the most important aspects of the application such as response time, garbage collection, number of errors, transactions, CPU usage, disk space, number of threads, and the long list goes on.

For example, you have developed 25 test cases and during that time 5 bugs have been reported, then the bug rate equals 5/25 * 100 = 20%.

Tools/frameworks/strategies to define Product Metrics

Google HEART framework

This framework, developed by the Google Research Group, is designed to make it easy to measure key metrics around user happiness, engagement, adoption, retention, and task success.

The Google HEART platform uses a 2x2 table with key metric topics on the left and three fields for goals, signals, and metrics on the top.

The structure is intended to make it easier to set goals for each section.

Pirate Metrics

Pirate Metrics were invented by Dave McClure, a serial entrepreneur. The name comes from the abbreviation AARRR, which means:

• Acquisition - How do customers find out about your product? 
• Activation - How many customers sign up and use the product?
• Retention - How many active customers continue to use the product?
• Referral - Do your customers tell others about the product?
• Revenue - Does the business pay off the investment in this product?

Pirate Metrics work well for products and services with a clear conversion funnel.

For example, an e-commerce store or an online loan application (where conversion rate and user behavior are on top priority) is ideal for Pirate product metrics.

You can even use Pirate Metrics or any other form of the conversion funnel to organize your product backlog. This is a great way to use this data to drive prioritization.

Impact Mapping

Impact Mapping was introduced by Gojko Adzic in 2012. It is a visual and interactive way for teams to set strategic goals.

“Impact mapping is a lightweight collaborative planning method for teams that want to make a big impact with software products. It is based on user experience design, performance-based planning, and mind maps.” - impactmapping.org

Impact mapping is an excellent exercise for the whole team as well as for product owners to do with stakeholders.

It helps facilitate conversions about what results we would like to achieve and who we can target to achieve them. They also open the door to having more than one solution for the same impact or outcome.

How to analyze them and take action

We have seen that the subjective product performance metrics are far more important to business success than the traditional, objective measures of quality of the past. The effort required to find and measure relevant business metrics for functions is outweighed by the knowledge and learning opportunities gained. 

Business conditions and opportunities are constantly changing, so instead of a generalized formula to follow that can be fragile, our business strategists offer six rules of thumb or heuristics to help you stay focused and agile. Let them help you on your way to quality software and business success!

1. Metrics can't tell you the story; only a team can do that.
2. Comparing snowflakes is pointless.
3. You can measure almost everything, but you cannot pay attention to everything.
4. Business success measures drive software improvement, not the other way around.
5. Every feature adds value; either measure it or don't do it.
6. Measure only what matters now.

Wrap up

As you can imagine, there is a lot to measure in a team's work, but that doesn't necessarily mean that all those numbers are needed to help the team succeed. A general overview of Agile metrics touches on all possible features of the project and the team's composition. Of course, not everything applies specifically to your case. You really only need to track the metrics that are relevant to the team and the digital product development you're working on.