We develop products for large companies and create strategies for global dominance. To succeed in both directions, we have to work a lot with metrics. We figure out which metrics are relevant to the product, where its strengths are, and what needs to be fixed. Our main tool is the "Metrics Pyramid" framework. In this article, we will explain why we love it and why you will love it too. If you are a product manager or analyst, then this text is definitely for you.
When we want to make a product successful, the first thing we need to do is understand the criteria for success. In our case, these criteria will be the values of metrics. The right development is the right metrics: relevant, comparable, understandable, measurable, and changeable.
We monitor the metrics, manage their value, and ultimately it affects the business indicators. Sometimes the correlation is direct, sometimes not so much, but it is always there. Therefore, it is important to remember exactly those metrics in which the value for the user consists. The light didn't converge on Retention and LTV alone.
To find out which metrics are relevant to the product, the "Metrics Pyramid" framework is most suitable. It's a cool tool, often it only lives in the heads of product owners and is not accessible to the product team. At the same time, there is no universal metrics pyramid, everything always depends on the business and the product.
Metrics Pyramid and Metrics Hierarchy
The Metrics Pyramid and Metrics Hierarchy are often confused. They indeed resemble each other: both are hierarchical models. But we will focus on their differences here.
Example of a metric hierarchy
In the "Metrics Hierarchy" framework, we break down the key success indicator into metrics that directly affect it. These, in turn, are broken down into even more specific metrics, and so on. The hierarchy helps to systematize metrics, identify dependencies, and understand which metric should be actively worked on to achieve the most significant result.
As a result, we:
- understand how changes in processes affect the improvement of product indicators (dynamically, including the long-term effect);
- know which indicators may worsen with the improvement of the selected metric;
- see order in the backlog and simplification in task prioritization.
It is clear that the metrics hierarchy is useful. But in this framework, we can’t always understand which metrics are more significant, which are intermediate, what to focus on in the next sprint, and how to globally avoid hyperfocus.
This is where the Metrics Pyramid helps. We distinguish four levels, ranging from micro-processes (platform metrics) to macro-structure (business metrics). We will talk about this below.
North Star Metric
Let me remind you what North Star Metric (NSM) is. It is believed that the NSM is an indicator that, by tracking and influencing, the company will definitely achieve its goals. Sometimes this metric is called utopia, but often it works.
NSM is suitable for a specific feature or task within a certain strategy for a limited period of time. In this case, you can find a really good metric. For example, hours watched per month on Netflix, DAU (Daily Active Users) on Facebook and Twitter, the number of boards for collaboration on Miro, and so on. Metrics hierarchy is often built as a decomposition of NSM.
But this approach has drawbacks. Take, for example, a checking account for business. Consider two metrics: the number of new customers (focus on marketing) and the number of monthly active ones (focus on the product). Which is more interesting? It depends on the goals. And if we take Revenue as the NSM, it will be universal for all products, but won’t help much in terms of decomposition.
The Metrics Pyramid is needed to prevent hyperfocus and keep a finger on the pulse of the product and business, while the Metrics Hierarchy often works the opposite way. It's like if a patient is treated by a doctor who is sure that if the temperature reaches 36.6, the other symptoms of the disease will go away on their own.
How We Build a Metrics Pyramid
We don’t obsess over NSM (North Star Metric) and its constituents. Working with such a pyramid is only possible with auxiliary and control metrics, which one way or another makes such decomposition a waste of time. We consider NSM as an indicator of value for clients, hence, our NSM features efficiency metrics and added value metrics. Enhancing the added value of a product is the main task of the product team, and product metrics don't always help in evaluating the efficiency of solving a client/user's task.
Measuring the values of such indicators is not a trivial matter. So here, we'll simply stick to high-level definitions:
- Efficiency metrics show the amount of effort and resources required to solve a user's task or to achieve a certain progress within its resolution;
- Added value metrics show how much more effectively one product solves a task than another.
Accordingly, our metrics pyramid construction goes slightly differently:
- In the audit stage, we identify the data being tracked and the data that needs to be tracked;
- Then we classify indicators to avoid hyperfocus;
- Next, we add a basic hierarchy;
- We establish connections between metrics.
As a result, we get a metrics pyramid that helps to prevent the occurrence of hyperfocus. And this pyramid consists of four fundamental "bricks": platform metrics, interface metrics, product metrics, and business metrics.
Platform Metrics
At the base of the pyramid are metrics related to the availability and technical reliability of our product. If the product cannot be used "for technical reasons", then there will be nothing to measure.
Interface Metrics
Next come the interface metrics, which display the user's interaction with the product. This includes the effectiveness of advertising campaigns, conversions of forms filled out by the user, and conversions of buttons like "submit a request."
Product Metrics
They characterize user behavior and the economics of the product, answering questions about the product itself. They allow understanding how the product converts new users into other entities.
Here are some examples:
- Retention shows how new users turn into active ones;
- LTV (Lifetime Value) shows how new users turn into profit over the entire usage period of the product;
- Conversion to the first purchase shows how the product turns new users into paying ones.
Business Metrics (also known as Growth Metrics)
If product metrics describe the product itself, growth metrics describe the business built around this product, showing the end result of converting new users with the product into other entities. That is, product metrics describe HOW transformations occur, while business metrics depict WHAT they turn into.
Examples of growth metrics:
- DAU or Daily Active Users (New Users * Retention).
- Profit (New Users * LTV).
- The number of users sending messages (New Users * Retention into Sending a Message).
- The number of new subscribers (New Users * Conversion into Subscriber).
- Revenue, Profit, margin, volume of products sold, and so on.
When evaluating product changes, it's not advisable to focus on growth metrics as these indicators depend not only on product characteristics but also on the influx of new users.
What's Next?
Once we have built the metrics pyramid, matched the indicators with our business model, and studied the metrics values, the most interesting part begins. It involves working with the metrics and monitoring their changes, predictive analytics, and influencing the achievement of the desired indicators — essentially, the continuous development of the product and, consequently, the business.
Metrics Pyramid and OKR
The combination of OKR (Objectives and Key Results) and the metrics pyramid is a powerful tool for the product team. Now, we'll explain why this is beneficial.
Aligning objectives. OKR helps the product team identify the primary objectives and expectations associated with the product or functionality. The metrics pyramid allows translating these objectives into measurable performance metrics that can be tracked and analyzed. The combination of OKR and the metrics pyramid helps the team understand how their work affects the overall goals of the organization and ensures alignment at all levels.
Aligning objectives. OKR helps the product team identify the primary objectives and expectations associated with the product or functionality. The metrics pyramid allows translating these objectives into measurable performance metrics that can be tracked and analyzed. The combination of OKR and the metrics pyramid helps the team understand how their work affects the overall goals of the organization and ensures alignment at all levels.
Measuring progress. The metrics pyramid provides the team with a set of performance measurements reflecting various aspects of the product or functionality's work. These metrics allow the team to track their progress and evaluate how successfully they are achieving their objectives. OKRs, on the other hand, provide clear and specific results that the team strives to achieve. The combination of OKR and the metrics pyramid helps the team measure and demonstrate their progress towards achieving the objectives.
Focus on Key Metrics: The metrics pyramid helps the team identify key performance metrics that are most crucial for achieving the product or functionality's objectives. Focusing on these key metrics will enable the orientation towards the end results and making informed decisions to improve these metrics. OKRs help the team set priorities and work direction, while the metrics pyramid helps the team focus on those metrics that are most essential for success.
Adaptation and Improvement: The combination of OKRs and the metrics pyramid allows the team to assess their results and adapt their work based on feedback and performance measurements. The metrics pyramid helps identify weak spots and find opportunities for growth.
Continuous Learning and Improvement: OKRs and the metrics pyramid stimulate continuous learning and improvement within the product team. Regular analysis of performance metrics allows the team to identify areas requiring improvement and take appropriate measures to optimize the product. The team can use the acquired data and results to make more informed decisions, conduct experiments, and make changes to the product with the aim of achieving better indicators.
Let's illustrate this with an example. We have a business goal — to increase sales on the site by 30% in the next quarter. To achieve this, we set OKRs — increase the conversion on the site from 2% to 4%, and also raise the share of search traffic by 15%. To measure progress in achieving these results, we can use the metrics pyramid. At the top level of the pyramid will be general business metrics such as total sales and total revenue. Below may be metrics related to marketing channels, such as the number of clicks on an advertisement and the number of new visitors to the site. Further down will be more detailed metrics related to specific user actions on the site, such as the number of additions to the cart, the number of completed orders, the number of abandoned carts, etc. To measure progress in achieving the key result — increasing the conversion rate on the site from 2% to 4%, besides tracking the conversion itself, we analyze the values and affect a number of related metrics reflecting user behavior. Thus, by reducing the bounce rate, working with the number of abandoned carts, etc., we move towards our key goal, while recording the result.
Predictive Analytics
Here we will delve a bit more into the topic of predictive analytics. If we can predict expected indicators of profit, churn, and others, then developing the product can be done much more efficiently. To engage in metric prediction, it's necessary to understand what it actually entails.
Predictive analytics is the process of using data and statistical models to predict future outcomes and user behavior in the context of the product. Key aspects of predictive metrics analytics include:
- Data Collection.
Predictive analytics requires access to historical data about the product, that is, across all constituent blocks of our pyramid in retrospect. - Identification of Target Metrics.
Defining key indicators that are currently primary for us in evaluating the efficiency and success of the product. - Data Modeling.
After collecting and aggregating data, they are run through ML (Machine Learning) algorithms for analysis and prediction. Examples of models include linear regression, time series, decision trees, and other methods. - Forecasting and Optimization.
Forecasts can be presented in various forms: numerical values, graphs, forecast ranges, etc. It all depends on what you agree upon with your Data Analyst. Optimization is based on a cycle of experiments and continuous analysis of the results to check the effectiveness of the presumed changes. - Model Accuracy Assessment.
Comparing actual metric values with forecasted ones allows evaluating the accuracy of predictive models. This includes calculating values of accuracy assessment metrics such as MAE (Mean Absolute Error), MSE (Mean Squared Error), determination coefficient, and others. Assessing the models' accuracy helps to determine their reliability and applicability in real scenarios. As data accumulates on specific indicators, the models make increasingly accurate forecasts. - Decision Making.
Based on the forecasts, it's possible to determine product development priorities, marketing investments, user experience optimization, and other strategic activities. This all seems complex. Therefore, it's more convenient to use a change prediction calculator.
Change Prediction Calculator
The Change Prediction Calculator is a tool mainly used in Digital Marketing for assessing the planned changes in the effectiveness of an advertising campaign. A simple example is the "Budget Forecast" in Yandex Direct. However, it's not limited to marketing; forecasting changes according to metrics is also possible.
The working principle of the change prediction calculator for metric effects may vary depending on the specific case, but it always relies on statistical methods, models, and ML (Machine Learning) algorithms. Working with such a calculator (regardless of the source of its acquisition) will be based on these steps:
- data input;
- data analysis;
- change forecasting;
- results visualization;
- interactivity (optional).
The calculator operates based on the data provided by the user and can predict how key metrics will change with various adjustments to specific parameters.
However, it's important to remember that it cannot account for all influencing factors, so the results obtained with the calculator may only be approximate. It's always necessary to consider real conditions and monitor metrics in real-time for a more accurate assessment of the forecasted changes.
However, it's important to remember that it cannot account for all influencing factors, so the results obtained with the calculator may only be approximate. It's always necessary to consider real conditions and monitor metrics in real-time for a more accurate assessment of the forecasted changes.
