E.g., customer interviews, screen recordings, focus groups. Qualitative metrics are low-volume and high-context they provide meaning and contextualize quant data.E.g., daily active users (DAUs), churn, engagement, sales volume. Quantitative metrics are high-volume and low-context they help understand facts and user behavior.A useful guardrail metric could be month two retention to help you understand whether you’re providing a good enough experience to retain these users over time (before further investing in user acquisition). For example, you might run an experiment to increase paid user acquisition, offering a heavy first-month discount. They’re what we watch to ensure a change we’re making doesn’t negatively impact another part of our product. Guardrail metrics are used when running A/B experiments. For example, if Spotify’s true north metric is “aggregate listening time,” three signpost metrics might be the (1) number of unique users, (2) unique songs played per user, and (3) average song length. But, using our black box analogy, they help us understand why our true-north is moving in a particular direction (or any other “important to keep an eye on” parts of our product). Signpost metrics don’t define our product’s success. A few examples from products you’ve probably used: True-north metrics are usually “late-funnel,” i.e., the culmination of a funnel involving many different steps. There can only be one, but we can supplement it with signpost and guardrail metrics. True-north is the singular metric used to track your product’s success (or output in our black box analogy). Health: do we know our product’s “health”? Is it working reliably or X% of users?ĭefine True-north, Signpost, and Guardrail Metrics.Empathy: can we identify meaning? Do we know how people feel about our product?.Success (measuring output): is our product moving in the right direction? Why or why not?.Instead of attempting to be exhaustive, I’ll focus on the core concepts applicable to most sets of product metrics. Metrics vary across industries, companies, & teams within companies. The Types of Metrics Used by Product Teams Align our efforts: keep our team(s) swimming in the same direction.React and adapt: alert us of product health issues or trends that put our goals at risk.Make decisions: run experiments and decide how our product should evolve.Understand output: reveal why we’re seeing a particular output.Measure output (i.e., measure success): quantify progress towards a business goal.Metrics help us quantify output and cut windows into the black box to understand why we’re seeing a particular output. Using Andy Grove’s framing from High Output Management: without instrumentation, telemetry, or indicators-all referred to as metrics below-our product is a “black box.” In this black box, input (code, design, hardware, time) goes in, and output (revenue, people using the product) comes out. How do you know whether you’re increasing user engagement? Why are people using your product more or less? And what the heck is “engagement,” anyway? You have billions of users and a complicated ecosystem. Now imagine you’re responsible for growing YouTube’s engagement. Imagine you’re flying a plane at night: how do you know which direction you’re traveling? Do you have enough fuel? Are you going to crash into a mountain? Product teams love using aviation terms like “instrumentation” or “telemetry,” and not just because it makes us feel clever.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |