post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Guest Post

Building for change: When transformation and product management meet

Product Development

What are product features? Definition, examples, and 5 types

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

Why teams often struggle with OKRs: A conversation with Jeff Gothelf

Lessons from my first year as a Prod Ops Manager

How Google makes AI work at Enterprise scale – Miku Jha (Director, AI/ML and Generative AI, Google Cloud)

5 ways to invent and simplify as an AI/ML product manager

Yoav Yechiam

<img loading="lazy" decoding="async" class="alignnone wp-image-14070" src="https://www.mindtheproduct.com/wp-content/uploads/2018/10/Upside-down-analytics-300x156.jpg" alt="" width="885" height="460" srcset="https://www.mindtheproduct.com/wp-content/uploads/2018/10/Upside-down-analytics-300x156.jpg 300w, https://www.mindtheproduct.com/wp-content/uploads/2018/10/Upside-down-analytics-768x399.jpg 768w, https://www.mindtheproduct.com/wp-content/uploads/2018/10/Upside-down-analytics-1024x531.jpg 1024w" sizes="(max-width: 885px) 100vw, 885px" />
Web and app analytics are fundamental to business these days. No serious developer, product manager, or CEO of company operating in the digital market, will consider not using them. The analytics market is also highly competitive, with providers of all scales and agility fighting to get their three lines of code or SDK into your software. As a result, the platforms are packed with features and very easy to use.
So if implementing is so easy and visualizing results is so simple, then the “data driven” methodology should be just killing it. But is it?
We get slick and shiny reports containing vast amounts of important information sent to us automatically on a daily basis. What do we do with that information after the first month or two? In most cases the answer &#8211; I confidently predict &#8211; is pretty much nothing. As my collaborator Paul Sartori, head of analytics at consultancy Station 10, says, nobody in business needs a report: “They might think they do, those very important people you do analysis for might say, &#8216;I need to see the company sales every day&#8217;, but think about it; that’s a statistic, not a report, and if it has no action it’s not worth spending time on”
<h2>Where’s the Problem?</h2>
The problem lies at the very beginning of any analytics implementation. Product managers and analysts have become accustomed to working through analytics in stages.
These stages usually look like this:
<ol>
<li>Map out all the possible events (don’t miss anything!) to an events table</li>
<li>Implement events table</li>
<li>Release version</li>
<li>Rest assured that everything is being recorded &#8211; good for you!</li>
<li>Build a dashboard to monitor new and returning users, retention metrics, and onboarding conversion rates… and that’s pretty much it, right?</li>
</ol>
What’s wrong with that? Not much, but is it enough? I hope for you it’s not.
The problem is that, in applying this approach (and so many of us take it), you find yourself constantly digging for the right answers, when what you should be doing is looking for the right questions.
Unless you have support from a super-talented machine learning team, having hundreds of events doesn’t serve you well at all. You’re bound to encounter data overload, making it very hard to decide where to begin and how to prioritize. In practice, having so many events confines you, as you see yourself bound to these events and you’re sure the answers to all your questions hide within the data. But you’re just making life harder for yourself.
Furthermore, defining too many events necessarily results in “weaker” events &#8211; events without thoughtful properties. I’ll dive deeper into the wise use of event properties and other techniques another time.
If the report says everything is fine are you all going to stop work? And if it says everything’s rubbish aren’t you going to have to investigate the reasons why and what you should do about it anyway? As Paul Sartori adds: “Why not do that in the first place instead of wasting time on pretty but pointless dashboards.”
Why is this happening? And why is it happening to nearly everyone? It’s because we’re all looking at it upside down. We’re looking for needles in a haystack of data instead of understanding that we have the means to track them as they are released and fall through the air. It’s time to flip our analytics perspective right side-up.
<h2>The Right-Side-Up Methodology</h2>
The term data-driven is very commonly used, but many miss its context. The idea was never for data to just sound the alarm when things go wrong. Its original context is data-driven decisions.
In fact, I would take it a step further and call it data-guided strategy. When we’re focused on our goals and not distracted by tactical constraints, we allow (or force) ourselves to choose the most relevant metrics and ask only the relevant questions.
For example:
<ul>
<li>Is our key metric absolute revenues, revenue per user, or maybe the rate of user growth?</li>
<li>Is our current product approach working? Is it serving our goal?</li>
</ul>
This way we plan our analytics implementation from the top down:
<ol>
<li>Begin with goals</li>
<li>Choose the metric / KPIs that truly matter</li>
<li>Ask strategic questions to explain cause</li>
</ol>
Once we’re confident we’ve asked all of the relevant questions we move to drawing out potential causes and effects. “What could be affecting this goal and what are the possible causes?”. The further and deeper we can take our questions, without relying on quantitative answers to ask the next question, the better. The moment we add that first metric it gets very hard not to become fixated on it…
For example:
<ul>
<li>We don’t need to know the actual onboarding conversion rate to know where users drop.</li>
<li>Eventually when we have quantitative figures we will, of course, want to see where biggest drops are, but at that stage we’ll be fixated on the numbers  &#8211; making it very difficult to be open-minded.</li>
</ul>
Our next steps are then tactical:
<ol>
<li>Select investigation methods to answer the questions</li>
<li>End with the event table / tracking plan</li>
</ol>
We need to select the most efficient and effective methods to investigate our chosen strategic questions out of the set of tools provided by our chosen analytics platform. At this point I must emphasize that this selection process isn’t just “nice to have” for tidiness’ sake. This is a supercritical task that must be done.
The reason is that analytics analysis can consume so much time if not performed efficiently. In fact, Hotjar reports that it is one of the top reasons that developers stop using its products, despite finding them valuable. Don’t underestimate the &#8220;weight&#8221; that the analysis part will have on you &#8211; a mistake I’ve made too many times. Over-exploring can, and usually will, cause more harm than good. That’s why you must lighten it up where ever you can by optimizing and focusing your investigation and methods.
So how do you become more efficient and effective with your investigation methods? There are a number of techniques you can use, which I will share in my next post. But most importantly, you should always try to find the shortest path to your answer. The more complicated the path, the greater the chance that a mistake will take place in the implementation process and corrupt your data…
Where most analytics start, we will end &#8211; with the events table. Once all the above are set, writing the events table should be a game of joining the dots and making sure no dots are left out. Remember you should:
<ul>
<li>Try to keep the number of events to a minimum</li>
<li>When it comes to event properties be lavish</li>
<li>Make sure you provide a detailed explanation of when each event is fired and what each event property is stored. Provide examples and the full list of input options if limited.</li>
</ul>

core/freeform

Common transformation pitfalls by Marty Cagan

Using principles to make better decisions faster by Martin Eriksson

How to build for creators by Peter Yang

How to navigate tricky work relationships

Organizing around complex products at the intersection of physical and digital by Matthew Cornford

Why do you Need to Rethink Your Analytics Strategy?

Yoav Yechiam

Read more

Comments