post

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Product Management

Sunday Rewind: Whose job is it anyway? The rise of Product Ops and why it deserves to be an independent function

Guest Post

Creating metrics that matter

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Building Product Teams

Business Strategy

Data Driven Product Management

Digital Transformation

Diversity and Inclusion

Minimum Viable Product

OKRs

Prioritisation

Product Design

Product Research

Product Development Process

Product Ethics

Product Discovery

Product Leadership

Product Management Career

Product Strategy

Stakeholder Management

🌎 <a href="https://www.mindtheproduct.com/world-product-day/" style="color:#FFFFFF;text-decoration:underline;">Happy World Product Day!</a> Be sure to celebrate product, yourself, and your peers today! 🎉

How much are product managers getting paid in 2024?

Leading product ops at Delivery Hero with Koen Dubelaar

Connecting fans to creators at Spotify (Dariusz Dziuk, Product Lead, Spotify)

“You’re not as competent as you think” 5 lessons from Ken Chin on navigating a product career (CPO, Spark Networks)

Everything wrong with the product management job market. And how to fix it

Mark is Head of Product at Ravelin, building ML-driven fraud detection and payment acceptance products. He has worked in product management for a number of startups and larger organisations such as the UK Foreign &amp; Commonwealth Office, Yahoo!, AltaVista and the BBC.

Head of Product

Mark Barlow

<p>Machine learning (ML) based products have particular characteristics and challenges, from data quality to counterfactual problems and explainability. What then are the implications of ML products for team structure, focus, and hiring?</p>
<p>Data science jobs are <a href="https://searchbusinessanalytics.techtarget.com/feature/Demand-for-data-scientists-is-booming-and-will-increase" target="”_blank”" rel="”noopener” noopener noreferrer">increasing at around 30% year on year</a>, and if you don’t already have a data scientist in your ranks there’s a good chance you will soon. Perhaps you already have a number of <a href="https://www.mindtheproduct.com/data-science-and-digital-businesses-by-ali-gulez-2/" target="”_blank”" rel="”noopener” noopener noreferrer">data products you use to segment customers, predict prices</a> or improve your product in other ways. Or maybe your core product <span style="font-style: italic;">is</span> the machine learning, making recommendations and predictions for healthcare, security, ad tech or other applications.</p>
<h3>ML Challenges</h3>
<p>ML lives and dies by the data it relies on; garbage in, garbage out. The wrong decision can be made if data is missing or biased, if there are collisions, or if data is received at the wrong time or in the wrong order. Furthermore, many ML models’ predictions will be used to determine a certain path in real time or near-real time. So once the data has been fed in and a decision has been made, it’s too late for a fix. The user has already been ushered down the wrong path.</p>
<blockquote><p><strong>Going all out for accuracy without considering how your users will digest the output can undermine trust in your product. How can you trust this thing if you can’t understand it?</strong></p></blockquote>
<p>Whether the data is good or bad, once interventions are made a counterfactual problem arises. The path having been taken, you cannot prove what would have happened otherwise &#8211;  and by extension whether your product was right.</p>
<p>A/B testing can address this, in principle: let a small number of users through without intervention, and see if your predictions were correct. But in practice this can be a tough sell in cases where there is a very measurable cost &#8211; financial or otherwise &#8211; of not intervening. And even if agreed, there may be a temptation to constrain the test artificially and thereby undermine its premise.</p>
<p>Then there’s the well-publicised black box challenge of ML products. We may know the input and the output, but not what the algorithm did in between. This is the question of “explainability”: the extent to which a human can analyse and explain the reasoning behind a prediction made by ML. In the UK, the Information Commissioner&#8217;s Office and The Alan Turing Institute are developing <a href="https://ico.org.uk/about-the-ico/ico-and-stakeholder-consultations/ico-and-the-turing-consultation-on-explaining-ai-decisions-guidance/" target="”_blank”" rel="”noopener” noopener noreferrer">practical guidance and checklists</a> to help with this challenge.</p>
<p>Different ML techniques have different levels of explainability. For example, <a href="https://medium.com/@chiragsehra42/decision-trees-explained-easily-28f23241248" target="”_blank”" rel="”noopener” noopener noreferrer">decision trees</a> are more explainable than neural networks, which are more sophisticated. There’s often a tradeoff between accuracy and explainability. Going all out for accuracy without considering how your users will digest the output can undermine trust in your product. How can you trust this thing if you can’t understand it?</p>
<p>If you’re offering a B2B service, this can affect both the buyer of your product and the end users. The end users may perceive machine learning as a threat, rightly or wrongly. Even products which set out to augment existing teams will change how they work, and change is hard. For example, an insurance analyst may shift from personally reviewing applications to directing ML-driven review tools and investigating macro level trends.The perceived impact needs to be thought through and mitigated.</p>
<figure id="attachment_18566" aria-describedby="caption-attachment-18566" style="width: 851px" class="wp-caption aligncenter"><img loading="lazy" decoding="async" class="wp-image-18566" src="https://www.mindtheproduct.com/wp-content/uploads/2020/01/decision-tree-shutterstock_566548609-300x200.jpg" alt="" width="851" height="567" srcset="https://www.mindtheproduct.com/wp-content/uploads/2020/01/decision-tree-shutterstock_566548609-300x200.jpg 300w, https://www.mindtheproduct.com/wp-content/uploads/2020/01/decision-tree-shutterstock_566548609-768x513.jpg 768w, https://www.mindtheproduct.com/wp-content/uploads/2020/01/decision-tree-shutterstock_566548609-1024x683.jpg 1024w" sizes="(max-width: 851px) 100vw, 851px" /><figcaption id="caption-attachment-18566" class="wp-caption-text">Decision trees are more explainable than neural networks (Image: Shutterstock)</figcaption></figure>
<h3>Team Structure</h3>
<p>Given these particular challenges, how should the ML product be managed?</p>
<h4>Data Quality</h4>
<p>To address the critical role of data quality, you need to focus on how data is received and stored, and who is responsible. Data engineering is commonly thought of as the plumbing, but I don’t think that really does it justice; it’s also the architectural plans and foundations of the house. Without good data engineering the edifice will fall apart. So it’s important to consider who will work on it, their level of experience, and how much time should be dedicated to it. Rather than distribute the responsibility, it may be beneficial to have a centre of excellence in the interests of consistency, efficiency and ensuring the data gets the attention it deserves.</p>
<p>At Ravelin, for example, one of our teams is explicitly focused on data engineering. The team supports the needs of multiple other teams such as data scientists, analysts, and integration engineers.  This runs the gamut of raw data being received to insights being extracted. In between, data may need to be normalised to ensure consistency and aid comparison; it may need to be enriched with complementary data sources; numerical values may need to be fed into calculations to populate a different data field; the list of actions goes on. And it’s not just the data this team grapples with, it’s also the surrounding infrastructure and pipeline for training and deploying models.</p>
<h4>Data Science and Analysis</h4>
<p>When it comes to data science, you could choose to create a centralised shared service team, or else distribute data scientists in multidisciplinary teams. Each approach has trade-offs.</p>
<figure id="attachment_18565" aria-describedby="caption-attachment-18565" style="width: 732px" class="wp-caption aligncenter"><img loading="lazy" decoding="async" class="wp-image-18565" src="https://www.mindtheproduct.com/wp-content/uploads/2020/01/data-scientist-shutterstock_1072419839-300x200.jpg" alt="" width="732" height="488" srcset="https://www.mindtheproduct.com/wp-content/uploads/2020/01/data-scientist-shutterstock_1072419839-300x200.jpg 300w, https://www.mindtheproduct.com/wp-content/uploads/2020/01/data-scientist-shutterstock_1072419839-768x513.jpg 768w, https://www.mindtheproduct.com/wp-content/uploads/2020/01/data-scientist-shutterstock_1072419839-1024x684.jpg 1024w" sizes="(max-width: 732px) 100vw, 732px" /><figcaption id="caption-attachment-18565" class="wp-caption-text">Ensure close collaboration between data science, product and engineering, regardless of structure (Image: Shutterstock)</figcaption></figure>
<p>With a centralised team, especially one where the data scientists are largely working on the same problem area, the central model makes it easier for lessons to be shared daily and applied rapidly. It creates a shared purpose and a focal point of responsibility for the accuracy of the ML: the buck stops here. It may also be the only option, depending on the size of the company or number of data scientists. The risk is the team becomes a functional silo, not working closely enough with product-engineering teams, with goals and priorities which are not in sync.</p>
<p>Conversely, embedding the data scientists in cross-functional teams can help with alignment and the early stages of new product development. That team will have reduced dependencies and increased autonomy. But this may not fit so well with the skillsets, areas of interest and future development of the data scientists. It may disrupt the balance of operational data science work, such as constantly training new models, and new product development work. Asking a data scientist to focus only on one area might be equivalent to asking a product manager to only look at the UX, or only the tech, or only the business.</p>
<p>The most important thing is close collaboration between data science, product and engineering, regardless of structure. Make sure you don’t miss out on the considerable brains trust of the data science team, including in less obviously “data sciencey” areas. On the one hand, you can add further diversity of thought to your decision-making. And on the other, you should ensure you aren’t making product changes with unforeseen impact on the quality of your ML. For example, you could unwittingly stop models from learning by encouraging user behaviour which cuts out particular data signals.</p>
<p>Another consideration is who is responsible for data analysis. Just the data science team? Or are there other people, teams or roles which could augment the machines with the benefit of real-world context, intuition and interpretation? You might look at existing analyst or customer care teams. At Ravelin we created a specific investigations team for this purpose, a blend of<a href="https://www.ravelin.com/blog/why-we-have-our-investigations-team" target="”_blank”" rel="”noopener” noopener noreferrer"> client support, fraud investigation and data science</a>.</p>
<h4>Product Management</h4>
<p>I don’t think a particular profile of product manager is needed for ML products, certainly not across the board. We’ve recruited a mix of technically oriented product managers and generalist product managers.</p>
<figure id="attachment_18567" aria-describedby="caption-attachment-18567" style="width: 689px" class="wp-caption aligncenter"><img loading="lazy" decoding="async" class="wp-image-18567 " src="https://www.mindtheproduct.com/wp-content/uploads/2020/01/analytical-skills-shutterstock_587671397-300x200.jpg" alt="" width="689" height="459" srcset="https://www.mindtheproduct.com/wp-content/uploads/2020/01/analytical-skills-shutterstock_587671397-300x200.jpg 300w, https://www.mindtheproduct.com/wp-content/uploads/2020/01/analytical-skills-shutterstock_587671397-768x513.jpg 768w, https://www.mindtheproduct.com/wp-content/uploads/2020/01/analytical-skills-shutterstock_587671397-1024x683.jpg 1024w" sizes="(max-width: 689px) 100vw, 689px" /><figcaption id="caption-attachment-18567" class="wp-caption-text">Good analytical skills and attention to detail go a long way (Image: Shutterstock)</figcaption></figure>
<p>With ML products there is more emphasis on the data and APIs, and good analytical skills and attention to detail go a long way. Likewise proficiency with Chrome Developer Tools and SQL. You’ll regularly need to tell the difference between a bug and an issue with the underlying data. Then think of better ways to handle that data.</p>
<p>But the product management fundamentals for ML products remain the same. Good teamwork and communication, a large dose of curiosity, and an eagerness to learn are more than enough for someone to be successful working on ML products.</p>
<p>Before you start on a product role in this area, you should read up on metrics such as precision, recall, F1 scores and other common measurements. And read an <a href="https://www.amazon.co.uk/s?k=machine+learning+beginners" target="”_blank”" rel="”noopener” noopener noreferrer">entry-level machine learning book</a>, check out the Google <a href="https://www.youtube.com/watch?v=cKxRvEZd3Mw" target="”_blank”" rel="”noopener” noopener noreferrer">machine learning recipes</a> videos, or this <a href="http://www.r2d3.us/visual-intro-to-machine-learning-part-1/" target="”_blank”" rel="”noopener” noopener noreferrer">visual intro to ML</a>.</p>
<p>In the first month you’ll have to get your head round many new concepts. The existence of something called a <a href="https://en.wikipedia.org/wiki/Confusion_matrix" target="”_blank”" rel="”noopener” noopener noreferrer">confusion matrix</a> will never seem more apt. But soon things start falling into place. You’ll figure out what you need to know, and what you don’t need to know. And it is of course a great place to be as a product manager: in an increasingly important domain with interesting challenges and many opportunities to learn and grow.</p>

core/freeform

AI ethics advice from former White House technologist – Kasia Chmielinski (Co-founder, The Data Nutrition Project)

How Google makes AI work at Enterprise scale – Miku Jha (Director, AI/ML and Generative AI, Google Cloud)

LLM workflows for product managers: 3 key takeaways (Niloufar Salehi, Assistant Professor at UC Berkeley) – ProductTank SF

The future of product management: Insights from ProductTank San Francisco

Product lessons learned making early moves with AI in media: Lindsey Jayne (CPO, Financial Times)

A year with ChatGPT and product innovation: Navigating the AI landscape

How to keep your head about generative AI (when everyone is losing theirs) by Claire Woodcock

Why Machine Learning Products Should be Managed Differently

Mark Barlow

Read more

Comments