Benefits and pitfalls of using machine learning to mitigate customer churn

Machine learning is a practice often associated with behind-the-scenes algorithms and processes which automate repetitive problem-solving where human effort would be too costly or impractical. When I hear people talking about machine learning (ML) in product management, the typical line of thought is about building ML into a software product to create some new feature. This can lead to great product experiences that wouldn’t otherwise be possible—think photo applications like Photos on iOS which recently launched the ability to recognise pets in your photo library. This ML process runs on your phone when you’re not using it and enhances your photo library without any human effort. While fascinating, I think limiting discussions around ML to these types of product experiences shortchanges the practical applications Machine Learning can deliver for better product experiences as well as business outcomes.

Seeing an opportunity for ML

A common problem product managers face is customer churn. Product managers spend weeks, months, and years crafting a product experience they believe their customers will love. The customers come, they use the product, they love it, but over time their interest may fade, their needs could change, or something else pulls them away and they churn. How much a PM knows about why their customers churn can be highly correlated with just how big of a problem churn is. For subscription products, monitoring, addressing, preventing, and mitigating churn could be the difference between product success and the abysmal failure of an entire business.

I had the fortune of working with a product org that saw the benefits of machine learning in both the end-product experience, as well as in data analytics. With a team of dedicated ML engineers, we were able to build a model which could analyse customer behaviours commonly associated with customers who were likely to churn and apply a score for customers likeliness to churn. The problem we aimed to solve was knowing which customers were likely to churn so steps could be taken to mitigate that churn. We managed to solve the first part of the problem by establishing a churn prediction score for customers and did so with a high degree of accuracy. The next problem we had to solve was determining what steps to take for those customers with high churn scores—and decide if that problem could be (or should be) solved with more ML development, or something else. 

With a list of customers with high churn predictability scores, we pursued an optimal combination of CRM tactics (e.g., email, in-app messaging, push notifications) and marketing messages. When crafting messaging to mitigate churn, we considered (1) reminding customers of benefits they have (or may not be using), (2) notifying customers of new features, and (3) offering customers an incentive to renew or keep their subscription (e.g., a discount). These efforts are better constructed, delivered, and evaluated by product marketing experts rather than ML-engineers, though. The bigger challenge, then, was to bring all of data and findings to substantiate the overall investment.

Understanding the value ML has provided

In a quarterly business review where results on ML investment were being shared, a question was posed by the leadership team—”how many customers have we prevented from churning as a result of this ML work?” This was an absolutely valid question but one the initial customer problem and eventual solution didn’t set out to solve. Was the ML work successful in identifying customers who were likely to churn? Yes. Did we figure out what messages we could send customers to prevent them from churning? Yes. So, why couldn’t we succinctly answer that question with a number? We lacked experimentation results that showed a reduction in customer churn we could attribute specifically to the churn model, separate from the marketing messages that were sent—so we set out to do that next. 

Up until this point, we had also made investment to operationalise more of the churn prediction work, finding ways to integrate it more natively in our CRM tools and setting up automations—and we had built a roadmap to do all of this. We had gone too far down the path of the ML churn prediction work under the assumption that “of course churn prediction scores will help us prevent churn!” but we hadn’t proven it. We paused the development work and focused on substantiating further investment with data and measurement to determine if it was worth investing more, or moving on to something else. In retrospect, we had gone too far in our work before determining if this ML investment was the best use of the resources or if it was akin to a “science project”.

Working with ML engineers

Outside of the benefits and challenges of leveraging ML to improve your product, PMs should be thoughtful about their approach to working with ML engineers. Authoring and building ML algorithms requires a different skillset than what is used in more traditional software engineering roles for application and website development. But, as with working with any type of engineer, it’s a best practice to include engineers in the early stages of the product development cycle. Building a shared understanding of your customers, the problem you’re trying to solve, and why the work is important will absolutely aide in the development process, regardless of the engineering expertise needed for whatever you will build. When it comes to ML development, though, the watch-outs are to know when to cut losses and work on building something else and avoid the sunk cost fallacy. Being prepared to have this conversation with your ML engineering team or manager will help you overcome obstacles or changes in strategy.

To sum up

Machine Learning can bring a lot of benefits to a product, as well as to product marketing and engagement measures that keep your customers active. With ML engineers in place, spend time thinking about how their function can best unlock value for your customers as well as your business. At the same time, re-evaluate your ML investments and decide if they’re worth pursuing further, or if your ML resources could be deployed to solve another problem with a better outcome that can be proven. Lastly, anticipate the questions you’ll get from business leaders to really prove the value or results that ML efforts yield.