Artificial Intelligence (AI) & Machine Learning (ML)

A guide to machine learning Part 2: Set yourself up for success

November 21, 2022

8 min read

Share on

Eira Hayward

Eira is an editor for Mind the Product. She's been a business journalist, editor, and copywriter for longer than she cares to think about.

Is it a problem machine learning can solve?

ML is good at classifying, at automating repetitive tasks, and at finding patterns in complex data that human brains would take a long time to do. It’s not suitable for solving human problems, creative challenges, codifying culture, or social interaction. It’s starting to get better at this, but it’s not where its strengths are, particularly for people who are just starting to use the technology.

People are often tempted to over-engineer problems with machine learning. Graham Paterson, Product Partner at Connect Ventures, cautions: “People try to shoehorn machine learning projects into their products because founders love it – they think it will help them to attract funding – and people think it looks good on their CVs.”

Development cycles are different

Bear in mind that ML development is quite different from a normal agile development cycle – the test, iterate, put it in customers hands, moving very quickly that all product managers are used to. Machine learning starts with offline experimentation. You continue offline until you’ve developed a model you’re confident performs well with your offline dataset. Only then can you let it be used on real people.

Often you’ll find that offline and online don’t correspond, says Claire Woodcock, Staff Product Manager at Twitter research lab Cortex Applied AI Research. You test it online,and see where the model’s strengths and weaknesses lie. You might decide you need to gather more data, and you have to refine and refine. This can go on for quite some time. Says Claire: “Realistically, if you’re starting from scratch, we’re talking six to 12 months, before you would confidently have something that you could put in the hands of consumers.”

The right data

We covered this in part 1 of this series, but it’s worth repeating a few headline thoughts. You need a lot of data – good quality, fresh, reliable data, in a form that can be consumed by a ML algorithm. You need consent to use it in the way you want – and in some territories this means you need explicit user consent to use their personal information to make an algorithmic decision about them. It needs to be secure, so that it’s not vulnerable to injection attacks. Making sure you have the right data in the right format is an expensive business.

Outsource if you can

The buzz around machine learning means it’s very tempting for a product leader to think they should build a machine learning algorithm in-house and hire people to do it. Claire Woodcock advises: “Look at your problem, it might be the case that your most efficient route to market is to purchase that capability outside of your company. You might outsource the problem to someone else to develop because talent is scarce.” There’s a raft of businesses to outsource to, from AI consultancies, who employ applied scientists, through to businesses building AI as a service.

Graham Paterson’s advice is also germane here: “If you can outsource it, do. Only do it in-house if you’ve got a specific niche problem, or if the problem is a real differentiator.” Graham was a senior product manager at Deliveroo for some years. He says Deliveroo had dispatch technology, like similar businesses, but the market wasn’t big enough to sustain a third-party supplier of dispatch technology so Deliveroo developed its own. He says: “If it’s important enough, then I strongly feel the best thing to do is create a dedicated team.”

If you have to develop yourself

A machine learning product team is different from a regular product team, says Graham, partly because it has data scientists rather than designers. And whereas user facing teams will be able to take small problems, fix and ship them very quickly, a machine learning team takes a long time to get going.

It can still be an agile way of working though. Says Graham: “What I found key as a product manager was to encourage the team to think in a lean and agile way as much as possible. To break things down into as small chunks as we could and treat every step like the validation process.”

He also points out that many data scientists come from academia, where they may have spent five years finding the best theoretical solution to a problem. “A big part of my job was convincing these people to get something together in two or three weeks. For me it was slow, for them it was light speed. A big part of my role was cajoling them and working out where we could and couldn’t cut corners.”

Ethics and unintended outcomes

Machine learning decisions can sometimes feel wrong. Assigning riders to deliveries in an online food delivery business is hard to design for, for example, and machine learning might give an outcome where one rider makes no money one week. Humans can see that as unfair, and end up trying to design a system which becomes very complex, taking many parameters into account, and which takes far too long to build.

A ride hailing app might build an algorithm that assigns drivers to rides, and which assigns all rides to its few top drivers. One outcome might be that the vast majority of drivers leave to work for competitors. Or perhaps its surge pricing mechanism means that it always optimises for making the most money and it becomes very expensive to use. Says Graham: “Machine learning can create outcomes that you’re not ready for, that you haven’t planned for. It needs very careful design.”

The parameters need to be very clear. AI typically goes wrong when it becomes long and cumbersome because the people designing it don’t have the right expertise, says Graham, or because the parameters aren’t clear. “You need to set an objective function and be clear about exactly what it’s going to optimise for, and what it’s not.”

Claire Woodcock says: “Just like humans, machine learning does not perform accurately 100% of the time, there are always going to be edge cases. As a product manager, you have to set the tolerance and understand where you want the algorithm to index.” This means making sure that you serve all your customer demographics properly and be crystal clear about what good performance of the model looks like. This in turn harks back to a fundamental of product management: knowing your customers. Consider potentially underserved elements of your dataset, and measure whether they are treated fairly.

Don’t forget the user experience

It’s easy to forget that at some point a human will probably have to interact with a machine learning interface. Claire says sometimes machine learning technology can be so hard to develop, that researchers will introduce a user experience to make the algorithm easier.

She gives the example of Mastercard’s “smile to pay” biometric checkout program. Customers have to smile or wave as they pay so that the program knows it’s looking at a real person in real time, and some may find this an awkward or embarrassing thing to do. “Just because it’s making the algorithm’s life easier, doesn’t mean that it’s a good experience for your end user. And that’s something to really keep in mind when you’re building the solution,” she says.

She has some final words of encouragement: “When you work with machine learning for the first time you might feel quite daunted and not have confidence in your product sense.” Even if you’re intimidated by the technology, don’t forget to rely on your product skills – you will need them.