How is AI reshaping the landscape of product-led growth? Join us as we chat with Nichole Mace, the SVP of Product and User Experience at Pendo, who offers an insider’s perspective on the AI revolution within product management.
Featured Links: Follow Nichole on LinkedIn | Pendo | 'Six things we learned at the Pendomonium + #mtpcon roadshow London 2024'feature by Louron Pratt
Episode transcript
Lily Smith: 0:00
Hello and welcome to the Product Experience this week. Randy and I take you back to earlier on in the year when we met Nicole Mace, svp of Product and User Experience at Pendo. We had a really good chat about how AI is shaping the future of product-led growth and how Pendo are embracing AI in their product development. The Product Experience podcast is brought to you by Mind, the Product part of the Pendo family. Every week we talk to inspiring product people from around the globe.
Randy Silver : 0:33
Visit mindtheproductcom to catch up on past episodes and discover free resources to help you with your product practice. Learn about Mind, the Product's conferences and their great training opportunities.
Lily Smith: 0:45
Create a free account to get product inspiration delivered weekly to your inbox. Mind. The Product supports over 200 product type meetups from New York to Barcelona. There's probably one near you, Nicole. Welcome to the Product Experience Podcast.
Nichole Mace: 1:06
Thank you.
Lily Smith: 1:07
I am thrilled to be here. It's great to be talking to you in person and, before we get stuck into our topic today, it'd be great if you could give our listeners and watchers a real quick intro to what you do in product and how you kind of got to where you are.
Nichole Mace: 1:22
Yeah, sure, sure. So I'm at Pendo and I head up a part of our product at Pendo. I'm specifically focused on the end user experience, so I have the luxury of doing a lot of fun things. I came to Pendo to start the growth team. I'm a huge PLG lover and practitioner, so there was nothing better in the world to me than using my favorite product to do my favorite thing. And since I've been at Pendo, my role's expanded and I get to head up design and research and education and training and a couple of new products, one of which is an AI product. So, yeah, so, and I've been in product my entire life.
Nichole Mace: 2:02
I was thinking back that I've been in product since I was 23 years old and I kind of fell into it. I interviewed at a startup I didn't even know what product management was and they said well, tell us what you do. And I had just graduated college. I hadn't really done much, but I had worked in the computer lab at the business school and I had minored in art history and I also minored in information systems. And when I was doing a project for art history, they said we need a website. And I was the most technical one of the art history people, and so I was the one who did the website.
Nichole Mace: 2:37
And so when I interviewed for the job, they said, well, you know, what have you done? And I said, well, I did this website. And they said, well, why did you put the picture here, why did you put the text here? And I thought about it and they said product management for you. And that's been my whole career and you know I'm so grateful to that company. That was ultimately bought by Intuit. That started my career and it was wonderful to have Intuit as a company to learn from. Their focus is do right by the customer at every turn, and that's just defined who I am Lots of usability testing, lots of qualitative research, and so that has driven me and provided a lot of fulfillment for me personally.
Randy Silver : 3:18
That's great and, nicole, one of the things we wanted to talk with you about explicitly was, now that you're working with AI, how it's been a very different experience and how it's been disruptive, both about developing products but also about the general approach to PLG. Before we get into that part of it, you said you started with Pendo on other stuff, on PLG and growth, and then you got handed this AI product. How did you get up to speed on it? How did you get ready for it? This is a new area for you, or was a new area when you took it over?
Nichole Mace: 3:49
I know, and I think many of us have these stories how did it happen? Well, ai came on the scene. They needed someone, and I think, as product managers, we spend our life with people telling us just focus. You're executing, get this thing out the door, don't look at that next shiny thing. But that's always hard, and I've definitely been one of those people who get excited about new technology and the next shiny thing. So, as soon as you know, really, chat GPT, really, you know, hit the market. I was just enthralled. I was so excited. I read everything I could, I played with it. It was just a lot of fun. And so when they needed someone to lead this one AI project in particular, I said please, please, let it be me. And it's been a lot of fun, yeah.
Randy Silver : 4:37
How did you get yourself up to speed on how to work with it, because it is a different experience, yeah.
Nichole Mace: 4:42
Yeah, and there are kind of two different things. Right, there's, you know, the LLMs that you can sort of just do an audit of your product and say what can we do, how can we make our product better, summarization themes, things like that. There's just some table stakes now, so that was just kind of an audit of the product. This AI product that I'm working on is based on some machine learning that we've been working on for about three years and super, super exciting. It actually takes a look at all of your data and provides recommendations. And what I think has been so disruptive about it to me as someone who's been in product for a long time, there are certain things I know. Right. I know that if the button is below the fold, people aren't going to be able to find the button right, so you've got to put the button above. Maybe you put two buttons. I know where the image should go. I understand the volume of copy people want. I understand what the navigation should be. There are things that you just know, and with AI it's all brand new.
Nichole Mace: 5:44
One of the things that surprised us with our recommendations and I was thinking about actually Netflix Netflix is the master of recommendations. We're all consumers of the Netflix recommendations and those recommendations are provided. They're sort of low stakes, right. You know they suggest a rom-com. Maybe you don't like it, who cares? Maybe you try it out. It's just not a big deal. But with recommendations on a business level, actually users do care deeply about that. They want to know how did you come up with that? And that's great. Your AI recommended it. But I need to double click into it. I need to understand how you got there so I can support it and get on board and figure out how to move it forward. So those user expectations of what they're going to do, how they're going to interact with the AI, is brand new and sometimes you're not putting that time in your roadmap.
Nichole Mace: 6:38
You can, with just traditional software development, have a pretty good idea. You get your estimates, you're able to move forward, you're able to deliver on a date. With AI, there are things that pop up around how users want to interact with it. That's different. You just can't quite anticipate. So I've enjoyed it. It's been a little bit of a windy road. I think the other piece with it is how do you incorporate machine learning into your software development? If you think about it, there's sort of design, front end, back end and then machine learning. So for a few years I think, everyone had separate machine learning teams, entirely separate. What we've tried to do is incorporate into our sprint planning, into our overall software development machine learning. So that's been really, really interesting and we've learned a lot through that as well. So still learning there, but it's a new way to develop for sure, people learning as well as machine learning.
Lily Smith: 7:37
That's right. You mentioned there about how you know you have your front-end, back-end design and then kind of machine learning and you're trying to bring all of that together, one of the things I find really interesting. So I worked with a product that used a lot of machine learning a long time ago and you know it was very much, I guess, owned by the engineering team. Right, how do you bring design into that conversation so that and I guess, kind of testing and user testing and user research into the conversation of like machine learning, because it's a very technical aspect of the product it is, and I think what's interesting is, while it's technical, the users are still trying to do the same thing that they've always tried to do right.
Nichole Mace: 8:23
They're trying to solve a problem, and so matching that machine learning with solving that problem is absolutely critical, and so it does require that UX research, a lot of it, and to really dig in. Is this solving your problem? What are you going to do next? So we've done a lot of prototyping to get feedback. We've put out models that were very, very early, just to see how users would interact with it. So I think my answer is a lot of iteration, a lot of iteration and then real focus on deeply listening to our customers.
Randy Silver : 8:58
All that iteration. It takes time to do this right and people are always being asked. You talked about estimates earlier. People are always being asked to commit to dates or at least give some guidance for others. We push back, we do now, next, later, roadmaps and things like that, but there's still some expectation of it. But now you're working with a much higher degree of unknowns, or unknowns in the process, at the very least. What's been your experience? How do you deal with that?
Nichole Mace: 9:27
You know, I think the reality is we're all on this ride together, and so are our customers. So, you know, we have, for example, recommendations that are super interesting and we're going to be releasing it. It will provide value. But it will provide even more value two years from now as we continue to get more information. So, with machine learning models, you're gathering more information, you're learning and you're improving those models. So I say we're inviting our customers, we're building this machine learning together, right, and that's the beauty of it.
Randy Silver : 10:05
So one of the keys is setting the horizons for success. It's we're not going to get the full value of this in two months or six months. This is we're on a mission. This is going to take a while.
Nichole Mace: 10:11
That's right. That's right, and the reality is no machine learning is 100% and we're in a world of AI assist and consumers are so happy to have the assist right. We're solving problems that people have spent so much time, I think, digging into data, trying to find the insights, trying to find what you should do next as a product manager, as a lifer, I've just spent weeks and months digging into data, right, just to try to figure out what to do next, so to be able to serve up to someone. Hey, here, digging into data, right, just to try to figure out what to do next, so to be able to serve up to someone. Hey, here are some ideas. Right there, you are adding value to their experience.
Lily Smith: 10:48
So you mentioned at the top of the discussion that you're passionate about product-led growth. How has this project and you know, this kind of new way of working or kind of a new consideration, I guess?
Nichole Mace: 11:08
affected how you think about product-led growth, yeah, yeah. So, for one thing, product and with product-led growth, you're doing a tremendous amount of experimentation and so understanding, using machine learning to understand well what are the drivers, for example, to conversion, or what are the drivers to get someone to actually come back. That can help you move along more quickly. So I would say, with product-led growth, often it's trial and error. You're starting with some information, but not as much. Now you're able to start with even more information and really have tighter experiments around.
Nichole Mace: 11:40
Huh, it seems like from our data that if people do these three things, they're actually going to stay longer. So you're starting with more information. You can be more targeted with your experimentation. I would say the other key thing that you can do now is more personalization, right? So you know, we found over the years that if you can provide more personalized experiences, people are more likely to stay engaged, whatever that means to you. But they will stay more engaged and that sort of matrix of what you can support with AI is so much greater than by doing it manually.
Randy Silver : 12:15
This episode is brought to you by Pendo, the only all-in-one product experience platform.
Lily Smith: 12:20
Do you find yourself bouncing around multiple tools to uncover what's happening inside your product?
Randy Silver : 12:26
In one simple platform. Pendo makes it easy to both answer critical questions about how users engage with your product and take action.
Lily Smith: 12:33
First, Pendo is built around product analytics, enabling you to deeply understand user behavior so you can make strategic optimizations.
Randy Silver : 12:41
Next, Pendo lets you deploy in-app guides that lead users through the actions that matter most.
Lily Smith: 12:47
Then Pendo integrates user feedback so you can capture and analyze how people feel and what people want.
Randy Silver : 12:53
And a new thing in Pendo session replays a very cool way to experience your users' actual experiences.
Lily Smith: 13:00
There's a good reason over 10,000 companies use it today.
Randy Silver : 13:04
Visit pendoio slash podcast to create your free Pendo account today and try it yourself.
Lily Smith: 13:10
Want to take your product-led know-how a step further? Check out Pendo and Mind, the Product's lineup of free certification courses led by product experts and designed to help you grow and advance in your career.
Randy Silver : 13:22
Learn more today at pendoio slash podcast. You were just saying starting with more information, you can be much tighter in your experimentation. What kind of stuff do you need to set that up right? You're not just taking a big amorphous mass of stuff and trying saying here and I'll go do it for me, what's the secret to getting that right?
Nichole Mace: 13:48
I don't think there's a secret to getting it right, but I do think you know. I'll give you an example. We have our free product. We have lots of users coming in, we have lots of data. So instead of spending a tremendous amount of time sort of digging in and what are bunch of experiments well, if we drive them to do this or that, that will get them to come back. We can run our machine learning on top of the user's experiences and our machine learning can say to us hey, actually if people go try out this report, they're going to come back more frequently. So we're able to say with our machine learning, they are 14 times more likely to retain if they do this one thing.
Lily Smith: 14:44
So then that's an experiment that we want to run, yeah, and I guess for you building products, for product managers, that just feeds all the way back into the products that you then it does.
Nichole Mace: 14:54
It does. It's very meta. It's very meta and it's a lot of fun. Right we're solving our own problems and I think it's really progressed product management, because no longer are we sort of very, very in the weeds. We're moving to a place where we're developing product that is really solving business problems, really connected to those business outcomes. So that's exciting.
Randy Silver : 15:18
Where do you want to see this go? Where do you think it's going to go over the next few years? What's going to be the next step change?
Nichole Mace: 15:24
Oh my gosh, that is quite a question. I think we're going to be producing a lot more software and we're going to be sunsetting a lot more software. So ultimately, we will move into a world where it's much more personalized, where, as a user, you feel much more empowered. You're coming in, you're getting just what you want. Your aha moment in a PLG experience is happening just the way you want, more quickly than you thought it was going to. You're going to see impact there. So a personalized experience is really where we're all going and I think we're going to see a huge step up from that from where we are today.
Randy Silver : 16:01
Nicole, at the very beginning of our chat you talked about one of the things that the use cases was for auditing your product. What's your advice for other people to get started on that? How would you use, how do you use AI and what's the advice you give to somebody new to this approach?
Nichole Mace: 16:18
Yeah, yeah, it's a great, great question. Most of us have had our roadmaps be completely blown up and then you think, well, how am I going to put my roadmap back together? That includes AI and machine learning, and so I think there are two things that you need to do, two approaches that ultimately come together. One is you need to go through your product, think about all the things you can do with AI and machine learning, and go through your product and do an audit and say what are those basics that you can add to your product? What is unique about your product? What unique data do you have?
Nichole Mace: 16:54
Imagine if you could do something really special, perhaps provide some kind of recommendations with your data. How can you make your end users life easier? And then marry that with the problems that you're trying to solve? So, as product managers, we're really really focused on solving problems.
Nichole Mace: 17:12
I think what's interesting and new is that with AI and machine learning, there are lots of problems that we haven't even tackled, that aren't even on our list, because we never thought we could do anything that could really move the needle there. So add to your list of problems that you think you can solve and then take your audit and take your problems that you're solving and marry those and that's a great great starting point. And then I would go through as a second layer and see how you can add personalization throughout the experience, add to that matrix of what you want to offer what is your dream that you want to offer to your end users? And make it consistent outside of the product, from email into the product, into your help center. I've got someone I know told me they banned consistent outside of the product, from email into the product, into your help center.
Randy Silver : 17:56
I've got someone I know told me they banned the use of the word AI when people were asking them for features and development, because it's just like saying can we put magic pixie dust on it? Yes, and they say be more specific. Are we talking about LLMs? Are we talking about ML? Are we talking about even RPA, things like that, any other acronyms under the sun? I'm wondering, when you do this, is it targeted? Are you looking and saying these are use cases, these are capabilities that we didn't have in our arsenal before, and that's how you approach it? Or is you just going more wide open and saying what are all the things we dream of and then looking at capabilities afterwards?
Nichole Mace: 18:33
Yeah, I think you need all of the above right. This is a really, really exciting time, and no idea is a bad idea. So I would throw everything on the board, whether it's your Miro board that you're using to brainstorm or your whiteboard. Throw it all up there and really think big. And your story may not be this sort of immediate story. It may be. This is where we want to be, and it's going to take us a couple of years to get here, but you've got to get going now. This is not the time to sit back and watch and see what happens. The world is moving very, very quickly. Consumer expectations and user expectations are changing dramatically, so you've got to move. You've got to move, but have fun with it. Right, this is exciting.
Lily Smith: 19:18
What's been your favorite use case of. Like you know where you've seen AI or ML, I'm like don't use AI now.
Randy Silver : 19:27
I think I'm sorry, the right terminology.
Lily Smith: 19:30
Yeah, but like any kind of these emerging AI technologies, what's your favorite use case of? Like wow, that's such a good use of this and I can really see that changing people's lives.
Nichole Mace: 19:42
Yeah, I mean, I think one thing you know we did a study, a generative study, looked at the time of product managers and look at all of their calendars and if you look at a product manager's calendar it's just booked all day long With meetings. They might have blocked off some time to dig into one thing or another, but they're extremely busy. So a lot of times that deeper analysis of qualitative data, for example, takes place before work, at the end of the day on the weekend, and that's not great. Nobody wants that.
Nichole Mace: 20:18
But I think now with all the LLMs, with really looking at that qualitative data that's coming in from lots and lots of places, that not only summarization, finding the themes, finding the insights and actually driving to ideas of what you're going to do next, that's not all automated. So what I'm excited for as well, now as product managers, we actually don't have to do that. It can be served up to us. We cannot obviously dig in and go deeper if we want, because it's all available to us. The evidence is there. But we're not going to be spending our time doing this very sort of tactical work. It sort of rises us up as more business leaders than as sort of like tactical summarizers and insight finders in qualitative or quantitative data.
Randy Silver : 21:08
Nicole, when you're recruiting people now, are you looking for people who already have technical abilities technical skills in this space or are you looking for people who already have technical abilities technical skills in this space or are you looking for people that are just general ads to your team and you figure okay, I learned this, they can learn this too. We can teach them.
Nichole Mace: 21:24
Yeah, it's funny that you say that, because not that many people have this experience, right, if you really talk to people, people have six months a year, you know really being involved. And it's more curiosity, right, when you talk to someone, what tools are you using? Have you tried anything out? That's cool, so that someone sort of has a fire in their belly around trying new things. But in the end, we as product managers are judged based on business goals and solving user problems. So those are still, in my mind, the absolute most important, because there's always going to be new technology coming and we are on the advent of even more and more and more right, and so you really have to be prepared and figure out how to harness that technology to solve business problems, to solve your end user problems.
Lily Smith: 22:13
And what are the kind of potential gotchas with this? What are the things that people should look out for, you know, if they're starting to explore using AI tools to improve their customers' experiences?
Nichole Mace: 22:28
Yeah, I guess I would say you know there have been some funny stories about chatbots providing the wrong information and hallucination, and it's not perfect. It is not perfect and you have to set expectations within your organization and outside that you know what we're going to be right 90% of the time and that's what it's going to be and that's going to provide value. So what I would say there is tell people, teach people that this is a technology that's going to help you. It's not going to solve all of your problems, it's not going to be right all the time, but once you embrace that, you're able to move forward and take advantage of it. We are in the world of AI assist. We are not in the world of AI taking over and running everything, building software from requirements that they figured out all on their own. No, we're not there yet and we're not going to be there for a long time. Is that long time? 10 years? Is it two years? I don't know, but we are in the world of AI assist.
Lily Smith: 23:27
It feels very similar to me, like the boom of user generated content.
Lily Smith: 23:31
A long time ago, Right right, Everyone was like but what if we let users say this to everyone? Then you know what are they going to do. They're going to listen and, like you say, ai assist like just being really clear that this is how this recommendation was generated or how this content was generated. And then, I guess, educating consumers. Luckily, your product managers are using your platform, so they're probably aware that you know that means it needs a bit of qualification or it means you know it might need a little bit of extra validation, like don't just go and take it all verbatim.
Nichole Mace: 24:08
For sure, for sure. You know transparency is everything today and you know, over time there will be some things that are lower stakes and they will shift. Right now, you know our examples Netflix is lower stakes, but business recommendations are high stakes. Maybe a chat response in a consumer app is low stakes, so it's okay, and others are more important and higher stakes. So I think it's going to evolve over time. But where we are right now is we want to be as transparent as possible and set expectations.
Randy Silver : 24:44
Nicole, thank you so much. This has been a lot of fun.
Nichole Mace: 24:46
Thank you. It's one of my favorite topics, so thank you for giving me the time to talk with you guys and share some of the cool things that are happening right now.
Lily Smith: 25:05
The product experience hosts are me, Lily Smith, host by night and chief product officer by day.
Randy Silver : 25:11
And me Randy Silver also host by night, and I spend my days working with product and leadership teams, helping their teams to do amazing work.
Lily Smith: 25:20
Luran Pratt is our producer and Luke Smith is our editor.
Randy Silver : 25:24
And our theme music is from product community legend Arnie Kittler's band Pow. Thanks to them for letting us use their track.
Comments
Join the community
Sign up for free to share your thoughts