What does it take to integrate Generative AI into the highly regulated world of banking? Join us as we sit down with Tariq Maonah, SVP of Product and Engineering for Generative AI at Citi, who reveals the secrets to navigating the complexities of AI-driven product development in the financial sector.
Featured Links: Follow Tariq on LinkedIn | Citi | 'Six things we learned at the Pendomonium + #mtpcon roadshow London 2024'feature by Louron Pratt
Episode Transcript
Lily Smith: 0:00
This week on the Product Experience Podcast, Randy and I chat to Tariq Maonah, SVP Product and Engineering for Generative AI at Citi. We discussed how Tariq and his team are approaching use of AI within their organisation and the lessons they've learned along the way. The Product Experience Podcast is brought to you by Mind, the Product part of the Pendo family. Every week we talk to inspiring product people from around the globe.
Randy Silver: 0:29
Visit mindtheproductcom to catch up on past episodes and discover free resources to help you with your product practice. Learn about Mind, the Product's conferences and their great training opportunities.
Lily Smith: 0:42
Create a free account to get product inspiration delivered weekly to your inbox. Mind, the Product supports over 200 product tank meetups from New York to Barcelona. There's probably one near you.
Randy Silver: 0:59
Tariq, thank you so much for joining us today. Live in person on the podcast.
Tariq Maonah: 1:04
Yes, yeah, really excited to be here. Thank you very much.
Randy Silver: 1:07
So people here at the conference here in London they've had chance to see you talk, they know who you are, they've gotten the introduction, but people on the podcast audience haven't. So can you give us a quick intro? Tell us a little bit about what you're doing now and how did you get into products in the first place?
Tariq Maonah: 1:21
Yeah, great. So I'm a Senior Vice President for Product and Strategy and Engineering at Citigroup. How did I get into product? That's a really good question.
Tariq Maonah: 1:31
I would say it was more of a natural transition. I would say I had a very much technology sort of analyst style work. I was really working in banking my entire career for the odd 20 years now and I was working more in the I would say support lines, working in banking my entire career for the odd 20 years now, and I was working more in the I would say, support lines, working on banking software, fixing bugs, that sort of thing. And then I moved more into when the digital marketing trend took off, I would say, and digital marketing was the big thing back in the early 2000s, sort of 2010. And the focus was all on experience. And that's really when I started merging my technical skills with experience, which then led into a product role. So I've come from sort of a marketing engineering background and then morphed that into product because product's very holistic. So it was a natural progression for me. I would say.
Randy Silver: 2:23
And the product area that you're working in specifically is generative AI. I wonder. You've been doing this stuff for a long time, as you said. How is working in this space different? How is development in this space different than what you had done previously?
Tariq Maonah: 2:36
I would say it's been the biggest curve in terms of learning curve that I've had for a while, because the digital marketing space stayed for quite a while it went on for a decade and as we've gone into AI, the style of product management is a completely different beast, because when we're talking about AI, you're talking about things that move very quickly.
Tariq Maonah: 3:01
So you could be working on a product and experimenting on something and then a week later there's a new feature, there's a new model.
Tariq Maonah: 3:08
It's really hard to track, I would say, where you want to be, because your goalpost keeps moving as the industry is releasing. And also, what makes it quite difficult is it's non-deterministic in nature, which means you don't always get the same response, which again opens you up to a wider range of, I would say, instability in terms of the metrics and what you're actually trying to measure. It becomes a lot more agile and you have to paint that picture more to senior leaders, who are probably more in the space of, I would say senior leaders, who are probably more in the space of, I would say, portfolio management, where they want concrete timelines or concrete, I would say KPIs, and it's really a journey for them as well, and you have to take senior leaders on those journeys as well with you to say look, you know, we're experimenting together. This could fail. This could be great. Let's just be transparent and see what works and what doesn't that's a really interesting point, actually.
Lily Smith: 4:06
And how, how has that communication gone? How like?
Tariq Maonah: 4:10
do you?
Lily Smith: 4:10
have some advocates for it and some, you know, detractors who are just like no, just do, do things the normal way no that's.
Tariq Maonah: 4:19
I've experienced that a lot in my career. I would say I've always been, I think, in the early stages of the product revolution. I would say I was knocking on doors that were closed and it was really hard. Half of my job or my span at those companies was just getting people on board rather than actually getting into the iteration it was. It was very much, especially especially in the financial services sector, particularly because it means seismic change for a lot of organizations that are quite structured. But at Citi it's been a complete opposite, I would say.
Tariq Maonah: 4:55
For such a large organization with such a large user base of 250,000, the attitude towards change is extremely agile, which I think you know. I've only been there two and a half years now, I would say. But that attitude to change really surprised me and that's what has led me to really want to invest and stay at Citi and drive development there. And senior leaders are open to change, they're open to working in agile ways, but also they're comfortable with not knowing the outcomes. There are obviously situations where they're regulatory, where you need to have an outcome you need to know that and obviously we have to fully be on board with that but by and large they're comfortable with. Let's go along with this and if it's going to fail, that's fine. We've learned from that and we can move forward onto another project. So a lot of the time, my team, we pivot, very often when we feel like this is we've experimented enough and we don't think this is going to add value, so we're just going to pivot and we have that conversation openly.
Randy Silver: 6:00
So what does that look like on a day-to-day basis? You know you're starting to work on this stuff People. So what does that look like on a day-to-day basis? You know you're starting to work on this stuff. People are committing resource, time, effort, people around you. They're making some plans and then you know it's a big company that's putting a lot of gears in motion and you decide to change. What does that actually look like in an organization like yours.
Tariq Maonah: 6:22
How does that change manifest? So, in terms of generative AI, specifically, because it's a journey that has been orchestrated, I would say, from the very top. Jane Fraser has been very vocally and publicly vocal about generative AI being something we're fully invested in. We know it's going to revolutionize not only our bank but the banking industry. We did a recent research paper that's publicly available on our website Generative AI and Finance and we predicted and seen that trajectory of growth and profits 9%. So the whole of the banking sector's profits will be over and around $2 trillion and that sort of impact in such a short space of time.
Tariq Maonah: 7:02
You have to really get involved, invest this. So it's a journey for the entire organization. So with that culture and that already set the tone set, it becomes a lot easier to have the conversations around pivoting. So, for example, if you're experimenting on a use case, which is I'll give an example of one that I've been working on where we're looking at code generation, and how say not, not very experienced engineers could probably take Code that exists and convert it to another language or make it more modern, using at generative AI to help assist you in that space. It's quite a Challenge we found with the models and having to convert code had a lot of human effort, I would say so. It wasn't as autonomous or, as you know, assisted as we thought had the conversation around.
Tariq Maonah: 7:52
Look, this is the journey we've been on. These are the stats we're seeing. These are sort of levels of human interaction still required and it still needs a lot of, you know, massaging what comes out of the ai. It's valuable, but not as valuable as we thought, and then we'll move on to another idea or another sort of use case and build a product around that. But the conversations right now, because the industry is still really in its infancy when it comes to Gen AI, it sets the scene for a lot more acceptance in terms of pivoting.
Lily Smith: 8:23
And you mentioned earlier that you know, as you're working on testing these emerging tools, they're kind of evolving under your feet, like as you're doing that. So does this require a different process? Or, like you say, you kind of like using the same process but then going is this right for now? For now, yes, no, park it if it's not, and then come back to it later. Or, you know, is there a different type of process for assessing and deciding on what you're working on and whether you continue to work on it?
Tariq Maonah: 8:56
yeah, yeah, I think that question also stems from my approach with product, because I've always advocated that. You know, less process is better in products, because you have to be quite fluid to change and you need to take on board feedback at any point. And when you think about the evolution of agile and the evolution of products, it was always very, you know, sort of scrum methodology, two-week sprints. I could never live in a world where I have to release two weeks. For me that is too long a distance. I mean I like to.
Tariq Maonah: 9:30
The team likes to release daily, every like two times a week, for example. So scrum just doesn't work in the Gen AI space. You know you need to just be fluid and you'll release as soon as you've got something that's passable, release it out to users internally and find out it's working. So you really have to be comfortable with no process effectively and the team has to be as well. So you have to ensure that the team is comfortable with ambiguity, but also comfortable with working very closely with stakeholders and with customers and working out if there's nibbles in something or you need to really ping pong a bit with the releases. That's where we have to be comfortable with effectively to operate in the Gen I space.
Randy Silver: 10:17
So you've been working in banking for quite a while before moving into this side of the development space. What's been the biggest surprise? What's something that just you could not have prepared for In the 20 years?
Tariq Maonah: 10:30
gosh, there's been such a change in banking in the 20 years. I remember even the fact that I had to wear a suit when I was building PCs in a room that no one saw me in and I had to be fully suited and booted and I just found that so strange. But it's really absolutely changed. So culturally it's changed, yes, but I think the receptiveness and you know, I know we're way beyond the pandemic now, but I think the pandemic was a bit of the catalyst for just accepting what you don't know is going to happen at a very senior level and just getting people to work across the silos and just come up with solutions to the remote working problem. And it's very similar culture. Now, with the gen ai, it's like this is hit. We want to be on that s curve, and that was in one of the presentations earlier about you know, being on the curve before you miss out. There's a real culture now that you get on the curve learn, experiment and get products in place until you're ready to release them out to the world effectively.
Lily Smith: 11:32
And, like you say, being in banking, there's a layer of compliance that you obviously have to be aware of and make time for. How is that impacting how you're working? Because obviously you know these are high risk use cases of Gen AI.
Tariq Maonah: 11:52
Yeah, extremely well, because we've seen this before in other guises, in other sort of waves, whether it was blockchain or other waves that come before this.
Tariq Maonah: 12:03
The risk and compliance side is very important. You've got to really be on that and we were already prepared to spin up a team, work with the risk departments, compliance department and say, for an average software product that you just traditionally deploy and get signed off by the risk and compliance team, that could take six months in a normal cycle where you're not on this sort of exploratory S-curve, but we targeted six weeks would even be too long. We need approvals, we need teams to get together more frequently and discuss the use cases and we need that sign-off process firstly to be as self-service possible but then also as quick to turn around. So even before we started looking at the actual core of the use cases or what the products we were going to build, we were also working with the risk teams to say, before we even go live with this and get the company working in this fashion, we've got the process in place where this is going to get signed up pretty quickly or will be complied, et cetera, pretty quickly.
Randy Silver: 13:08
Do you do that by changing the process or by recruiting people from Risk to be part of the team? An extension of the team?
Tariq Maonah: 13:16
It is actually more holistic, as you mentioned. It's the actual process. It's introducing fast track processes, essentially, and those fast track processes can then be used later on for other ways or other subject matter areas. But it was holistic change.
Randy Silver: 13:34
This episode is brought to you by Pendo, the only all-in-one product experience platform.
Lily Smith: 13:40
Do you find yourself bouncing around multiple tools to uncover what's happening inside your product?
Randy Silver: 13:45
In one simple platform. Pendo makes it easy to both answer critical questions about how users engage with your product and take action.
Lily Smith: 13:53
First, Pendo is built around product analytics, enabling you to deeply understand user behavior so you can make strategic optimizations.
Randy Silver: 14:00
Next, Pendo lets you deploy in-app guides that lead users through the actions that matter most. Then Pendo integrates user feedback so you can capture and analyze how people feel and what people want, and a new thing in Pendo session replays a very cool way to experience your users' actual experiences.
Lily Smith: 14:20
There's a good reason over 10,000 companies use it today.
Randy Silver: 14:23
Visit pendoio slash podcast to create your free Pendo account today and try it yourself.
Lily Smith: 14:30
Want to take your product-led know-how a step further? Check out Pendo and Mind, the Product's lineup of free certification courses led by product experts and designed to help you grow and advance in your career.
Randy Silver: 14:41
We're in more today at pendoio slash podcast, so this has obviously been a big change for the business, for you. Lots of people are making this transition right now. Lots of product people are starting to work with new technologies in this way. What's something that you wish you had known a year ago. That's made a big difference, something that other people can really benefit from as well.
Tariq Maonah: 15:07
Okay. So with that question, I say there is a misconception that you need to be extremely skilled in the Gen AI space as a product manager or technical enough to really know the ins and outs of how to use Gen AI. So I would say that, firstly, there are three courses that I found out later on about that you can embark on. For example, there's the famous Google pathway for generative AI, which really, even though they're geared towards engineers as a product manager, should go through those just because it's a new vocabulary and that's the biggest thing you know with any you know, even as humans. When you're introduced to a new vocabulary, it's obviously quite baffling, I would say. And in the Gen AI or AI space in general, it's a very different vocabulary. I think get familiar with the vocabulary, because that will help you when you're speaking to other departments or if you're working closely with data science or engineering.
Tariq Maonah: 16:03
And secondly, I would say the key to successful Gen AI delivery is secondly, I would say the key to successful Gen AI delivery is 90%, I would say, in the soft skills, because the main thing is the people who are bought into it, the stakeholders, the senior leaders. You have to take them on that journey. If you're lucky enough to be in an organization like I am, where are open to that and you know the friction is pretty low. But if you're in an organization where this is a new thing, that's startling your senior management or you know they're not really comfortable with the ambiguity, it's up to you as a product manager to really help distill that, make them comfortable, reassure them. You know, try something small, low risk, risk, prove a point. You might have to vary your approach a little bit, but I would say always start with something low risk and move from there and really give them that reassurance.
Lily Smith: 16:57
I think that's great advice and, as leader of this group that's working on Gen AI, how have you structured your teams in terms of objectives and projects?
Tariq Maonah: 17:08
That's a great question as well. I think the product space. Usually in an organisation, even outside of banking, product managers manage other product managers. So I'd say a product leader is usually a leader of other product managers. But to be truly agile and to be really flexible in this space, you need to either have the opportunity to manage a fully rounded team which includes engineers, content writers, et cetera. If you have the ability to manage a full sort of lifecycle team, that really helps because it's very close, the pace of change is very close. But if you can't do that, then you definitely have to partner with people who are willing to spend, you know, 50% of their time with you. They're an engineer. You need to find those people to work directly with you. That's the best way to structure things.
Lily Smith: 18:03
So you're in charge of the business unit, so you're in charge of engineering, product and design for Gen AI.
Tariq Maonah: 18:11
Yes, so there's various Gen AI teams across Citi. My one is a fully functioning, I would say, has engineering product, some design and content writing, but we're really focused on building the Gen AI tools, whereas other Gen AI teams are focused on specific slivers of that. A lot of the tools we have built have been widely used internally by the user base.
Randy Silver: 18:35
Tariq, we've asked this question of a couple of people today. But I'm curious when you're recruiting now let's take this not just for a product, but for the other functions you're looking at how much are you looking at experience in AI-related skills? How much are you looking for just general competence, curiosity, those kind of features and thinking okay, we can teach you the technical skill set?
Tariq Maonah: 19:00
Well, citi is one of the I think has some of the highest vacancies rates or volume of vacancies that are in the Gen AI space to really showing our commitment to wanting roles to be specifically around Gen AI and that doesn't mean just engineering, it means even in the product space, it could be anywhere. Just to let people know that that is the landscape now you will be working on those products. Me myself, I've got 25 roles open globally and that's incredible, but at the same time, so much competition and volume because everyone wants to be in that space now. So, yes, it's a balance. So I may find individuals who have the Gen AI capability from an engineering perspective or a product, and that's absolutely great and we'll look at that.
Tariq Maonah: 19:49
But a lot of people won't have that experience. It's quite niche and now it's becoming mainstream. And for me, it's more about the soft skills. It's more about can this individual showcase that they are comfortable with ambiguity? Can they deal with not having requirements and just building from an idea? These are the sorts of skills you need to be successful and the technical and all the knowledge will come after. And Citi at Citi and also, I'm sure, other industries we're very comfortable with that and my particular team is very built on those sorts of sensibilities.
Lily Smith: 20:23
I would say Are you finding, with the recruiting, like you say, you've got quite a few roles open but there's a lot of competition coming through because people are interested. Do you find that people? You know you're attracting people who are generally attracted to the next shiny thing. So it's like they've done maybe big data and then blockchain and then web three, and now they're they're like cool, now I'm going to do gen AI or try to do that is there. Is there an element of that going on?
Tariq Maonah: 20:51
I think I wouldn't say an element, I'd say it's quite widespread prevalent.
Lily Smith: 20:54
Yeah, it's prevalent yeah.
Tariq Maonah: 20:56
So I would say that there's, you know, cvs we get. The short listing has to be very. I think it's harder to recruit because you're looking for, sometimes, someone if they have the Gen AI knowledge, then you're thinking, okay, right, well, I'll interview this person and I'll go through it. If there's someone who's got no display of that on their CV at all, I have to look for signs of maybe they picked up a blockchain when it was on the rise and they did well in that space, effectively and they built some product that's been highly regarded or done well profitably. So it's a balancing act between if there are the individuals that are jumping on the next thing, have they proven that they in the past, that they're able to jump on something and be successful in it, as opposed to someone who's come from an AI background and is just looking for another AI role?
Randy Silver: 21:46
Okay. So, speaking of all these technologies and approaches that were very popular for a while and sometimes didn't really have a specific use case, it was more about the hype than the specific thing. The nice part about Gen AI is it does have real, applicable use cases. But how do you know, how are you evaluating that this is the best opportunity for you, that you're not a technology driving a solution, but you're looking for genuine opportunities where this is a good approach?
Tariq Maonah: 22:17
Yes, so effectively, what we've been driven by is the need from the business. In terms of before we even embarked on looking at Gen AI, it was more about what problems are there in the bank right now where, say, we can drive better productivity for employees or we can cut down a process time in over half, and those instant sort of opportunities where you just introduce even very small Gen AI changes and see an impact where I've saved hours or days on a task. For example, if it's writing a report that uses seven people and it's in a chain where someone has to write the first bit, get reviewed, go to the next person, you can basically close that gap and make it hours with Gen AI rather than just days of seven people working on one report together. So when you take a use case like that and you work on it, you build a tool around it and it's now being used.
Tariq Maonah: 23:18
The gains productivity gains are amazing and I think that in turn, has that impact of productivity. Gain means profits are higher and there's an attractiveness there and I think you can see that even just from a POC with other technologies like blockchain, for example, which obviously is revolutionary, but it's harder to see that instant benefit that you would get from that? I think yeah, whereas productivity gain. It affects everyone. It's a core criteria for anyone in a job.
Lily Smith: 23:48
And I think it's interesting as well the way that you have it structured within Citigroup, where you're not just approaching this like an innovation exercise. It's very much your new way of working, essentially.
Tariq Maonah: 24:03
That's an excellent way of articulating it. Essentially, that's an excellent way of articulating it because innovation and innovation labs I think sometimes they get a bit of a bad rep because they sit on the side of a business, not in a business, and a lot of what they do just gets stuck at proof of concept and there's no real buy-in then to push that forward, whereas when you're changing the way the company works and effectively starting with processes and saying this is going to be your new way of doing it we've built this tool that will help you do it and it's revolutionizing the way people work and that's really critical to then we are ready to externalize when we have confidence in the fact that the models and the robustness and the security around them not hallucinating or, in other terms, giving you bad responses or toxic responses. When that level of certainty is great, then we'll be ready to then externalize to customers because we've done so much internally. That's been tried and tested and it's working.
Lily Smith: 25:02
So we were having a chat with John Haggerty the other day on the podcast and one of the things that came up in the conversation was the fact that AI can't say no. So we were having a chat with John Haggerty the other day on the podcast and one of the things that came up in the conversation was the fact that AI can't say no, or like it can't tell you. I'm not really sure what the answer should be. So how do you handle that, knowing that you know the response is always going to give you something, even if it's not sure what it should actually be?
Tariq Maonah: 25:24
Yeah, so that was always something we had to be conscious of from the beginning. So when we were building our products and launching them out to the user base, internally we always ensured that the product had a way of rating the response and seeing if it's actually good quality. If it isn't good quality, we need that feedback from the user to say it didn't answer my question properly or it gave me an answer that was not related. It just wanted to give something back, and that's quite important to us because it helps us manage the prompts. Not to get too technical, but how we build the underlying applications often relies on a prompt that's hidden that we have built. So it helps us to refine those as well and make sure that if it's going to think it doesn't know the answer, it needs to stop there, the AI or it needs to come up with a suggestion that is in line with the topic at hand. So it's just fine tuning.
Randy Silver: 26:21
I think that it helps with Well, let us know when you've got something that works in real time and is bionic, because we could really use that in terms of our question quality sometimes. You can talk, yes, it was the royal we Been here too long. Tarek, this has been fantastic. Thank you very much. This was a great chat.
Tariq Maonah: 26:41
Absolute pleasure to be here. It was a great chat and see you soon, hopefully. Absolute pleasure to be here. It was a great chat and see you soon, hopefully another time at another event. Thanks, Tariq.
Lily Smith: 27:01
Thank you the Product.
Randy Silver: 27:05
Experience hosts are me Lily Smith host by night and chief product officer by day, and me Randy Silver also host by night, and I spend my days working with product and leadership teams, helping their teams to do amazing work.
Lily Smith: 27:15
Louron Pratt is our producer and Luke Smith is our editor.
Randy Silver: 27:19
And our theme music is from product community legend Arne Kittler's band Pow. Thanks to them for letting us use their track.
