Lessons Learned as a Product Manager in Data Science

Speaker 1 [00:02]

Yeah, so just a small disclaimer, I'm currently sick. I have a flu, so I'm running on medicine, so maybe I'll die. But anyway, I won't die. So basically, in this talk, I will tell a small personal story and how I got there and why. So basically, I have worked in data science and in engineering for over 10 years. I have a PhD in information retrieval from, like, when there was no data science times. I would now put it on my CV like I did AI for over, like, 15 years if I would do that. Anyways, so in all this time, for, like, a very long time, I actually thought product management is, like, bullshit. It's like, ah, it's, like, such a useless, like, we, I worked in teams where we basically, we don't need one. We didn't have one. and um and then like about two years ago i became one um just did it for one year until we find somebody to replace we just we needed to do it we needed in the team we had this need so so i'm gonna tell you the story about like how like i did this job without actually having like any um schooling in it um and uh yeah so basically so i work at free now and uh one of the products that we have is called a fair estimator who here has ever used a taxi before okay by app yeah so it tells you how much it costs when you put the address right so that's and you're glad when it's accurate right so basically what is free now this is a mandatory slide um so it was founded in 2009 in hamburg it's actually older than uber um we have like a lot of drivers some of the numbers might not be totally accurate so we are like operating in over 100 cities in europe and uh we have uh developers locations in hamburg barcelona and berlin we have over a thousand employees now and out of which i think we have over 40 dataists dataists is everybody who works like in the data department marketing intelligence bi data engineering data science hamburg and berlin and uh so basically that's all about free now so basically what like estimating the price of a trip if you think that we are operating in over 100 cities so you think like you can calculate in your head for berlin like you already took a taxi maybe once you came here but try to tell me how much a taxi would cost in rome now right you've they have a completely different way of operating they have a completely different infrastructure in the city they might have way more traffic jams than in berlin who here has been in london have you ever taken a taxi in london or like a bus so so in london it's better to take public transport than take a taxi because you might not make it in time so um all of these things go into like basically you end up having for each city you need a different model right And so we basically, we used to have a solution which was implemented by a lot of software developers and various product owners and various C-level people. You know, the company was small and everybody had their stake into how algorithms are and everything. So this was like very rule-based in the database. If this city, then do this. If this city, do that. Then add this, multiply by that and something like that. So they came to us and said, can you help us? So we said, okay, let's do this. We help you. So we had our first meeting, and we said, yeah, I mean, this is like a linear problem. Look, you can calculate it in your head. If you go to a city, you can say, like, if it's five kilometers, and you know how much the taxi costs, roughly you can say how much it is. No. This cannot be. We tried all the possibilities already. What do you mean you tried all the possibilities? How could you have tried all the possibilities? Like, yeah, no, this definitely has to be done with neural networks because this is, like, a very complex problem. So I was, like, I wasn't, for the first meeting, I wasn't really prepared for this kind of dialogue. So then we, this was, like, a very circular discussion, and they were also, like, yeah, and then we can evaluate it. We can do A-B testing on the users to see if it's, like, but we don't need to do A-B testing on the user. We have the data. We can say already if it's, no, but we do testing on the users. No, we don't do testing on the, So this went on and on. So basically, so we had another meeting where I went like a little bit better prepared. And I said, okay, let's look. We're going to do the first thing as a linear model. You could not have tried all the possibilities because when you say all the features, I think other features. When you say five o'clock, I mean five o'clock Friday. It's not the five o'clock on Saturday. And you go to the city center, you come back from the city center, all of these things, we will put that in the feature pool. so and we will build a simple linear model and we will put it in the old application so the next one we do again the linear model but we build a service for it so then we own it and we manage it and the product owner of the other team wrote down the minutes meeting minutes the meeting minutes came okay version one two version two version three neural networks how did that one get there again so anyways so this was like our first try at doing actually a data science project in a company that has been there for a while and was not very data sciencey so basically what I learned is and now I'm going to talk a little bit about the product manager role so I have this nice quote I spend a lot of time searching for quotes so I think this is a product manager it's like a lot of people into one, right? So it's engineer, designer, and diplomat, right? So let's take a step back and think about this as a metaphor. So who here is from Hamburg? Okay, you know, we have fish. So the ones that are not from Hamburg, well, maybe you eat fish, like Barcelona anyways so so let's take the market vendor he sells fish he knows everything about what he's selling he knows whatever it is to get that fish to sell it he knows he cannot sell fish on monday because nobody's delivering fish on sunday so he's not going to sell fish on monday he also knows um like if you come as a customer to him he will know how to treat you like you say i want to do fish and chips then he's gonna give you like whatever nobody cares about the fish or you say i'm gonna do like sushi and then it's like okay like then you have to get the best fish because you're gonna eat it raw and then he knows how to treat you like somebody who knows about fish or he knows how to treat you when you don't care about fish so so basically so and in order to sell the fish have you ever bought something from a merchant who's grumpy angry no so he's like always in a good mood and he's like yeah even if you insult him he's gonna be like but the fish is good and and so so it this all the all of the skills we have like we have the same kind of situations in industry when you deal with all sorts of people at all sorts of levels so from this one i'm taking uh going back to the product manager and going to what are the stakeholders of the product manager so on the first stakeholder that i can think of is the user and the user is not somebody who works in the company the user is somebody who uses your app or whatever you deliver as a software data science software or non-data science software so basically unless you're building an app that is on purpose a machine learning product where users actually know that it's machine learning serving them and then they are really picky and like really paying attention to what happens to their data how it gets they really want to know unless you're doing that actually users don't care how you got there all they care about is that they have an accurate they have a perceived seamless natural experience with whatever you're giving them so who here remembers good job recommendations? One person. Who here remembers bad job recommendations? Huh. So the same kind of intuition happens to our app users, right? So if you tell me, like, I'm going to take a trip, it's 10 euros, and you tell me 11, it's fine. You tell me 5, and it's 10 euros, it's not fine. I'm going to be like, I didn't sign up for 10 euros, I signed up for five euros so so from the user perspective you actually have to come up with a metric that is a user metric so this is something that not all the data scientists are usually equipped to think about it like that because this is like a business it comes from the business world so you have to think like what is the user going to do with this and when is the user going to be unhappy and leave your app and when is the user most of the time the user doesn't doesn't care as long as it's bad when it's bad then it's like everybody sees it and everybody talks about did you see that it was awful i used to work for xing and we had like i had friends and they were like come on i got like an internship offer i'm already a like senior why is it offering me an internship as a job i mean some people do internship after senior but that's not everybody So everybody knows that when something was wrong. And basically, this accuracy metric that you're defining from the user perspective has to stay with the product that you're building the whole time. And you have to basically come up with it and figure out how it fits to the machine learning part. The next stakeholder is other product teams. You're usually going to interact with other teams that have done things before, and they know everything and they've already checked everything, right? So you cannot just tell them, we are going to do it better. Like, if you start like that, they're like, what do you mean? We are stupid? No. So you're going to have to say like, yeah, you have to make them feel like you're working. I mean, you can actually, you don't have to pretend. You actually have to do it together, right? you have to involve them in the decision making about how and what should be delivered and they have to be on board with the user metric they have to be on board with you tell them we can deliver this mvp in two three four weeks or well if it's longer than that then they're gonna be like okay so break it down into deliverables and make commitments that then you keep and also make it very clear that the initial approaches are going to be rather simple rather than complicated, right? And, yeah, so, oh, yeah, and actually the most interesting thing, who here works in a company that was invented more than six years ago? Okay. Do you do tracking of data? of yeah so in order to whenever you build a data product you need to track what you give the user and what the user does with that information because otherwise you cannot really evaluate so if you cannot really track the fact that you showed the user that the trip was going to be six euros and the trip was actually eight euros and then the user one month later is still not booking a trip because they got upset you cannot really know were you it was it your fault or not or you worst case scenario is that you say six euros and the developer team that is actually doing facing the user things that they know that they should add 20 to that and then they change your numbers and if you don't track you don't know about that and this happens so this can happen so you need to basically push for tracking for everything that you do as a data service as a data product you actually have and this is not a nice job to push for tracking because nobody wants to do this seemingly useless kind of coding that doesn't have like immediate impact on anything so anyways the other stakeholder of a product manager in a data science team is the are the data scientists themselves and this is where i think like in general it helps a lot when the product managers are data literate so because you have to talk the language and you have to talk the language of the data scientists and you have to understand how do you translate the user metric that you promised to the stakeholder how do you translate that to machine learning metrics and you also have to be able to define and to communicate to the team because you agreed upon it with the stakeholder when is good good enough you know so of course we can do rocket science if we really wanted to do rocket science i mean i mean it would take me like a lot of years to go into the field but i could but nobody would pay me for that right if i would say no i would no okay so what i mean is that it's not always obvious when good is good enough and this is something that you sometimes can get lost in translation. It happens also for normal software engineering teams when they do what is called over-engineering. Over-engineering happens also in data science. Over-data engineering, we can call it. And so basically, yeah. And yeah, so that's what we're saying. And this is one of the biggest, like the most important thing, one of the important thing when you talk to, um both data scientists and product teams do not promise anything time wise until you have an idea if it can be delivered or not in one sense so who here has worked in a place where you had somebody that said this can be done in a month and it was like impossible yeah so without even like exploring the data like to realize that like you don't even have the data like how like are you gonna like do it in a month so again depends on how the company was built in the first place and so on the last stakeholder this is the funniest one is the upper management so the upper management you see this is the magic book the upper management has this opinion about data science they they were like they they read all these blog articles and medium articles and they went to some conferences and they learned that neural networks are going to solve the universal problem and come up with the number 42 and they basically invested and they said we need data science teams to get there now and then they're like why is it everything why is everything so slow we read it on medium it's fast so basically and this is why a data science team has to always present their results intermediate results bad results good results bad results we always learn a lot from the mistakes and the bad results but we rarely communicate them and then if we do not communicate them there are people who are like wondering like they remember oh but we have a data science teams oh i haven't heard them in like two months now what are they doing we shouldn't hire anymore anyways so so basically a lot of presentations and you also have to be able to have like you have your data product and if you can actually translate that all the way to business kpis to to find a way like approximately if we do this right then we have this increase in whatever people care about then this is also really powerful to communicate and and the other one is also that you have to explain that a lot of data science projects and ab testing and offline evaluation is really useful because then you end up testing on the data and not on innocent users long term you're not upsetting your users so i made out of everything a little like mind map about the data the data science product management, all the things that one has to do. So basically, you need to be able to gather user insights and translate them to data ideas. You need to speak the product language because you need to speak to the other product teams. You need to speak the data language because you have to speak with the data scientists. And you also need to speak with the number language because you have to talk to the upper management where you present often and you have to push for tracking you have to have an accuracy metric and this is this accuracy metric again i'm stressing it like half of the word is gone it's a perceived accuracy metric it does not exist in any mathematical book right it's yeah so to summarize he said it yeah but back in the days so i added a little bit uh yes so so basically coming back to my opinion from two years ago because this is something that a lot of the people are like a lot of the time i hear data scientists say like we've been in this company for a long time and most of the things we've been working on do not make it to production so a lot of the work that goes to moving something from a jupiter notebook or like a little bit of coding to actually getting it to the user out there is involving a lot of communication with everybody and a lot of product management whether you like it or not so so basically if you actually want results you cannot have an attitude or a mindset where you say this is not my job depending on the size of your team sometimes as a data scientist you might have to do these things and it's maybe not fun maybe it's fun for some it's fun for some it's not fun and um you will see so basically if you want to you know these are like my stuff i have a twitter account i use it quite a lot since i deleted facebook um i'm organizing the hamburg python pizza conference 9th of november in berlin in hamburg sorry the next day there's a panel coffee and cake with The PyLadies with PyLadies Berlin, PyLadies Hamburg, PyLadies Munich are also going to be there. And otherwise, I do artsy stuff, specifically with dinosaurs, because I can always invent dinosaurs, and nobody has to ask me about the names of the dinosaurs because I invented it. So, yeah, that's everything that I got about product management. If you have questions... Questions? There is one already. Hi, Teresa. Hi. Congratulations again for yesterday. I actually forgot what it was, what your title is. Anyway, can you talk a little bit about how your work has improved since you started taking up the role or since someone started taking up the role of project manager? Well, I didn't have to write Jira tickets anymore. So that was like, for me, I was like, I said, so it was like this, we always had this dynamics between, you always have in the team dynamics between product management, product manager writes a lot of stories, writes a lot of like stakeholder stuff. And then the team itself goes into, we don't have enough autonomy. we want to like do this stuff and then like when i was the product manager i said like you know what yeah like just do it i don't do it then you do it and then nobody was writing stories so i think yeah so it's i don't write gyro stories anymore and actually our product manager he likes talking to stakeholders and goes to all these meetings and then i don't have to go to so many meetings i go to other types of meetings but yeah i changed one i don't know if it improved it's just different hi so i had a question thank you for your talk um and i was wondering about the role of the product owner and in your company and then the product manager and then like you it sounds like it's all mixed in or something i don't know can you i don't i'm also not sure about the definitions so that's why i i may i condensed product owner and project manager into product manager i try to like mix all of them together it's like uh it's it's like one person doing um project management but that's also the product ownership part where you go to stakeholder and then you collect the requirements from the stakeholder. We could have two, but we at the moment have one. Question for me. Thank you very much. I think that was a very inspirational talk. In software development, we have the agile, or at least we try to make an agile mindset and also put this into how we develop. What's your take on agility in data science products? Yeah, we try. How? So, So far, we align to the agile of the company. So they do two-week sprints, and we do two-week sprints. And I think what we are struggling with, is do we do for more data exploration stories? Does it make sense to assign Fibonacci numbers, or should they be time box? And this is something that there's all sorts of opinions coming from all sorts of places. And I wish somebody wrote an Agile Manifesto for Data Science book. Yeah? Is it good? Good. Okay. Thank you. I'll check it out. How do you plan, like time-wise, especially when you're trying a new algorithm or something, like how do you plan as a product manager for a data science project? yeah so so i think that would be the time boxing thing they say like spend one or two weeks and see where you get what you get like you don't have like i want to have the root mean square error here but i want to see like with just exploring what do we have in this problem then if it's interesting then you would obviously give it more time after the time box if it looks like it could work but then that kind of mixes up your whole roadmap right so long-term planning is yeah it's a nightmare yeah it's it there is no as yeah i will read the book i will check the book out because yeah we we are struggling with that because we try to be a um a team that does both software engineering and analytics that we actually put our models in production and that's bringing even more not conflicts but complications, right? There's benefits and it's still complex, yeah. Cool. What was your biggest surprise after transitioning into the product manager role? hmm well yeah so i realized so so for me at the time was like yeah why is everybody like having such a bad opinion about these people uh then i did it and it's a lot of hard work and it's like also like very ungratifying work so like nobody says thank you to the product manager it's like what did they do right just talk to some people so um yeah so i think um I think the biggest the most interesting challenge and this is one colleague gave me the advice when I started he said like you have to treat your team as a stakeholder they are not they are your stakeholder you have to convince them that they should do stuff and it's not like uh just tell them because this has to be done it's you know you have to treat them also as a stakeholder and that was like something that I didn't like know before I will have a panel in an hour In the big hall Awesome So there's one last question Hey, thanks for the talk Do you think you can be a good product manager for a data scientist project without being a data scientist beforehand? well considering that these days anybody can learn data science right so if you basically upskill yourself a little bit to the to the level like you would say you would not be comfortable applying for a data science role like but upskill yourself in us to the purpose of like having an understanding on the ins and outs of a data project, and you have a background in project management, then I think if you come into a team that is willing to help you where you need help, then yes. It depends on the team, then yes. Everybody can learn data science. That's a strong statement. Thank you, Teresa.