Avoiding ML FOBO

Speaker 1 [00:04]

Thank you, Christian, for the introduction. Yes. So just before we start, we'd like to introduce ourselves. I am Dania. This is my colleague, Rachel. I'm a senior data scientist here in Berlin, moved from Brazil four years ago. And currently, I'm working for an e-commerce doing machine learning models for predictive analytics. I am also a volunteer at Data Science for Social Good Berlin. And as mentioned, I'm a teacher at a data science boot camp.

Speaker 2 [00:32]

And I'm Rachel. I'm from the US, but I've been working as a data scientist here in Berlin for the past two years. I also teach at a data science boot camp here in Berlin. And as Christian mentioned, both Dania and I are two of the co-founders of the AI Guild, which is the go-to community for data scientists, engineers, machine learners, basically anyone doing anything with data. And what our goal is, is to accelerate the adoption of AI in Europe. So if you're interested in hearing more about that, you can find us at the end.

Speaker 1 [01:01]

So, now starting with the topic. So, FOBO stands for Fear of Better Options. It gives a name to the spiral we fall into when we obsessively research every possible option when we are faced with a decision, fearing we'll miss out on the best one. So, it looks like we are always optimizing and living in a world of maybes. We can be paralyzed at the prospect of committing to something because we fear that we are choosing something that isn't the absolutely perfect option. So that can be overwhelming and it can block us. In the reality, we know that we'll never be able to examine every possible algorithm, package, technology, tool before we have to make a decision in our data science project. Specifically, when we're talking about data science, we can think about all the machine learning algorithms we have and how to choose from them. So here I have an overview map of the machine learning algorithms from this blog post of Data Science for Everyone. And we can see here it is split into four main categories, classical learning, deep learning, reinforcement learning, and the ensemble methods. Some of them even have subcategories. For example, in classical learning, we have supervised and unsupervised learning. It looks like it's a lot of options, a lot of choices to make. Maybe it gets better if we just focus, let's say, on Python implementations And even on one package So the most popular package for classical learning, for example, scikit-learn So yeah, we still have too many choices This is an interesting overview because it also tries to give us a path So we have a start And depending on the situation we are into, we can find what is the most appropriate for our case but this is only for classical learning. What about deep learning? So here an overview 11 different Python frameworks and libraries to work with deep learning. And we know that every day more and more packages are being released, new algorithms are being published. So how can we handle that. So this is one of the reasons why data scientists have this fear of a better option. So the field is growing very fast and it's always progressing. That means as data scientists we will have some gaps in our knowledge base. Yes, it is uncomfortable to feel like you don't have all the answers. When you are new to the field, especially if you're fresh out of academia or from a research background, maybe you're focusing too much on the technology and on the theory, not on the application. Managers also focus on the hype, something that we saw in the previous presentation, instead of focusing on the applications and the implementations. So when you're new to the field, everything seems equally important because, or it seems better than what you know, just because it's new. And this is something that we have to watch out, for example, when we are interviewing or hiring. So we have a story we want to share. Yeah.

Speaker 2 [04:14]

Yeah, so in an interview that I did actually not too long ago with a very senior head of machine learning at a pretty well-known company, I was asked, so are you the type of data scientist who would be coding a neural network from scratch and putting that into production? and i kind of thought about this for a second um and then this person elaborated and said you know because we're really concerned about our ml ip and obviously writing for example a neural network from scratch so in numpy a and putting into production is not um the best idea obviously there's so many well-tested open source um frameworks and packages but then also this is in no way your intellectual property necessarily if you're basically copying from scratch what has already been done. So yeah, a lot of times this does come from managers who really are also kind of following this hype.

Speaker 1 [05:17]

And, yes, that can be really overwhelming. So maybe the four solutions that come to mind are the most hyped ones, the ones that you hear about in the media, and maybe they're not the best-feeding ones. So how should we decide what to use? What if we make the wrong decision and we don't bring enough value from the beginning? Or what if what we choose to use isn't the state of the art? Don't despair. So this is what the presentation is going to be about. We are going to present here a practical roadmap for our data science project, split into three parts. We start in what to focus on first, then how to choose the algorithms, and most importantly, how to iterate on them. And the last part is about sharing some tips for convincing business leaders to focus on what works and not on the hype. So what to focus on first. Maybe you have seen this already, the data science pyramid of needs. It was published on HackerNoom. And we can think about this as in parallel with the human pyramid of needs. On the top of it, we have AI. And it would be the human need of self-actualization. And yes, it's great. But we as humans first need food, shelter, and water, which would be data literacy, data collection, data infrastructure. So here at the bottom of the pyramid, the first three slices, they're about data collection, creating a reliable data flow and data pipeline, and doing data cleaning and some data preparation. On top of that, we start aggregating some data and generating metrics, also collecting labeled data for training the machine learning models. And then the top two slices are about starting with simple machine learning algorithms and building a framework that allows experimentation. So something that we need to iterate on so we can gather feedback from the end users and also pass this feedback of the models to the business stakeholders. And as we move up with more complex algorithms, we reach the top of the pyramid with deep learning. And also, this is a very famous chart, and people that are working as data scientists can relate to this. A lot of the times, people think that AI is about coding and about the algorithms, about building neural networks. But in reality, we know that a lot of the work is getting the training data. And you can think about this because big companies like Google, Amazon, Facebook, they are publishing their research with the latest algorithms, but they're not sharing their data. And getting training data might seem like it's a simple task at first, but I can give you an example. Let's say you have a big catalog of, for example, wine, and you want to use machine learning to uncover what is the best wine from this catalog. And you're saying, well, okay, to start with this, we have to collect some data. So let's say we will try to find what is the best wine, but what do you mean by best? So first we have to agree on a definition of best. Let's say we agree on a rating from 1 to 10, and because we want to collect data, a lot of data, and we want to collect it fast, we ask everyone here in the room to every time you're drinking some wine, take some notes on the name of the wine and put the rating on it. So after a couple of weeks, we have a lot of data and we can use that as a training set and train our model, right? what can we expect as the performance of such model? It's tricky, right? Because the taste is very different from one person to the other. So it could be that we have the same wine with different ratings. And it could be also that the variance of the ratings are pretty big. So something that tastes good for me is awful for you. It's also tricky because what constitutes a good wine might not be something that you can quantify. For example, the smell. We know that the smell affects the taste, but how can you quantify smell? So that means a task that sounded simple in the beginning might need some iteration, and you might want to change your strategy. In this case, for example, instead of having one model, considering everything as one training set, maybe you want to have different models. Or maybe you want to take the ratings for the same wine and take the average of them. We just need to remember that the machine learning is only one part of the solution. So here there is this data science environment diagram, which was published by the Google researchers. We have 15 different components in data science, and machine learning is only one of them, and a small one. And it's one that cannot live up by itself. So it needs everything around it. We have to think about a data science solution as building a system end-to-end. Okay, cool. So now you know that you have to start by understanding the parameter of needs and where you are, and also understanding where you want to achieve. And this is not done only with the data scientists. This should be done together with the business, so they understand what is the process that you are trying to optimize or what is the problem you are trying to fix. We should also involve the data engineers or the software engineers in the process to understand what is the data looking like, where is this final solution going to live, and how it will work. Now that we know where we are and where we want to go, how to get there, what is the path looking like? we have to start choosing an algorithm and it really helps when we think about the entire process as iterations so taking almost all of the 15 components of the data science environment and organizing them in a sequence we can think about two main loops so we have the big loop which is for the final solution so where we're starting defining the goal and defining the metrics, defining what should be the minimal viable product, and then we start the development cycle, which contains the collection of data, data preparation, training the model, and evaluating the model. Once you run through this cycle of iterations, and you have a model that fits what you defined at first as a measure, then you can go for the deployment, start making the predictions, and you monitor your system together and analyze some insights. So the idea here is to really have this fail-fast attitude. So you want to debug, iterate, and experiment. Instead of perfection, you should focus on iteration. Have a good enough model to start with, and then ship it, get some feedback, improve on it, and repeat the cycle. The more you do it, the better it gets. It's the snowball effect. you'll better understand the algorithms you're working with. You'll be better at anticipating how long it takes for you to finish your projects. You'll be faster to spot low-hanging fruits for model improvement, if it's a new model, if it's on hyperparameter tuning, if it's on feature engineering. Also, it's easier to stay motivated after the first results that might be poor, but when you understand this is only the first cycle of iteration and you're doing a lot more, then you are improving. In the end, you are going to be able to solve bigger problems using machine learning.

Speaker 2 [13:11]

Um, so when you're very first starting, so say like you have your use case, you're starting to model. Um, one of the things that as data scientists, we forget a lot of the time is to start with a lazy estimator. So this is something that you're, that's not really going to give you, um, a ton of, um, predictive power, but it's really to set the baseline and figure out for all of your future modeling, if that's actually an improvement on the lazy estimator. So for regression problems, this means always predicting the mean or the median, so that's what this is here. You have the actual target variable on the right, and then we were just predicting the mean. A lazy classifier is always predicting the most common class. One example could also be, for those of you who were at the talk just before this, when Alexander was talking about the typical Titanic problem that everyone knows, if you just predict that women always live and men always die, you get 80% accuracy. So then that helps you kind of figure out, okay, when you have 81%, if you hadn't done some type of lazy estimator like that, you might think, oh, 81% is pretty good. But if you're actually only getting 1% improvement over a lazy estimator, then that's actually not such a great improvement. And so any machine learning or any more significant model or more complicated model that you use should very significantly outperform this lazy estimation.

Speaker 1 [14:31]

Thanks.

Speaker 2 [14:33]

But then a lot of the times, too, it can be difficult to know where to go as your next step, so actually modeling as a data scientist. There are a couple of good rules of thumb, for example, with binary classification problems, always starting with logistic regression. For regression problems, starting with a random forest is usually a good rule of thumb, but it's important to remember that there's no one model that works for every problem. um but another really good rule of thumb is to not start with a neural network why they take a very long time to actually um train and tune so when you're working with neural networks you're not just tuning hyper parameters you're also tuning architecture and there's almost an infinite combination of architectures and hyper parameter combinations and this just takes way too long. When you're coming from a place of fast iteration, failing fast, you want to just get something working that can be up and running really quickly. Another thing is that neural networks only really show their true power and their true success with a lot, a lot of properly labeled data. But this talk is definitely not anti-neural networks. Obviously, deep learning has made amazing strides and really improved a lot of use cases. But that's just the thing. It's very good for very specific use cases like image recognition or, again, when you have really a lot of data. But for the typical first modeling that you need to do, especially in a lot of company settings where maybe it's not a specifically AI-focused company like a company that's really focused on autonomous driving, but rather, for example, an e-commerce company with a specific use case wanting to use machine learning, most of the time a neural network is not the place to start. Another thing that data scientists sometimes forget is to try ensembles of models, particularly also using different models or even if it's the same algorithm, so train on slightly different data for different possible distributions within your data set. So a good example of this might be sales data that's drastically different on certain days of the year or week. So say, you know, you're a retail store and you have drastically more sales on Saturdays and on holidays. Sometimes it can make sense to train a separate model just on those days. But the point with all this is to really focus on getting something started right away. And then anything that you do on top of that, so whether it's feature engineering, whether it's trying different algorithms or training different models, that should be with a purpose to actually cause improvements over some baseline that you have. So it's really focusing on what is the output? Why do we need another increase in accuracy? Or what metric are we actually trying to improve? And why does the business need that rather than focusing on the technology? Like, oh, okay, I really want to do deep learning. So yeah, those slides were really about, as a data scientist, kind of cutting through the hype and knowing where to start when you have a use case. But there's kind of one level above this, which is how do you define that use case? So like Danya mentioned, when you have a person coming from your company and saying, I want a model that predicts the best wine, then you kind of need to work with that person and say, okay, how are we going to define the best wine? So setting your expectation of what is the metric, where is this actually going to be used? Because a lot of the time, most of the data scientists in the room will understand this feeling. um of a business person just coming and saying i need a perfect model that predicts x and then the first question should be okay but where is that prediction going to go who's going to use that is the customer going to see it is it an internal model for say the marketing team who's the end user um and then once you have that use case then figuring out okay is this something we actually need machine learning for to start should we start maybe with just a rules-based approach So once you have that use case, this is kind of a managerial side, something that's becoming more popular. So in a lot of companies, it's still the case that data science teams are very siloed. You have all the data scientists sitting in one place. Sometimes they kind of work with one business person or something like that. But we're moving more toward this idea of execution teams. So this means that for a given use case, you build up a team of product owner, data scientist, data engineers. So data architects, that's people, again, working on every step of that hierarchy. UI designers, basically, you're really trying to integrate across the entire organization and make sure that it's something that's an end-to-end product. um and something else that's really good for really focusing down on and deciding on a use case is this ai product canvas project canvas excuse me um so donna uses this in her company i think it's a really great tool so the square in the middle is highlighted because you always start with the value proposition and then you can totally see how when using this something like show me the best wine really breaks down if you make that person say, okay, so what's the value proposition of that? Is just having the best wine, no value is coming from that. So what are we actually doing? Is the value proposition, okay, we're going to show the best wine for that given person to then make them buy more wine and then increase sales. Then the value proposition is very clear, but that's also a very different problem. And then you've started to actually narrow the focus of the use case.

Speaker 1 [20:45]

Yeah, and also then you would have to go back and change how you were collecting the training data, for example. So when you have this defined from the beginning, you can iterate faster. It helps you to be more effective.

Speaker 2 [20:56]

and you can also um as you start to fill this out then maybe that changes how your execution team will look so as you start to fill this out and you realize for example following the wine example again um if we decide okay we do want to make it um really a recommender system for kind of each customer maybe you need um someone with kind of specific skills there on your execution team yeah so this is what we've um gone through so far so for so these most of these are really um tools for particularly the data scientists in the room but also managers so if you're a data scientist bring some of these things to um to the business folks that you're working with and the managers when they say hey i think i could have a use case um try bringing the ai project canvas to that meeting. Try bringing the hierarchy of needs, looking at, okay, for this use case, what data do we actually need? Do we have that? Do we need to work with maybe a data engineer to collect more? For example, maybe adding new points in Google Analytics, something like that.

Speaker 1 [22:09]

And just the final takeaways. So we split for data scientists and for decision makers.

Speaker 2 [22:16]

Yep. So the first three are separate for each. For data scientists, again, really, really focus on something that delivers value. And you should always be thinking, where is the prediction from this model going? Because if it's just like dying in a Jupyter notebook, that's not helping anyone. So what team is using it? Who is the final recipient of whatever prediction? And that will really help you focus on iterating quickly, getting something out there and really focusing on why you actually need those improvements if you need improvements in your model. Make sure you're communicating with business people throughout the whole process and get used to showing beta versions and showing, okay, we've done this so far. We're here on the hierarchy. We have the data. This is what we did for the past two months because otherwise it can be hard to keep business people involved and excited. and also make sure you're integrating with those teams.

Speaker 1 [23:15]

Yeah, for decision makers is to understand the machine learning workflow and the iteration in cycles is different from software development workflow and how it is important to define the use case and the metrics from the beginning, because this will allow for the iteration to be guided, let's say, having a true North Star. But for both sides, data scientists and decision makers, it is important to integrate with the other teams, so to work together cross-functionally, to set expectations from the beginning, and to have the mindset of experimentation and iteration. This is what we wanted to share with you. Thanks for listening to us. Thank you. If you have any questions...

Speaker 3 [24:03]

we have time for questions then there's one

Speaker 4 [24:19]

Thank you for your talk. It was quite interesting. I wanted to ask, let's imagine a situation when the company is not really data-driven and only the team of data scientists, it was just organized and the executors, they don't really understand yet how they can define clearly what they need to predict. So how would you advise, like, who should make these decisions? Should data scientists, like, prepare some kind of, like, detailed explanations of things they can do? Or how can it be, like, organized if there are just data scientists, no data architects yet, no people who understand how it's all, like, conducted with the means of data science and machine learning?

Speaker 1 [25:08]

Yes, I think that the beginning should be really about educating and having this overview of what a data science project is and all of the components that it involves. So maybe you as a data scientist, you are focusing on the machine learning part, but you need the support from everything else around it. And one way that I have been practically doing this is using the Canvas. So when you have a meeting, let's say a kickoff meeting for a project, and you have there the data scientists, if you have also support from engineers, you can have UX designers if it's a facing customer application, and the business people. So you have everyone together in the room and you try to figure out first what is the value proposition, to understand what is the value for the project. And it looks like it's a simple question, but I have experienced being in a meeting for one hour just to fill this one part of the canvas. And when you have this value proposition understood and agreed between everyone, so everyone is on the same page, then you start going to what is the data we need for that, what are the kind of skills we need for that, if we already have them, what is the output going to look like, how this will be integrated, and who else is involved as a stakeholder and as customer. And maybe you'll figure out, yes, I'll have some costs, but I'm also generating some revenue. So this is a good framework to iterate on this.

Speaker 2 [26:31]

I don't have a mic, but I can kind of yell. Yeah, I think this is a big thing, just showing this to a business leader, because a lot of the times they're not really involved. Like they just say, okay, data scientist here is what I want. Go away and do it. And then this really encourages them to be involved and realize, like, what is this actually going to take? What do we actually want to get out of this?

Speaker 1 [26:52]

Yes, and because we never know what we are going to have before we look at the data. And we know that. We are clear on that. But maybe the expectation on the other side is not like that because they have never worked with data science before. So it's about understanding that we have to iterate. Maybe also on the business value we will iterate.

Speaker 3 [27:13]

Thanks for the great talk, first of all. I have one question in the pyramid that you described. When you finally get to do the AI part, what is the, what you produce, the artifact? Is it an analysis that allows people to make more informed decisions or is it a product in the sense that gets integrated into a software and is supposed to run for a couple of years?

Speaker 1 [27:39]

You want to finish it up?

Speaker 2 [27:41]

Yeah, it really depends on your use case. So that's kind of the thing is why that's at the top is because at that point it should be something. Thank you. So yeah, just to reiterate with the mic, why it's at the top is because at that point it should be something that you really need AI for, deep learning for. So in that case, for example, it's really getting like that last improvement and accuracy on something that was already there that's already out so for example with our wine prediction um there's already been something rolled out maybe with a simpler model um but maybe it's just not getting that such a great increase in sales and so then adding deep learning at that point would be to actually you know get that x amount greater sales um so it's hard to say really like ahead of time until for each use case until you're actually there, what it should be. Yeah.

Speaker 1 [28:40]

Yes. And if I can add to the wine model example. So when you just get asked as a data scientist, you get asked, we want a machine learning model to predict the best wine. And you don't know what is the use of that model. So it could be that we are producing wines and we want to understand what are the attributes that constitute the best wine so we can produce that best wine ever. And for that, maybe you don't want to use deep learning because you can't really explain the result. but and on the other hand maybe you just want to have a really good personalized experience for your end customers and you want to predict the best wine so they will buy the wine from you then it don't it doesn't really matter how you can explain the model it just matters that you get it right so that's why also it's important to think about how it connects to business in the end of course as with data scientists we also want to work on the most accurate model possible That is always the goal, but it's good to understand how is this going to connect in the end with the business.

Speaker 3 [29:43]

I have one more. Do you have any recommendations if I want to raise the awareness or the understanding of this whole data field among the non-technical staff in my organization? Stuff like there was this thing a couple of years ago that the government of Finland decided that we want to train 1% of the population in data literacy. and it's not happening in germany at the moment as most of you probably know but we have to start somewhere where would you start

Speaker 2 [30:21]

So there is

Speaker 1 [30:22]

There is a really good course online, free, in Coursera called AI for Everyone from Andrew NG. So that is about explaining the basic terminology and what is AI capable of doing or machine learning capable of doing, what is not capable of doing, and try to demystify a little bit what we hear on the media about deep learning and AI taking over all the jobs. So that is a good start.

Speaker 2 [30:50]

I think, too, for business people and especially managers, the important thing is to really start with what it can do for the business and also to really like not focus on a lot of buzzwords, because especially high level managers do not want to feel stupid or like they don't know what's going on. So you really have to start with them of, okay, the business is already doing really well. Here's what, you know, kind of help them feel comfy and then say, okay, here's the, what, what use cases we could actually use. Cause then it feels so much more real.

Speaker 1 [31:23]

Yes, and building up on the presentation from before, it's also about building trust and communicating. So everyone is working together to have the best solution. It's not only the data scientists or not only the business partners.

Speaker 3 [31:41]

If there are no more questions, then let's thank the speakers again.