Build TikTok's Personalized Real-Time Recommendation System in Python with Hopsworks

Speaker 1 [00:06]

Thanks Anna. I don't want to contradict you, but you're fine to walk out. I won't take it personally. It's cool. I know how it is. Right, I spent the last few minutes re-implementing this workshop for Colab because I didn't realise the internet sucked. Are we good? in the middle so people on the recording okay yeah sure um so um so let's get let's jump in so what we're going to do i think i always like to start at the at the end and work our way backwards is we're going to do a demo of a tiktok recommender system and i'm a bit i'm going to hack up a bit here so let me see while i'm here hopefully the internet's good tiktok streamlit run okay let me just pull up in the window here so what we're going to build is it's not it's nothing too fancy right it's just um this one here and you basically everyone kind of knows how tiktok works you click on videos and it shows you some suggestions for videos you have like timelines that you follow and things like that and but what what it's well known for is having a really interactive recommender system. So what that means is that you click on a video that's sport or it's entertainment, and within a little bit of time, it adapts its preferences to basically knowing that you want to look more at this particular topic than another one. OK, so what we're doing is we're going to write five programs. And they are notebooks, and I did have the link up. Let me show you the link up again. You'll find all the links here on the slide. So if you haven't taken down that URL, please note it down now. And in there, you'll find a link to the docs. So the docs look like this, if you find it. Build a TikTok recommender system. I had designed it originally to work on Jupyter, but it ain't going to work. Look at the number in the room. I measured the internet here. It's a megabyte per second. And we're going to install TensorFlow, which is about 600 megabytes. So it ain't going to work, is the way I look at it. Right. So let's get started with what we're going to talk about. And the internet seems to be very slow here. So we'll let that work up in a sec. TikTok is really good. I have two boys who are 12 and 14. The 14-year-old has 30,000 followers. Apparently, he does really good slander. He doesn't get it from me, but anyway. The reason why it's really good is because it's personalized. It learns your preferences really, really, really quickly. Now, many people interpret that to mean that it's training a new model within a few seconds. And in fact, TikTok had a paper saying we train models really fast. And that paper is called Monolith. And that's not the secret sauce. So the secret sauce for TikTok is not that it updates the models every five minutes and learns exactly what you did five minutes ago. it learns it much faster than that. It learns it within a few seconds. And the way it does that is not by retraining the model, but updating your features, right? What you've clicked on. So if I click on a video that's sport or I click on a video that's entertainment or slander or whatever, that information needs to be available within a couple of seconds of making the next recommendation for the video I want to look at, right? And that's the infrastructure they built it on. They built it on technology built here in Berlin. Flink, Apache Flink, if you're curious. We're not going to do Flink. There is actually a repo with the Flink version of what we'll talk about here that scales to enormous volumes. We're doing this in Python today. But basically that's the secret sauce, that if you click on things and you make the information that you clicked on available very quickly, we can use that information as input to our predictions. So when you call model.predict, that's what you do when you have a trained model you call model.predict, you can say, well, the last videos you looked at were sport and entertainment. Now the recommender can know, okay, maybe I should show more of that. Okay. So it's been called digital crack. So Andrej Kaparthi from OpenAI called it digital crack. And that kind of stuck as a name because it infects your brain. It's AI that gets into you. And one of the reasons why is this fast feedback loop that we can see at the bottom. so when you want to look at the next video on tick tock and what will happen is it'll go through hundreds of millions of videos and get lots of what we call candidates that it's going to show next and the candidates will be in the order of hundreds and from those we want to personalize those candidates or rank them for you based on what you've done recently and then from there we'll get some recommended videos that go out there and the key points here i think that make TikTok different is you can see our total end-to-end latency the time between you you clicking something and the recommender coming back will be very very Low and we're not going to be that low today because we're doing it in Python and so on and it's not engineered to be fast, but it will that's the basic idea of TikTok and you can see we have this fast feedback loop What we tend to call that is we call them very fresh features that we want the features The input to the model to make predictions to not be stale or old They want to be a couple of seconds old at most if they're you know minutes old or days old It's not going to help it learn quickly Right and I was going to show this one here. Let me see. This is me going through a firewall Let me not show the firewall because the firewall is gonna suck. I have this one up here Let me okay. Is this one loaded? This is actually the video recommender is going to show earlier I'm running the whole thing as I'll reload it here as a what's called a hooking face space Has anyone heard of Google Space Spaces? You should. Oh, lots. OK, they're free. It's really cool. You get like 16 gigabytes of memory and two cores. And it doesn't cost you anything at the moment. So what I did was I deployed a UI written in Python, a framework called Streamlit. It's in the repo. The repo is linked in the code, which is here. This is the repo here. And what we end up at the very end is a web interface to make these predictions for us, okay? So you can pick your user, and then there's a button that will come up here. It's going a little bit slow, internet, but that's basically what we'll end up doing our predictions. Right, this is the framework that I mentioned at TikTok. It's called Monolith, and there's a research paper that they published about it. And the paper tells you a lot about how they do very frequent training of the models, but the key thing here is this feedback loop. And the user clicks on an action, like you like a video, or if you dislike it. I don't know if you can dislike it. But those basically get logged to an event bus called Kafka. This is extremely scalable. So Kafka can scale to petabytes in size. And then they have petabyte-sized Flink clusters that process all your clicks and all your actions. And they create features. And those features are then used by our model. So the features that we'll look at today things like, you know, what type of category of video did you click on, how many likes are on the video, how many people have viewed it, what's the watch time of the video. You can basically come up with many, many more creative features, but you don't need that many to be able to build a reasonable recommender. The recommenders basically need to know in very short time what you just recently clicked on and did you spend a long time looking at it or a short time. So you can create, you can even use it, you know, yourself, you can come up with features, You can say well if I watched the video until the end that's a good sign right if I just watched it for five seconds That's a sign that maybe I didn't like it You know you can impute or you can it's or you can infer That certain signals are that you liked it certain signals that you didn't you don't have to get people to click thumbs up or Thumbs down to actually generate signals about whether and these worked or not So that's what we're gonna look at today, we're not going to look at the the training We're going to look at training, something called the two-target model, but we're not going to look at this fast training in the monolith paper. So to start with, let's just be clear that TikTok is not a very good recommender from a social perspective. This is a quote I read recently. I'm from Ireland, and this is a quote I read in the news last week. And basically what they did was they pretended to be a 13-year-old who registered with TikTok. and because it's able to learn your preferences very quickly and it doesn't have any guardrails, within 20 minutes you're looking at self-harm and suicidal videos. That ain't good. So I'm not encouraging people to do that. That's not the reason why we have the tutorial today. The reason is to learn about the technology behind it and we can do better than this. It's also in the news, TikTok in the States are going to ban it, allegedly. Maybe they did ban it, did they, at the weekend? Anyone know? what did the vote go ahead sorry okay so it hasn't passed through the senate so maybe it'll get through maybe it won't okay now this is another side i i often show people why tiktok is interesting from a business perspective maybe you're thinking about doing a startup right you're saying well you know i'm kind of bored my job i want to do a startup you know where will the vcs be looking for money. I'm from Hopsworks. Hopsworks is a VC-backed company, so that's kind of where we go. I worked a lot with the Flink guys when they're starting out, so you know them really well. But what you do is you basically need to find an area where VCs are putting money. And one area they're looking at money right now is real-time AI or real-time ML. And the reason real-time ML is interesting is because you can generate a lot of business value. TikTok is a very, very profitable, very, very rich company. And you know that because the Americans are trying to shut it down. Otherwise, they wouldn't, right? So they're not trying to shut down Hopsworks yet, which is my company. So real-time systems create a lot of value, but they're not something you will learn typically in an ML course. Many of you have probably taken a data science course. You've done a basic intro to machine learning. You took a static data set, which is at the bottom. You split it up into training and test sets, and you made some predictions on the test set and said you were done, right? But that's not generating value. A machine learning model that generates value takes new data in and makes predictions on it. And the more predictions you make on more new data, the more value you create. Very simple. Now, you can do that with batch data. You can make a prediction once a day, which feels like that's what YouTube does. Like once a day, maybe it updates the For You list. Or you can do it the TikTok way. So as you click and click on things, the system will be learning your preferences. I use the term learning loosely. The system will be updating your recent activities and recent features to give better predictions about what it should show to you. So we're going to build an ML system. I use the term ML system, which isn't cool. It should be AI system, because we moved over from ML to AI. I get confused. It was ML. It was AI 10 years ago. Then it became ML, and now we're back to AI. But one of the challenges in building an AI system is that we're not going to just train a model in your notebook and a static data set, right? That means we actually have to bring in new data, and we have to train models on snapshots of that data. Because the data won't be static. It's going to keep coming in or keep getting new data. So we need to be able to take a static snapshot of that data. It needs to be consistent and correct. We're going to train our model, and then we're going to get our model and new data that's coming in and make predictions on it, OK? Are we going to do all of that in one big notebook? No, right? That's not a good way to build an AI system. So taking raw data, training a model, and making a prediction in one go is a crazy thing, right? Crazy talk. That's crazy talk. And what we're going to do is do what we do in software engineering is we're going to build a modular system. We're going to break it up into independent parts that can be composed together into a system. So what we're going to do is we're going to build something called a feature pipeline, a training pipeline, an inference pipeline, and different examples of those. There's about five or six programs here. But they fall into these general categories. So what a feature pipeline is, sorry, those in the back, you might be able to see it. I hope you have the slides. But I'll say what a feature pipeline is. It's basically a program. In our case, it's a Python program that takes as input the raw data. And the output will be the features and any labels we use to train our models. And then the training pipeline, the input will be the features and the labels, and the output will be the trained model. And then the inference pipeline, the input will be the model and the features, and the output will be the predictions. So this is a very natural breakdown of a full AI system into three natural components. One is we do our data processing or feature engineering. Another is we train our models. And the third part is we make our predictions or inference. So what we're going to do is you're going to write those programs, but we're not going to start with Docker. We're not going to start with Kubernetes. going to build the ML infrastructure for this. We're going to use a serverless platform, a company called Hopsworks that I work for. And you're going to have to register an account. So if you're going to want to follow the tutorial, you can do this while I'm talking. You'll have to go to app.hopsworks.ai, or just go to hopsworks.ai, and register an account. And then we're going to use that as the platform that manages the data throughout this AI lifecycle. So when I have a feature program that creates a data frame, I'm just going to write it to Hopsworks. When I want to train a model, I'm just going to read that data from Hopsworks, train the model, and save the model back to Hopsworks. And when I want to make predictions, I'm going to have a user interface, and some input data will come in. But I'm going to read more features, and I'm going to download the model from Hopsworks, and then I'll make my predictions with that. So that's how we're going to modularize our AI system. And then Hopsworks is the layer that helps us compose it together back into a system. OK. That looks simple. I think everyone looks kind of simple, oh, it's easy. But you can make this, obviously, way more complex. So our feature pipeline could be a batch or a streaming pipeline. I'm just going to show you a feature pipeline for creating some synthetic data to begin with. We're going to run a notebook that will create synthetic data for us. And it's going to write to an abstraction called a feature group, a table containing our features. We're also going to create a vector index. So this is an approximate nearest neighbor index. And we're going to write that again into a feature group in Hopsworks. It's an index for that table of features. And then when we want to train our models, we need to find the features that we want to use. Remember, a feature is just the input to a model. So in our case, it might be for a video. It might be the number of likes in the video. It might be the length of the video. It might be the category of the video. But me as a user, I may have some preferences as well. I may have an age. I may have a gender. And I may have country I live in. and so on. We can use a lot of attributes of me or features of me to help make decisions about content to recommend. So basically, we're going to work with pandas to create these features, and we'll write them to Hopsworks. When we want to train a model, we're going to select features from these different feature groups. And then we'll say, this is the features I want for this model. OK. Create some training data, train the model, save the model back to Hopsworks. And then when we make predictions, we're going to say, hey, give me some features, because I have this user ID, but I need to get features for them, pre-computed features, and then we'll use that to make predictions. Right, I'm going to start coding in a sec, but don't worry. I'm just going to introduce some of the abstractions. This is the really most important one, that when you're writing, we're going to write to these things called feature groups. They're tables of, think of them as Pandas data frames that just keep growing. So think of it as being kind of mutable data for Pandas data frames. I write in a Pandas data frame, and another one, and another one, and the thing keeps growing. And then when I want to get some data to train a model with, I just say, give me back Pandas DataFrames, and I get it from it. Now, you're not restricted to writing Pandas DataFrames. You can have streaming pipelines, or you can even have tables from Snowflake that are mounted in there. But when we want to read from the platform, we're not going to read directly from the feature groups mostly. We'll use something called a feature view. So we're going to say, hey, this model that I want to train, it has some data in the users table, some in the videos table, and I want to join them together. Give me these features. OK, give me some training data, and then later on we'll get some data back for predictions. So they're the two abstractions, the feature group we write to and the feature view we read from. So let's get started on the feature pipeline. I'm going to just pull up, because we're trying to do an interactive demo. I've been running this on my notebook, a Jupyter notebook on my computer. Has anybody managed to do a pip install on this and get it working, no? No one, god. All right. OK, so let's do Colab instead. I'm not a Colab kind of guy, but because it kind of sucks, here we go. Let me see. So this notebook, if you're curious, we're going to go into, this is the repo that we have here. It's called pydata-tiktok. Jim Dowling is the org. That's my name. And we're going to run this one called feature backfill. And what we're going to do is we're going to take videos, users, and interactions between them. And I'll explain that in a little bit more detail. But you can think about it. I've got users in my video platform. I've got the videos. And an interaction is when a user clicks on a video, or likes a video, or does something with a video. So they're the three tables that we'll have. I'll explain those in a little bit. But I just want to get us started in coding. So in Jupyter, what I would normally do is just run the cells, and we're good to go. But I'm going to run it from Colab instead. So this is me earlier. I just started a few seconds ago. And I wanted to download that. Let's have a look at this. So one of the reasons why I didn't want to use Colab... All right, Mimesis I have to install here. One of the reasons I want to use Colab is you can see I have to do something really ugly up here. So when we talk about modularization and Python, what happens if I want to pip install a library in a Colab notebook? Let's open my notebook here. We'll have a look at it. So you can see here. I've got this Module called features and I'm importing this users module for sorry this package called features and this module called users So I remember I said I've got tables with users videos and interactions. Well, I put my code in those modules I've got this directory here called features and we won't call users and if we look in here You've got a function basically to generate some users. So it's in synthetic data. It's gonna give us back a In this case, you can see it's generating a dict, and it's going to return a list of those users. So it's going to return a list with this dict inside it. It's in a separate module, but Colab doesn't like that, right? So those of you who've used Colab will know that won't work. There's no way to get these. So what I did, and I hacked out here while I was waiting, is I just downloaded them here. You can see that. So I actually forgot to download one of them. And then I wanted to pip install Mimesis, so let's have a look at Mimesis here. Please shout out if you have any questions at this point. There's one more library I have to install here. Yeah? Okay, so for all the online users you can ask questions on Slido. Whatever that is, I guess it's an online tool. Okay, so you can ask questions on Slido and then then we'll bring them up here So I'm gonna I was missing this library mimesis. This will ask you to restart your shell, which is fine And I did download that so this looks like it's installed Okay, so what I want to do here was you can see import these modules So this is something that really sucks in Colab what I don't like Colab But you want to organize your code into modules and the reason why you have a module in Python is because you want to reuse Code right I have six different notebooks Am I going to take the same function and redefine it six different times or four different times? No, it's insane That's non dry code. I want dry code dry means do not repeat yourself and So basically I put it in a module. So what we're doing here is and generating some random users Generating some random videos Generate some some interactions between those users and videos. So we've created our users. We've created our videos This one will take a bit more time. It's 250,000 This one I won't spend too much time on but it's a very nice library called great expectations in pandas or say in Python And what it does is you can define data validation rules So you probably heard the term garbage in garbage out and data science. You don't want any garbage in So what you can do with great expectations is you can define data validation rules. So for example, here we've got a column called age. And we're saying the minimum value for age is 12 and the maximum value is 100. This is very ageist because there are people over 100 who use TikTok, I'm sure. And there's definitely people under 12 who use it. Excuse me. But I'm defining on the Pandas data frames that I'm going to write in here. And basically I can define any types of data validation rules, and we'll see that they get Evaluated then when I want to write this data to our data layer what we call a feature store hops works And we also had another one expect last time. I looked we have three genders. You know male female non-binary It's called other here, and there may be more. I don't know and You can basically define here We've said there's only three if there's going to be four another one appears then it'll it'll give us a warning or an error Okay, and this is again another data validation rule we don't need to go through them in too much detail So Another data validation rule so it's looking at watch time of videos and saying it should be a minimum zero, right? So you shouldn't have any negative watch times. This is where you'll probably get stuck first, which is it's trying to log in in. So what it's going to do is it's going to ask you for your API key in Hopsworks. Am I going to show you my one? Probably I will have to. We can delete it later. I don't see any way around this at the moment, given the way it is. Luckily, it's not the end of the world. And now it's going to ask me for my project, because I have a bunch of projects. Oh, no. It logged me in. OK, if you have more than one project in Hopsworks, it'll ask you for that. But you won't have more than one. I have more than one. But anyway, so what I can do is I can click on this link here. Hopefully, it'll open up a new tab. And OK, I need to log in. Oh, it's logging in the wrong user. Sorry, I've got too many accounts on Hopsworks. I'm logged in here. So this is me logged in here. And when you come into Hopsworks, what you'll see is that there's a bunch of things on the left-hand side. So this is where we're going to manage our data. Remember, we said we're going to store data in feature groups. And we're going to have the feature views to have the selections of features for training. And then when we store a model, it'll be down here in the model registry, and then we'll have deployment. So I don't have anything here at the moment. So we're just going to populate these feature groups to get started. And basically, what a feature group is, think of it as Panda's data frame. It could be a Spark data frame. It could be even for Flink. It can be a data set in Flink. But basically, we're going to write it to the platform. So I have a name for it. I give a description, a version. You can define a primary key on it, which is a good idea, if you've got unique roles. So in this case, when I'm inserting user's data, each user has a unique user ID. So I'd say that's the primary key. I have this thing called online enabled. And I'll go through that in a second. But one thing that makes a feature store different to, let's say, Dropbox or Google Drive, is I'm not just storing the Pandas data frame. I'm actually going to put it in a table in a low latency database so I can retrieve rows of it at very low latency. And then finally, we're going to pass into this expectation suite because we want to do the data validation in here. OK? So you can see here it's now uploading my data frame. And that'll get started here. I can do the same for my videos and interactions. Start that as well. So it'll take a few seconds. Given the internet here, let's see how long it takes. It didn't seem to be too bad. That's 25,000 rows. This one is 25,000 rows. And then this one is 250,000 rows. So you can see it's uploading. Yeah, it was quick enough, actually. But what it's actually doing is it's uploading it to the platform. And then an ingestion job will get started to ingest that into the two different databases, the low latency database and then the ultimately, it's storing it in what's called a table format. So you've got Apache, Hudi, and Delta Lake, and Hopsworks. There's going to be a table in one of those as well. So this one's a bit bigger. It's 250,000 rows. Luckily, this is not internet here in Germany, actually. I forgot. This is running on Colab. That's why it's faster. And it's uploading the data relatively quickly there. OK. So that's the start, right? So we've created some base data. We're going to use this data then to create what we call a ranking model. We're going to look at retrieval. That's just us getting started. So you can try and get started. My colleague here, Javier, will be moving around. If anyone wants to raise up their hand and is stuck, he can try and unstick you for a bit. We have one up here already. Any questions? It's a big crowd, but don't be shy. Do it get out. Can I do it? We have one, but I can't see. Sorry. Is there a place to see the Google Colab link? ALFRED FULLER- To what? To see the Google Colab link. ALFRED FULLER- Is there a place to see the Google Colab link? Let me show you how to do the Google Colab link. I hope everybody has this link at the beginning, right? OK, if you haven't got this link at the beginning, you want to get that link and open it now. I'll leave it here for 10 more seconds. Shorturl.at. The first one, you have to go to the slide 27. It says instructions, and it's another link that brings you to the wall. OK, is there anybody who hasn't gotten this link yet? I'm going to take it off the screen. Yeah, I've got a question concerning the slides with the repo. Okay, so let me show you how to do Colab because we need to get into Colab here. So when you go to slide, this is TikTok tutorial instructions. You can see there's the link to the Google Doc here. If we open up the Google Doc here, you'll be able to see, when I zoom in, you'll be able to see this TikTok tutorial here. Now, I jumped over some steps, right? I'll admit that. I jumped over quite a few steps. Okay, but let's go through those steps. Those of you who don't know Colab, you need to go to colab.research.google.com. And colab.google.research.com, if you don't have a Google account, I'm sorry I can't help you. You'll have to live with the Wi-Fi here. But you can see when you go to google.colab.com, you've got some options on the left here. So one of them here is GitHub. If I click on GitHub, and I paste in the URL that we had earlier, and press return, return, what we'll see here is it will show me there's a bunch of files in there, right? Authorised with GitHub, yeah, whatever, OK. This worked earlier, I promise. Oh, look, something has appeared down there, so maybe we can just press cancel. No! OK. Anyone else get this? Obviously too many of us are doing it at the same time. Sorry? No, this is right. No, it's the wrong one, you're right, it's the wrong one. thank you thank you thank you thank you shit you got me okay so um thank you we have people paying attention not me so that one i showed you actually is uh is is the more complex one pi data uh tiktok is the correct thank you very much you got me all right so let's put this in Here. Small typo. Is there a typo there again? Tiki. Tiki. Tiki. Tiki-talky. Tiki-talky. Tiki-talk. That sounds good. I prefer tiki-talk. It sounds a bit kind of, it sounds more fun, less than self-harm videos, doesn't it? Tiki-talky. Check that URL. If that domain's not gone, I'll buy it before the end. OK. Sorry, so that's a mistake. Thank you very much for picking it up. So this is the repo here, pydata-tiktok. And when we go back to Colab, not this URL here. We're going to go to pydata-tiktok. The other URL, if you're curious, has all the Flink code and all the kind of fancy stuff. I wouldn't touch it. I think you're going to press the Huh? Yes, you're right. I need to put it in here, don't we? Yes, there we go. OK, so then when you go to this one here, this repo here, we'll get the link here, feature backfill, right? Now, the first thing you need to do here is you need to uncomment all of this, right? And the way I do it, here's some tips, pro tips here. Control A, go on, Control A, yes, and then Control forward slash, and then uncomment it, right? I'm sure 99% of you know those shortcuts, but anyway, I I just wanted to sound like I knew what I was doing. And then you press Shift-Return. And then it'll install. We need two libraries for this notebook. One's Hopsworks and the other one's Memsys. God knows why you need Memsys. I really don't know. I would love to say I wrote all the code, but I didn't. Max wrote most of the code on this. And that's basically it. So anyone who's stuck, Javier can help you get around that. Any questions on any of that? Yeah, so the other one that you want to do is you want to create an account on Hopsworks, right? So we're going to use Hopsworks as the storage layer. It's going to store our data, and it's going to store... So you need to create an account here. So if you have a Google account, you just press Google, continue with Google, press, press, and it'll create the account for you. Yeah? I'm not going to... If I press this, it'll just log me in, right? because I'm not registering an account. It literally asks you for two things when you're registering. I think your name and what you do or something like that. And then I need to create a new project for HopSource? No, HopSource will create a project for you. So it takes about a minute to create the project, I think. It asks me the name. Oh, yeah, you can put in the name then. Sorry, I haven't registered. It used to create a project for you automatically. The namespace for projects is a global namespace. So if you call it PyData, I bet you someone has taken it. So be original and call it PyData24. And I bet you someone's taken that one as well. Tiki Tok, that's a nice name. Tiki Tok. Don't anyone steal that on Namecheap. I'm going to register it after this. All right. Any other questions? All right, so let's have a look what happened, right? I think I... Did I run all these? I didn't run all of them. Let's run all of them so that we kind of get to where we want to go. I was here. Okay, yeah, I'd gotten this far. So what I'd done so far here was I'd uploaded those three data frames, Panda's data frames, and when I press refresh over here in my feature groups, it didn't refresh automatically, but I get those in here. You can see I've got the users the videos and the interactions So you can go in and have a look at them. Maybe you can press data preview No data there. Yes Okay And that data should be ingest. So it just appeared there one second Let's go back here If I click on offline, well, we can see some of the data's in here So I click on offline here and then the data went in here Okay, so you can see some of the data here. It also computed statistics on the data that it wrote in Excuse me You can see the age of the users their min their max the mean some the country. There's two hundred thirty nine That's more countries that are in the world. That's unusual and there's three genders and so on And then the same is the case for videos you can see there's some videos We can click also down here on activity to see what happened and you can see it wrote 25,000 new rows rows, and then if you go back here to interactions, and we look at activity again, there should be 250,000 rows of rows, okay, so that looks okay, right, so what I'm going to do next, so now those three tables are in there, I'm going to create something called a ranking feature group, and I haven't explained what ranking is, but I'll just go through the code just to get this notebook done, but ranking is what it's going to do, it's going to personalize this data to me. I want my recommender to know and learn about me. So if we look at the columns that we have in our tables, before we do any more, let's look at the columns. We have a user has a user ID, a gender, an age, a country, not a huge amount in there, of course you could have a lot more data in a real system. The videos have a category, a number of views on them, a number of likes on them, a video length, and an update length. This might be updated once a day, you might have a big batch job to update the videos, or maybe every hour, depending on how frequent you do it. Users might get updated once an hour as well, or less. And then we have these interactions, and these are updated the whole time, because users are clicking on things and making updates. So we can see the interaction types. So we're going to have a number of interaction types, like you watch a video or you like it, and then how long you watch it for. So we have this user watch this video for this amount of time, And then each interaction will have a unique ID as well. So it's going to be a primary key identifying this interaction as well. So given that data, and I want to train something called a ranking model, I'm not going to get into the details of what ranking models are. But what they basically are trying to do is personalize recommendations for me. I'm going to get back anywhere between 200 and 500 videos that it could recommend to me. I want to personalize them to me. So in order to personalize them to me, We need to create a data set that I can train a ranking model on. And here what we can see is in this data set, I have a user ID, a video ID. That means this user watched this video, which was sports. And that video had 39,000 views and 37,000 likes. And it was 25 minutes long. And it was uploaded on this date. And then the gender of the person watching was this, and the country, and the label, and so on. So then we have this thing called label at the end. See this label? Any guess what that label is? We'll see it here in the comment here. Here it is here. It's defined here. Anyone want to shout out at me what they think it is? The weight of the type and direction? Kind of. Anyone want to shout out what they think it is? Positive or negative. It's positive or negative. So you can see here it says that if the user viewed, liked, shared, or commented on this video, that's an interaction. When I'm building a ranking model, I can't just have ranking data where users interacted with them. I want negative ones as well, where the user, see we've got a label zero here, right? I'm saying this user didn't watch or like or comment or click on this video. We need to have a mix, right? So here you can see we've got some data where we have positive interactions and some where you have negative interactions. What you can see from these features here, these are all the features, we can find out that, oh, if I'm a user and my age is this and I come from this country, other people have tended to like this and other people have tended not to like this. You can see that that signal is in the data, right? We can see that's there, yes? Now, you can imagine personalizing it even more and then storing features about exactly what I did, right? What did I do the last hour? What did I do the last five minutes, the last 20 minutes? This is a more simplified version of it. Excuse me. OK. So I have another feature group for that. Okay, any more questions on that? So I've run all that notebook. That's my feature pipeline. No, have I run it all did we we close that one there Okay Here's my notebook and I finished it was I I'm finished. All right. Give me what is saying ranking DF not defined I missed the cell. Sorry, let's go back. I didn't run this cell today All right, so going super fast here. I don't know what's going on. It's thinking about it. We're killing the internet. Let's go back and do a bit of theory while that's spinning around there. I don't have the patience to watch things spin. I don't have much patience at all right keep an eye on the time we're good for time don't worry okay so we said that we're going to build the AI system out of three pipelines feature pipeline training pipeline inference pipeline and the feature pipeline so what we did there and I'll explain kind of what what that code was doing is what we're doing was simulating and creating the training data for something called a two-tower model. I'll get into what the two-tower model is in a minute. But to build a two-tower model, we need to store the interactions between a user clicks on something, or user views something, or comments on something. You need to log that data. So if you want to build a personalized recommendation system for your retail site today, you're going to have to start storing that data. It's the first thing to do. And you can have a label where you say, oh, the user clicked on it, one or zero. You can see I've got scores there, zero, one, five. You can have a score of five if you put something in your shopping cart, one if you clicked on it. If you share it, give it a two. You can basically come up with different scoring mechanisms to basically help your recommender system over time. But we need to log that data. Without that data, we'll see that it won't work. and I'll explain in a second why that is. But these were the tables that we created here, right? We have the interactions here on the left and these were our... We had the interaction type, the watch time and we had references to the user IDs and video IDs. This is something called entity relationship diagram in case you're curious in database world. And then we have a users table, we call them feature group and we have the gender, age, country and then we have the videos where we have the category, views, likes, and upload, and so on. What we haven't looked at yet now is something called video embeddings. So what we're going to do is we're going to create an embedding, a vector embedding for each video. So remember what a vector embedding is. It's we take all the information we have about a video and we compress it into an array of floating point numbers, a fixed length array of floating point numbers. So it keeps all of the semantic information about the video. I'm able to say, is this video, which videos are similar to this one? And I give it the array and it gives me back arrays for videos that are similar and that's the basics of Vector embeddings and similarity search, but we're going to do that in a second And we can see that we you know, we have a batch pipeline typically which would Update these now in the streamlet example later on that we look at the UI when I actually click on the videos and like them It'll push it into Kafka and this gets pushed in here. That's how our interactions are getting in there in a a TikTok-style system. But the other ones are updated by a batch pipeline. Batch means running on a schedule, like once an hour, once a day, once a week. Streaming means it's running all the time, 24-7. Days just keep coming in. So I said I'd tell you what a two-tar embedding model is, and it's actually something that's quite difficult to understand. It took me quite a while to get my head around it. And I was a professor, so that just tells you nothing. But it doesn't mean anything. But anyway, everyone knows, I think everyone, probably everyone in this room has seen the basis of what we call similarity search. You go to Zalando, and you look at some shoes, and it shows you shoes that look like them, right? And many of you will know that you can take the image of the shoe, you can convert it into this array of floating point numbers, fixed length, called a vector embedding, and then you can go to something called a vector index, or vector database and say, show me similar pictures to this. And you'll get back shoes that are similar, similar color, similar style, or whatever. You can do that with lots of things, not just with images. You can do it with text. You can do it with tables of data about users and so on. But what's really hard to do is what we're trying to do back here, which is saying a user clicked on a video, and the user previously had looked at these other videos, and the user likes these other types of videos, and the user's this old, and the user has this age. We have information about the user, and then we have the information about the videos. And they're two different worlds, right? They're two different tables, yes? The user table's here. It's completely different data from the video data. But I need the two to interact, because what personalized recommendation means is that I can take the user data and recommend a video, right? But if I generate embeddings on the video data and find similar videos, the user is over here. It's not touching the users. So how the hell do I get the user data and the video data together so that I can say, this is the user, recommend a video? That's hard, right? There are two different modalities. One is videos. One is user queries and user history. So the answer to that is something called the two tower model. What we want to do is, if we take a video and we have the row that describes our video and we convert it into a vector embedding and put it into our vector index, we'd like to take the user queries and their interactions when they clicked on a video, when they liked the video, when they commented on the video, and put it into the same embedding space, the same vector database, the same index, the same length of the vector embedding. This is the basis of the two-term model. Let's look at an example. This is going to show you an example of it working. And you'll hear the term, not video, but it's a book. We've got a book called Romeo and Juliet. It's going into the embedding space. Great Expectations, King Lear, these are all famous books in English, Pride and Prejudice. Now we've got an embedding space with books in it. User queries come in. Oh, look, Shakespeare Trilogy is close to Romeo and Juliet King there. How the hell did that happen? Because one was a user query and the other was a book. How the hell did they get close to those? Well, we had interactions. When a user queried for Shakespeare tragedies, they tended to pick those books. Now we've got an interaction. We've got some training data which says that this was user query, this was the book they clicked on or the book they liked or the book they commented on. Now I've got some training data that I can actually put both of these modalities into one bigger modality. Is that OK from a hand-wavy perspective? Yes? Hand-wavy. Feels kind of get where you're getting, but not really. Let's look at it from a little bit more technical perspective. On the left, we've collected all of this training data, the users clicking on videos. What we're going to do is, I guess many people have looked at LLMs and vector embeddings, right? You've tried out sentence transformers or something like that, and you've kind of ran some text through it, and it generated a vector embedding. Oh, that was great. What you did was you used a model. You used an encoder model, right? And that encoder model was trained by someone, right? But what we're going to do here is we're going to actually train our own encoder models. We're not going to take an off-the-shelf one from Hugging Face. We're going to actually train a user query encoder, and we're going to train a video encoder. And the way it's going to work is that we're going to train them using this interaction data. So when the user clicked on the video, we'll get this highest interaction. And when they didn't click on the video, we'll get a lowest interaction of 0. And what we basically say is that when a user entered this query or clicked on this as their most recent video, or this was their history, And then we present a video, and we get a 0, which means that there was no interaction there. That means that we had a loss, as in the model that was not close together. Close together, in this case, means often dot product or cosine similarity. We look at the angles between these two different vectors, and then we basically say, if the loss is high, because that's what we've got here, we have no interaction, we're going to update the weights on our encoder models using gradient descent and backprop. And if, however, the others, they were close together, if the user's history and so on was matched to a like for this video, then we're going to get a 1 out, which means that we have a lower loss, which means that we're going to update the weights to say that was a good thing. They're closer together. So what will happen is the embeddings over time will tend to co-locate in high dimensional embedding space. When users issue queries or whatever, in this case, clicked on videos related to sports, and then we have a video that's in the category of sports, they'll tend to be closer together in this joint embedding space. We have a question. Yep. Does each embedding represent each user and his interaction with video? So I'll get to that in a second. So these are called vector embedding models. What they do is you put in some input features and you get out an array of floating point numbers, yes? That's what these two things are, the models. But the actual data, this is what we do in the next slide, so it was a good one for the next slide. So what we do is we take all the videos in TikTok, and with this vector embedding model that we've trained, we compute vector embedding further and we write them into the vector index, yes? Okay, so let's do that. Let's do it now. I'm going to pull down my glasses, wherever the hell they were, because I'm an old guy. It'll happen to all of you too at some point. Let me run up my laptop because I'm honestly tired of Colab. Let me go here. Right. So what we're doing here then is we're going to use a package. You can see it's complaining here. I've pip installed TensorFlow already. It's basically saying I've only got some AVX2 instruction set. I don't have a GPU in this machine. It's just a warning. Don't mind us. But I'm using two libraries here. One is TensorFlow. And many of you know TensorFlow and say, well, what the hell are you doing? Everyone's using PyTorch. TensorFlow sucks, which is true. But TensorFlow has still this TensorFlow recommenders library, which is fantastic for doing what we want to do here. We don't have to write much code to do a two-tier embedding model in TensorFlow recommenders, so we're using it because it's the easiest way to get it done. You can go ahead and write everything from scratch yourself in PyTorch, but I'm not going to do it. So what I'm going to do here is I'm going to basically, firstly, we connect to Hopsworks. We get the references to this data we had before, the users, videos, interactions that we downloaded. And then what we're going to do is you can see we're going to create this thing called a retrieval data set. Now, I haven't introduced the term retrieval, but often this two-tar embedding model, in recommender systems, we call this a retrieval phase. What I'm doing is I'm putting all the videos in my vector index. I'm going to take the user query, compute an embedding, find similar videos, and then get back those down as my first phase. I'm going to get a couple of hundred candidate videos back. So we call this the retrieval phase in recommender systems. Now, to build this, I need the retrieval model. So we're going to get some features. So we have the user gender, age, country. You're probably breaking the law here using gender. You shouldn't train models on gender. I'm doing it anyway. And now, so oh, I'm going to be stuck in the internet here, so this will be interesting. So what I do is I create a feature view. Now, in Hopsworks, what a feature view is is basically I want to take features from these three different tables. So I've got the users, the videos, and then the interactions between them. And I'm going to select the interaction ID from the interactions feature group. Going to join in these features here, the gender, age, and country, joining them on the user ID. And then I'm going to bring in all the videos features, the category, the views, likes, and video length, joining on the video ID here. So with those selection of features, I create this thing called a feature view. And if we open up Hopstrix, we'll be able to see our feature view here. So the feature view will appear here. This is my retrieval feature view. It's just metadata, right? It's basically saying, oh, I've got some feature from the interactions feature group, some from the users, some from the video. It's not supervised learning, so I don't have a label or a target column. You could define a target column as well. And what I did here then was I can create training data from the feature view. So I can get it back as a Pandas data frame. Now what I did here was I called train validation test split. So I can actually break up my data into training, validation, and test sets. And I can specify the size of those training and test sets. And then I can go and train a model. So this is where we're going to use the two-tar embedding model. So this code will not be familiar to many of you if you haven't done this before, so don't feel too kind of intimidated. But remember, we have a thing called the query embedding, and then we have the video embedding, or candidate embedding. So they're the two inputs that we're going to have. You can see this one, the query one. If we go back to our slides, we can see it there. The query coming in here, that's those features coming in here, and then we have the video features coming in here. So they're the features that we're going to come in. TensorFlow doesn't like pandas data frames, so we have to convert the pandas data frame into a dataset, a TensorFlow dataset. And that's what we're doing here. We're doing it batch-wise, so I should run that code. I don't have a GPU, so it complains a bit. And yeah, so basically then I'm just making sure there's no just counting the number of unique rows and items. Now, this is where we get into the two-tar embedding model. This is a very small embedding size, so 16, a length of 16 in the number of dimensions of my embedding model. It's not very big, but it's OK for the moment. And I'm going to start with the query tower. Remember, we have two parts. We have the query tower, and then we have the video tower, or item tower. And I'm just going to run both of those while I'm here. Sorry. I'll start the training as well, just because it will go off. OK, so we started with the user tower. So let me go through the user tower here. We call that the query tower as well. It's a class. It's inheriting from Keras model. So we have an init method. And in the init method, we're going to set up all of the features inside this user query. So we have, firstly, the length of the vector embedding, which is 16, 16 floating point numbers. We can see this is our embedding model itself, the user embedding. And what we're saying when we create an embedding model, We have to tell it what are the users, the list of users. And we calculated that earlier in the previous cell. We've got our list of users here and our list of videos. So we pass that in as the vocabulary to our embedding model. And then we tell about the embedding model. We say, hey, we're going to have the embedding will have this many tokens in us. And yeah, so that's basically with this one embedding layer. So it's not a very complex embedding model. That's OK. And the output will be 16 floating point numbers. Then we have other features. We have the age of the person. We're going to normalize that. We have the gender. We're going to, I think we're one hot encoding it, but we're basically tokenizing it. And they were doing the same for the other categorical feature countries. And then we basically define a feedforward network. And the feedforward network here takes in our embedding, and we have a relu, and that's basically it. That's the definition of our embedding model, the layers. When we make a call and we want to train on it or make predictions, we'll go in here. We'll get our gender. We'll be one hot encoded. Our country will be one hot encoded. We'll take our user input data, and we'll concatenate that together. And then we'll run it through our feedforward network and get the output and return. That's basically what the query embedding model will do when it runs. So we defined that one there, and then we create an instance of it. And we're also normalizing the age as a parameter. We're doing that there. And then we're basically figuring out what is the data set that we're going to train this model on. So that's this query DS, and then we have our query model is passed against that query data set that we run that on. We do the same for the videos. This is the item tower. We've got, again, the embedding length, the same embedding model that we defined earlier for users we're using for videos, and the same tokenizers. We're normalizing the numerical features. We've a feed-forward model here again with the Relo activation unit. And the call is the same as we saw before. So both of these are very similar, the item tower and the user tower. And we basically initialize them here. So both of them are now initialized, but we haven't done much on them. We're going to do our training with this thing called the two-tower model. So if we think about it at a high level, we go back here. We basically defined that. We've defined this, but we need to kind of put them together so that the training data can go back here and then update the weights in both of these in parallel. So this is our two-term model. So what it has inside it is the query model as one, and it has the item model or the videos. And then we have our data set in here, you can see here. So when we do training, you can see it basically computes the user embeddings, the item embeddings, we compute the loss, and then that's basically it. And I think I've run this already. Did I run this already? No, maybe not. Okay, I should have run it already. Okay. So basically we're gonna, we instantiate, so I didn't write this two-tar embed model, So we're using the TensorFlow, so if we look earlier, we're inheriting from TensorFlow's Keras model interface, and then we can use some of the libraries that they have as well to make this go better. So actually, training it doesn't take too long. You can see it went pretty quick. But once we trained our models, we now upload them, and we're going to store them in Hopsworks. So let me tell you what was happening there. So, in this case, what we did was we had our query model. We have the two models come out, the query model and the video model. So, what we need to do in HopSearch, and this code is somewhat simpler in the latest version, but we need to define the scheme or input to the model. And here we can see that the output is the actual vector embedding, but the input is the same interface as our data frame, our query data frame. So, the query data frame, remember, had the user's age and then their country and so on. We need to do the same for the video one, and again, we'll take the input schema for the video from the data frame. And that's done down here. So that's done down here in our candidate or in our videos model. We, again, take the data frame from that to create the input schema, and the output schema is the same. So then what we need to do is register the model once we've defined its schema. We've also added a sample of records so we can store them in Hopsworks, and you can try them out then. But here, we're creating a TensorFlow model. So this is basically the syntax for creating a TensorFlow model, and then we save it here. We say save that query model, and that query model was saved to disk, I guess, up here. Yeah, so earlier on, we saved the query model to disk on this line and this line. You can see those two lines. So after we ran these lines, both of our models have been saved to local disk to a directory called query underscore model and then item model. And then when we call in Hopsworks, You can see it's doing that uploading thing there. This is basically uploading the contents of that directory to Hopsworks. Now, we need to do the same for the video model. I'm not sure if I run this already. It uploads it, and then we can go to Hopsworks to see our models should be up there, which is down here. OK. We've got our models are up there now. So that's our training pipeline. Any questions on that? That's for the retrieval model. We still, once we've created that, we need to now populate the vector index. We need to take all of our videos and use what we call the item encoder, or video encoder. We're going to take all the videos, read them up, run them through this video embedding model, and then insert them all into a feature group in Hopsworks. So that's the next notebook, which is here, Vector Embedding's Creation. You can raise your hand as well. Javier can go around and look. Yeah, go for it. I'm not sure I understood this correctly, but does the video embedding contain information about the content of the video? So we can look at the columns. You can see what the embedding does. So the question is, what does the video embedding contain? Well, this is our video here. You can see every row we have in our video here. If we have a look at a row here Let's have a look at some rows They're slow, let's have a look at the overview of the rows this one's a bit slow And so we've got we've got a category of the video We have the number of views of the video number of likes of the video the length of the video, the data was uploaded on, okay? So how do you handle the case of you have a new video without any new... Yeah, so this is what we call a batch feature pipeline, right? We would need to run it on a schedule. So you could run it every five minutes. If you're TikTok, run it every minute or something, you know? So when you upload your video to TikTok, it's put into a big directory, and some big program will take that video and its description, and it will write and update this feature group, yes? And then a video embedding... What I'm calling here a vector index pipeline will take all the new videos that have been uploaded in the last five minutes or whatever and then compute embeddings for them and then write them to the vector database, in our case, the vector index. There's no user interactions? There's no user interactions. No, this is unrelated to user interactions, but it was trained, remember, using user interactions, We needed to have the user interactions to train this video embedding model and to train the user embedding model Excuse me So what I did here was it created a candidate embeddings feature group and if we go back to here We reload here we now have a candidate embeddings feature group and this feature group contains our embeddings, right? so we just have the video ID and then we have the the array of 32-bit floating-point numbers, so 16 floating-bit numbers. So now, with this feature group, you can pass in an embedding. And you can say, give me similar videos. And it will give you back the IDs of the videos that are similar. So that's how we do similarity search. Now, we don't have a separate vector database for this. This is done in HopSearch. You just treat it as a feature group. That's why I use the term vector index now instead of vector database. Those of you who know the likes of Postgres, Postgres has its own vector index, Neo4j, MongoDB. So it's an open question as to whether vector databases will be an independent category in a few years' time, because so many databases are adding support for vector indexes today. OK. We have a question. A question online. Yes. How do you evaluate the quality of embedding models? How do you evaluate the quality of embedding models? I didn't show you anything there, did I? So normally, an embedding model is an encoder, right? So what we're doing here is where we have the training data is the interaction data, and we're basically putting in the data, and the model predicts whether it's a 0 or a 1, an interaction or not an interaction. So what you do is, in this case, we're looking at the distance metric. We're looking at the distance between the predictions and then the output. So you're basically trying to minimize that loss over time. That's all I'm doing in this particular case is I'm trying to minimize that loss in training now whether that's a really great You know So this is not supervised machine learning. So we're not going to use any of our classical like supervised machine learning Evaluation metrics for it. We're just looking at the distance between the predicted Interactions and whether it was correct or not All right, so that's that's number three finish so let's get back We're starting to lose a bit of energy here. I can feel it. Come on, we've only got 20 minutes left. Let's try and... We're at notebook... Where are we at? Number three. Number three of five. Okay, not too bad. So that's what we did here, right? You might think, like, Jesus, God, God. He said feature training and inference. We're only training. We did training already. But we're going to do the second training. We're going to train this ranking model, which is personalized to you. We actually created this feature group for this earlier. One thing I wanted to just say to you, those of you who kind of work in the space, if you have time series data, so time series data, this is all time series data in the real world, right? Users have a column saying when they were last updated. They have video when it was uploaded. You know, they have interaction, the time of the interaction. Creating that training data is quite hard, right? Because, you know, your age changes, right? So if I take the interaction from two years ago when I had a younger age, than my not-so-young age now. And, you know, if I had the wrong age in my training data because I just pulled the latest value of your age from the user's table, I didn't know what your age was two years ago, I didn't care or whatever, just pull whatever's in the user's table and join it with the interactions, then I'd have stale values, right? So there's some stale feature values in there. It wouldn't make my model work that well. There's a worst case, which is where you have what we call future data leakage. So you've got some time series data And you're trying to take the interactions as your starting point there. There's gonna. We're gonna create train data from there We're gonna pull in some features and the users pull in some features in the videos, you know But if you predicted things that you knew were going to happen in the future you get what's called future data leakage It sucks right because then your model your training and testing goes great, but then in production it doesn't work at all So here's a little tip how to create training data from time series data using as of left join Anyone know what that is, right? It's not so easy, right? And so that's where feature stores will help a lot is that this is something called slowly changing dimensions in the data warehousing world It's a very complex topic. But in our case, it's pretty straightforward We don't need to care about something change the dimensions. We don't need to care about less as of left joins What we're just going to do is let's run through the notebook here We're basically going to create some training data from our feature view and the feature view in this case I'm going to create a couple of other feature views just because I'm going to read from videos and users but in this case I'm going to use this ranking feature group and the ranking feature group pulls in features from the users and the videos tables but it does it runs this as of left join in the background to make sure they all get pulled together correctly okay so I'm just going to train this ranking model while we're talking here now this ranking model needs to be super fast, right? The reason why it needs to be super fast is because we need a very low end-to-end latency. And gradient boost decision tree models are really fast. Like, they'll make predictions in a millisecond or two or three. And CatBoost is one variant on it. It's categorical gradient boosted decision tree with typically mostly categorical data as input. But basically, I trained a CatBoost model there. And we can test to see what its performance is. You can look at the loss here, so we can see if the loss goes. I think it started about 50, 60, and then it drops to about 42. So this is not the world's best model. That's not the point. But we've trained a model, a cap-boost model. We basically just do some simple eva- this is some, because remember, we're not talking about deep learning models anymore. We're talking about decision trees. So we can do things like feature importance. We can say, hey, I have these features, age, likes, views, country, category, video length. Which are the most important? Well, we can see here that age is the largest indicator in this particular training data set. And then we can basically add that model again to Hopsworks, to our feature, to our Hopsworks platform. So we're going to upload the model. And let's have a look. It should be here somewhere. OK, so now we've got our model up here as well. It's called the ranking model. You can see it here. And you can pull them open. And you can see we can also store, when we upload a model, we can have precision metrics. You can also store images, PNG files. So I know a lot of people use Weights and Biases, and Neptune, and a bunch of other platforms, and MLflow. They're all really good. But I think here, when you save a model, if you store the feature importance images, and maybe even training loss in your model registry, it kind of is a poor man's experiment tracker. It works enough for me, because I don't store experiments for models I don't save. Put it that way. All right. So now we have a ranking model. We're making progress. So that's basically what we did. Like, at a high level, what we talk about in machine learning are what we call assets. So I had these views of all the features. I created some training data sets, the retrieval and ranking training data sets. With them, I trained the video embedding model, the user embedding model, and the ranking model. And I put them all in the model registry. They're the assets I have now. I've got the feature groups. I've got my feature views. I've got my models. When I put those steps together, what we've done already, what we basically do, you can think of this as being the offline things that happen. This is not the online system, the offline things. Offline things are things that are happening in the background, so batch pipelines, streaming pipelines. Our data's arriving in here. We're doing some feature engineering. We're storing it in the feature store. From there, we have the model training runs, and it can run at its own cadence, run once a day, once a week, once a month. We have the insert into our vector index. Well, we just said how often should it happen. If it's TikTok, it should happen very quickly, within a few minutes. Maybe for you, you do it every hour or every day. But that's a program that needs to run on a schedule, take the new videos that are added, compute the embeddings for them, and write them to the vector index. And then the ranking model, how frequently should we retrain that? Well, you can do monitoring, Hopsworks has feature monitoring built in, you can look for, if your model predictions are going stale over time, you can basically see drift in features and so on, then you can decide to automate retraining. But often it's just the case of just running it like once a week or running it when you see the predictions getting worse. So there's one last part to do, and then we're kind of done in training. And it's what we call the deployment pipeline. And that's going from a model as just, in this case, an asset. So it's just a file. So my catboost model is a .pickle file. We can actually see them if we have a look at them. Let's go back and look. This is our ranking file here. You can see we're up here. It says browse model files. Well, we have the pickled Python file. It's 426 kilobytes. It's not so big. We have a JSON for the schema. You can have a quick look at that. And the other models that we had, let's have a look at them here, the query model and the ranking model. And you can have many different versions of them, of course. You can see here that this one is slightly different. So in TensorFlow, models are .pb or protocol buffer files. And then we have a bunch of weights in here in this folder, the variables folder. So they look slightly different. OK, so that's all our models are. But when we want to use our models, we want to make them accessible from somewhere. So there's two kind of buttons. This is what people often call a machine learning system. So a machine learning system, or AI system, is the system. This is the operational thing, the thing that runs, takes the user input, uses the model, makes predictions, and goes on. But that machine learning system could be just a Python app with a UI, like Streamlit, that downloads the model and makes predictions. You can do it all in Python. That's one type of machine learning system. Another one is where we host the models on a network endpoint and make queries on them. And that's what we call a hosted model. So it's hosted model-serving infrastructure. That's what I'm going to do here. You can even have another case, which is where you have a stream processing application that has an embedded local model. And as data keeps coming in, it spits out predictions. That's another class of machine learning system. So in this case, I'm just going to create some deployments. We're going to deploy the ranking model first. I'll explain how this works. So we're using something called KSERV. KSERV is kind of the de facto open source framework for model serving. And what I'm doing is, firstly, I'm finding the best models. Because we put the performance of the models in there. And I say, get the F score that's max. That's my best model. Get that one. And then in KSERV, we have two components. I'll show it in slides to make it a little bit clearer. Let's have a look. We kind of skip over this. This is kind of what we're building, is retrieval and ranking. We retrieve videos from our vector index, and then we filter them out, and then we're going to rank them with a ranking model. But this is what we're kind of... And I'll skip over this. This is just the same thing again. This is really what... This is the nuts and bolts of it. Our user interface app will go to something called a transformer. This is a Python program that we wrote, and that transformer will call the user query model. So the user's recent history will come in here. We'll go to the user query model to get the vector embedding for us Then we'll go to the next model the ranking model and we'll say hey Here's the user query the embedding from the from the user query that we've computed It will go to the vector index find the closest videos then it will go to the ranking model and say please order these videos for me and Then it will return the the ordered videos for you So it's using this vector index and feature store to basically first the vector index is to find similar videos And then when it's found similar videos it goes back to the feature store and says give me all the features for these videos because the Vector index didn't have the features. It just had the IDs for them So it goes back and says hey, I got like a hundred videos. Give me all of the The features for them so that I can make rank them for me So what we need to have then is these programs a transformer program a predictor program a transformer program and a predictor program This is the way case or works your request comes in goes to transform first then it goes to predictor in this case We have two models So we go from the transformer to the predictor and then the transformer will call the second model This one the predictor model and then that will come back and answer so that's what's happening in the code here and We basically define this Transformer so you can see it's doing it slightly backwards. It's got the ranking one first. So so inside here. We basically Basically, when we initialize our transformer, we connect to Hopsworks. We get a reference to our feature view. We get the list of features. We get a reference to the embeddings as well. And we download our ranking model. And then when the request comes in, so this is just the initialization step. When the request comes in, what we do is the request looks like this. It has a number of instances. These are videos that we're going to rank. And the first thing it does is it finds the closest videos. So it says, hey, I got in this user input here. It's going to be the user embedding. And it says, show me the 100 closest videos to it. It gets back the 100 closest videos. It removes them. We do a bit of filtering here. We remove all of the features that the user has looked at before. Now, you should probably do that, say, in the last hour or last day. Often you'll do filtering like, hey, is this user under 18? Remove all the videos where the movie is not for under 18s. Then we get a list of these videos that are filtered. We put them into a data frame. And then what we do is we call ranking on the data frame. So we're basically going to say, OK, this is my ranking model inputs, this data frame. I added the video features to the videos that I got back. We've added a couple more features to them. And then we call the ranking model, which is basically going to make the prediction. So that's going to be the predict program. So at this point, this return actually goes down, I go down here, it goes into this predict method here, and then what it does in the predict method is it pops out the features and the videos, and it calls model.predict. So this is our, the CatBoost model that's ranking and ordering these features, and returns the scores and the video IDs. And they return back to the post-process step here, they return back in here, and then what we do is we sort them, like we wanna sort them by their scores, and then return them ranked. So that's the two programs, the transformer and the predictor. Let me run these briefly here. And then we do the same for, and now we're actually going to start that model running as well. So we're deploying that model. And we also have the, yeah, further down we have, let me go down further to the second one, which is, we'll go down here. Okay, this is a test for us, let's skip that first, we'll just run the other one. The query models, we need to deploy the query model and let's do that here. Okay, so let's go back and look in Hopsworks and see if they started there. So yeah, we can see now our ranking model has been deployed and we can run start on it up here. I don't have a huge amount of time left so we're gonna kind of move through this relatively quickly. Okay, it's starting, so it hasn't quite started yet. And then we can basically do there's some cells here that will test out your recommendations Now the one thing you will need to do here is you can see it has this user ID here so you actually need to change that it won't work in your code because I just picked a user ID out of my out of here, so you need to basically go back here to your users and Then click in data preview Rebuild this page and then we can take any of these user IDs here that we can see. So I can take maybe this user ID here or this one here and paste that into my code because otherwise it's going to look up a user ID that doesn't exist, right? So it needs to be a user ID that exists and then there's a query embedding and so off we go. Okay, so it looks like it's started. You can get the logs there to try it out if you want. Excuse me. And boom, there we go. And then our second one here should have started as well at this point. Yeah, it started Okay, all right We're good to go. Okay, so that that's it. It's kind of done, right? So then now our these cells are just to test it to see if thing is working We can basically go back and look at our actual system. So this is our system here Now this system here is a is a streamlit program and it's using our model. Let's look at the code before we move on The code is down here. It's in the stream list folder. It's called app.py. So what this program does, this is our inference program, our online inference program. It basically gets a reference to our deployments. We get a reference to the feature views. It prints out some stuff. Employments are ready. When I click on if I click go back here and click on one of my videos here, okay? I've clicked on one and it's doing a recommendation what it's going to do as well as it's going to write that Directly back into the platform so that was that code that I looked at Here, so it's getting my recommendations now not super fast and But it's basically inserting that back in so that everywhere as soon as I've clicked on it It's writing it back in so that it knows then for the next prediction and whether to recommend it or not Okay, so it brought me back some recommendations here So then I can click on the next one and then it'll recommend the next one It's taking a couple of seconds, but not to worry So that's a this is the whole user interface is written in Python in a framework called streamless And it's not that big and not too Intimidating I think Okay, so there's it gives us some recommendations there. So that's basically it I'm gonna finish up wrap up here and And say, look, you know, I understand there's quite a lot to go through in this. This is not something you might do in 90 minutes. But it's doable, right? It's like, what do we have? Six Python programs. Not super hard. This thing that I'm showing you here is a Streamlit app that's running. I'm not paying for this. Hopsterx is free forever on the free tier. So you don't pay for that either. So I can go off and show people, hey, look, I write a TikTok recommender system and go for a job interview. I don't need a job, by the way, in case you were wondering. But I could go for a job interview and say, hey, look, I built this and explain it to you. This is pretty cool, right? This is like a step beyond just training a model on some static data and making some predictions, right? So if you're interested in that general concept, I'm writing a book on O'Reilly. There's a few chapters out already, but you can download the first chapter for free. There's also a community called Serverless ML that I'm involved in where people build these serverless machine learning systems. And you get some good tips and hints for how to build these systems. So that chapter is free, and it'll kind of explain a lot of what we went through today. Obviously, LLMs will come in at some point, not in the first chapter, but after that. That's it. I'm out of time. Thanks for listening and putting up with me. and PyData's just getting started so have a great rest of the conference maybe see some of you around I'm with Hopsworks, we have a booth upstairs if anyone wants to chat to us you'll find us up there. There's five minutes for Q&A does anyone ask a question? She didn't tell me that before, sorry so are there any questions? No, we're good, let's go, let's go get a drink we have a question online alright, go on, sorry She's showing me signs. I've got two minutes left. Yes, you've got two minutes left, but we have time. Do you want me to read it? Yes, just read it. Where's my glasses? I've got to put my glasses here. Jesus Christ. All right. So I can... Okay, so Sayil says, how can we also maintain fairness to different content creators and users exposed to new content? That's a great question. I don't have an answer for it. Look, my answer is open source. Right, you know, that's one answer second question is with the two-term modern similarity search you already have a ranking for users Could the re-ranking be avoided having a single end-to-end model? That's a really good question and Yes, you can of course Avoid the the last re-ranking stage and any of you who've looked at LLMs will not notice this sounds kind of familiar, right? This is like rag and in fact recommender systems are rag, but they're more sophisticated rag So you want to do sophisticated rag do two-term betting model for rag? don't just do vector embeddings and the yes there is a point that the the first phase the retrieval phase is using your user queries is using your features and it's using the video data to find related and videos but it gives you a couple of hundred so there's many things you want to do when you're when you're actually giving back that I didn't go through you want diversity or you don't just want to give the top five or ten they always end up being the same you want to also put in some random selections for things like that things that are trending right and then the re-ranking stage is really about the personalized the most recent things I've done and filtering and you kind of need that if you want to make it personal otherwise it'll be kind of just generic it won't be as personalized to you that's it thank you everyone no okay I'll take questions after