From Trees to Transformers: Our Journey Towards Deep Learning for Ranking

Speaker 1 [00:07]

My name is Mikhail. I'm a senior data scientist at Catcher Guide.

Speaker 2 [00:11]

And I'm Theo, data science manager at GetYourGuide.

Speaker 1 [00:15]

And today we'll talk about how our team adopted deep learning for ranking activities in our marketplace. So for the agenda today, I'll start off by giving a little bit of background about GetYourGuide, like who we are, and why ranking plays such an important part in our marketplace. Then I'll briefly introduce the learning to rank framework and how it can be used to build machine learning models for recommender and ranking systems. after which Theo will talk about the state of our ranking system roughly about a year ago when we embarked on this journey towards deep learning and sort of what motivated us to do so and then as well as how we kind of approached this migration when it comes to experimentation and what were the kind of challenges and learnings that we faced along the way and in the end I think we should have plenty of time for questions So we're hoping for some engagement from you as well. And we'll just wrap up with an overview of the model that we ended up with, as well as the key takeaways. So for those of you who are not familiar with GetYourGuide, we are actually the leading marketplace for discovering and booking activities online. So we operate as a two-sided marketplace, which means we have two types of users. On one hand, we have the suppliers of the activity, so think someone like a museum or a walking tour operator who wants to join our platform to list their activity online. And then on the other side, we have the travelers who are people who are planning their next trip and want to find things to do there. And ranking essentially plays a very important part in matching the activities to the travelers, right? So whenever users come to our platform and query our marketplace, the first step for us is to retrieve the set of activities out of all possible activities that we have online that could be relevant to the user. So we use pretty loose matching here. So is the activity targeted to the location that the user is interested in? And after we do this retrieval, the second step is to actually rank those activities in order of relevance for that user. And, yeah, as we know, screen sizes and user attention can be pretty limited. So this really just means that inherently the top positions that we display are much more valuable in some sense to the user. So, really optimizing the ranking of the displayed activities for us can go a really long way to help the user actually discover and see what they are interested in and essentially maximize their chances of converting. There's many factors to consider, which makes this a rather complex task. Things like the seasonal patterns of the location, you know, is it in Europe or somewhere in the Americas, for example, can have different patterns, different user constraints like the language that they're interested in, the historical performance of the activity, and even the user's own unique interests. So everyone is interested in different things, everyone's unique. So, for example, here the landing page of Barcelona could look very different for two users if one is interested in more kind of culture and museums, let's say, or architecture, and the other user is interested in more outdoor activities. Both of these are very valid for the landing page of Barcelona. So how do we go about this then? And the learning to rank framework has really kind of emerged really for the better part of the two decades now as the sort of de facto framework for solving this task. And it essentially boils down to using implicit user feedback from their historical user queries. So as they search our platform, our marketplace, to kind of determine what is relevant for them and then use that data to train a new machine learning model and kind of use it to serve new rankings so as users come to get your guide and interact with our marketplace they essentially submit their search query maybe some filters to go along with that query and we then in return return a list of activities to display to them a ranked set of activities I should say and this happens with like millions of users every day and as that happens they also interact with different activities shown to them so for example clicking viewing and maybe even booking something which generates event logs for us and then what we can do in turn is under this learning to rank framework using those event log we can process those events and essentially train a machine learning model on that data. So using the clicks and bookings, we can kind of then retroactively determine what was relevant and then train a new model on that. So kind of closing the loop. And then we evaluate the model. If it looks good, we publish it to our model registry to be then available to the ranking service and kind of serve new rankings, updated better rankings. So this is done every day. And at this point, I'll hand it over to Theo to give a bit more details on this part. So what exactly was our machine learning modeling stack and why we decided to move to deep learning?

Speaker 2 [06:14]

Yeah, so now I want to dive in a bit more into the ML model, which is this part on red. And what we get basically from the user is the query, and after it has pre-processed, we have the query and the activities, and then we need to combine them. So first we will get also information about the activity features from an in-memory database, And we also get information from the traveler history thanks to our feature store. And when we have all this information, we can combine them and they are passed to our scikit-learn pipeline. The first step will be to preprocess them. And for that, we use some scikit-learn transformer with doing operation on Pandas DataFrame. Then the second part is to do the predictor with this XGBoost. And what we used until moving, what we had around one year ago, was this XGBoost ranker following the learning to rank that Miha just explained before. And what we want to focus on and change is this XGBoost ranker and moving it to a deep learning with PyTorch okay but maybe you were asked why why do we want to move to deep learning first we saw that we had diminishing return without rebase approach so we felt that we were kind of reaching the limits of what was possible with with XGBoost second on the other hand we saw that deep learning offer a lot of flexibility, in particular in terms of model architecture and also being able to customize the loss. We also know that deep learning will be able to exploit much bigger data sets. And finally, we will be much freer to use more features, especially the ones that have a high cardinality for categorical features, also being able to fuse different features like text, images, et cetera. And finally, and maybe the most important, we also had external validation from industry case studies. So we saw that other marketplaces like Airbnb, Wayfair, Etsy also moved at some point their ranking from tree-based to deep learning, and they all claimed that it was transformative. So based on that, we saw that we'll just use deep learning and throw GPU at it, but it was a bit more complicated than that. So a bit more seriously, by looking at the other companies, they also share a bit of their journey. And also, we talked with some people. We knew that we would be expect like a long and complex process and for some time without a business win. And indeed, yes, that took us six months to get to the first stage. Then we also know that going to this deep learning will probably go with some infrastructure investment. In particular, we expected to have a larger training time. And yes, that really is a problem. And actually, we had to change, reconfigure a bit our continuous integration to be able to handle that. Then we also expected to maybe have problems with GPUs. Actually, it was easy to train on GPU. However, training on multiple GPUs was a bit trickier, especially there are some weird things happening with embeddings and the max norm in there that was took us quite some time to figure out and finally the inference latency thought that maybe will be a blocker for us and we also need to tweak the torch cpu threads uh setting and also we are seeing also some weird things happening now with the memory usage and latency that play some interesting role but what we were the most afraid actually was what is not shared in the blog post so what will be the unexpected thing for us and actually we got one the docket size at the docker image when we added pi torch at first we just like exploded the docker image by getting more than 4 gigabyte which causes some problem to our continuous integration pipeline that was running out of memory, but we figure out a way around. So, yeah, that's the thing and what we were expecting and what really happened. Now I will show you a bit more how we broke down this deep learning to major milestones. And how we did that basically is first in Q2 last year we decided to focus on getting the technical parity between the between the tree and the deep learning model, in particular in midlife, just being able to understand the throughputs and the latency requirements of our model. And for that, we started with a minimal hackable network. We spent a whole team working two focus days trying to pair together to get that happening, and we just got with that, which is a very neural network that was just trained on four features instead of the 50 that we are using for tree-based for a tree-based model, and we were able to deploy it behind a feature toggle to be able to do internal testing. With this, we were able to start planning and preparing our second iteration, minimal variable network. So basically, we were using a very simple architecture, but we were able to use all the features. It was just okay on our ranking metrics, but it was still not optimized for all our business goals, but we still wanted to launch an A-B test and to get some learning of it. So for the one who are not familiar, A-B test is basically we divert 50% of our traffic to have the activities ranked by this new model instead of the tree-based model. So we have some real data of what will happen with this new deep learning model. And so we were able to validate that throughput and latency were there, but we still had the last step that was still not there. That's why we launched this minimal lovable network where we also were able to achieve all the business objectives. So we had the same business objective that the tree based. And also we had a ranking metric that was better. And that's how we concluded the first step of the first milestone for Q2. Yay, we were happy. Then in Q3, we continue and say, okay, now we are trying to move towards a business priority. Meaning that we want to have the same business performance as the tree-based approach. And so for that, we started experimenting with new architecture. And what we saw is changing with a transformer architecture with using self-attention help us. And we see big improvement in our ranking metrics. And we were pretty excited to test it in an A-B test. But what we saw is actually we had more people bouncing on our website, meaning we had more people on the B side when they saw the new ranking, just look at the page, but did not click on anything and just came back, closed our page. So with this learning, we say, okay, we need to go to something else, and we introduced the clickbait transformer, where we reduced the same architecture, but then we increased the data set to also add data of visitors who clicked on activity and not just booked activities. And then we saw a flat business result, and more specifically, we also saw that we removed this problem of increase of bouncing. So that's why we achieved our second milestone of business parity. And now we said, okay, we're ready. Now we can get to get better, right? We want to go to deep learning because we want to have a better ranking. And actually, in Q4, that was the goal. And it took us still quite a few experiments, so four experiments later. Without a business win, we finally got our first win. So here what we did is on top of all the things that I explained before on the transformer and the data set improvement, we added the semantic similarity between the activity that we have and the category of the page. For us, the category of the page can be like the wine tasting in France will be a category or like monk meditation in Japan. All of those are categories for us. And we saw some uplift actually in the ranking metrics, but more importantly, we saw a significant uplift in the number of bookings. And yeah, that was just before Christmas. It was a perfect Christmas gift for us. And then in Q1, this year we had our biggest win from the last year with the latest model. So here we started to go much deeper into the architecture and introduce a deep and cross neural network, also with added trainable embedding for our categorical features. And we also increased the size of the data set. And we saw a clear win in our ranking metrics, and the booking and revenue were also significantly be positive in our A-B test. OK, with that, I will give it back to Mihal, who will show you a bit more of the current architecture.

Speaker 1 [16:26]

Cool, thanks. So maybe at this point, some of you might be wondering, okay, so what does it actually look like? What is the model that you ended up with in the end? And well, here it is. I can run you through it quickly. So, basically, our ranking model that we ended up with at the end, as Theo just explained, is this deep and cross architecture, but it actually consists of two main components, so the first one being the deep and cross network that's shown here, and then on top of that, we have the ranking transformer at the very top of this graph, and it all starts with the embedding layer. So the task of the embedding layer is essentially to take your sparse features and create dense representations for them. And in our case, a sparse feature could be a simple categorical variable like the language that the user selected or even a specific entity, so the activity ID or the location ID that the ranking request contains. So we're training embeddings on the entities themselves, the entities being ranked. After that, we actually concatenate the embeddings that are generated by the embedding layer with other dense features. So this would be like the historical performance of an activity. And then there's two parallel kind of subnetworks here. The left one highlighted here is the deep network, which is your kind of standard fully connected network with some nonlinearities in between. So in this case, we're using a reload activation. And the task of this deep network is to essentially generate kind of highly nonlinear and highly dimensional features out of the, yeah, just generate the features. And then on the parallel to that, we have the cross network, which actually applies linear feature crossings to the inputs. So a feature crossing would be something like the example shown here, which is a second degree feature crossing. It's a simple and statement. So if the category of the tour is brewery tour, or activity rather, the category of the activity is brewery tour, and the language that the user has selected is German, maybe that activity should be ranked higher. So this kind of explicit linear feature crossing is something that the cross-network can do very effectively, but on the other hand, traditionally, deep networks with non-linearities kind of struggle with this kind of representation. So after that all is done, we actually concatenate the two outputs of those networks, the two sub-networks, and feed it further to the ranking transformer. So this is, of course, for those familiar with the attention is all you need paper, it's inspired by that. And this network actually consists of two encoder blocks highlighted here. So each encoder block has a three-headed self-attention. Followed by a feed-forward network with some skip connections and layer norms in between. And after that is done, there is the final kind of just linear layer that gives us the relevant score. So then based on that score, we actually rank the contents of a specific ranking request. And the task of this transformer is essentially to kind of look at the contents of the ranking request the query, so the different activities contained in a single request in a more holistic way, so kind of modeling the entirety of the query. So at this point, I'll hand it over back to Theo to give you some of the key takeaways that we had.

Speaker 2 [20:42]

Thank you. So, yes, I think from this journey we had three takeaways. The first one is actually start simple. So we actually reuse as much as possible from the XGBoost model from the start. That means that we use the same data set, the same preprocessing using the scikit-learn pipeline, and then we incrementally change, we incrementally test the changes, first offline and then online. And so in that, that's how we end up testing more than 50 iterations and launched 10 A-B tests in the six months. And that actually allows us to learn quickly and readjust all the things. One good example is the people bouncing. It's not something we could only learn through A-B test. Then the second one that I think was very key for this project was to split the project in different milestones. So here we break it in three milestones. The first one was just to get the model to have the technical parity. priority, then business priority, and finally get business improvements. So that allows us to celebrate the achievements and also help to show the progress that we have to our different stakeholders. And finally, stay agile and be ready to expect the unexpected. You can prepare and learn from others, but be ready for some bumps in the road. This is a big migration, and there will be some surprise on the way. Great. With that, that will be the end of our talk. I just want to say thank you, and in particular to all the people. So there was a full team working, and in particular, like Nikhil R.P.M., Julie also, a data scientist, and Olivia and Xingting, our MLOps engineer that's made it possible. So yes, thank you, and looking forward to your questions. Thank you.

Speaker 1 [22:47]

Thank you.

Speaker 3 [22:55]

Okay, thank you too for the presentation. We have a lot of questions from the audience. So the first question is, since you update your model daily, have you ever investigated or experienced cyclic model retraining behavior due to the day-of-the-week effect, for example?

Speaker 1 [23:14]

Sorry, what was it? What kind of behavior was it?

Speaker 3 [23:18]

What's that? Cyclic behavior in the model .

Speaker 1 [23:22]

Yeah, for sure. So for sure, as I mentioned, there's like seasonal patterns that we need to kind of be aware of. And this also includes a shorter kind of term. So within the week, what day of the week, typically on weekends, for example, we see a lot more traffic at different times of day and et cetera. So that is something that we need to be aware of and take into consideration. So it could be as simple as adding the day of the week as a categorical feature and maybe creating an embedding out of that. So that would be the way we would go about it.

Speaker 3 [23:58]

So how did you realize or quantify that you are reaching the limits of XGBoost as a method?

Speaker 2 [24:07]

I mean, basically, we were still experimenting on our tree-based approach, and what we were saying is, like, getting the next wins was getting harder every time to get a successful A-B test. So we had a few iterations of experiments where we saw, like, okay, we see that offline, we don't see as much uplift that we hope to get to a significant result. And also, we also get to the point of we think that the next step will be to do that, but actually it's not part of the framework and we'll have to hack things around in different ways to be able to make that happen or actually come naturally when you go to deep learning. Like, for example, we had problems with high cardinality, cardinal feature, which if you want to do it tree-based, it's just like hard encoding. But when you have 1,000 cardinality, you have a very huge vector which has a lot of problems in terms of latency and also just like... Ah, I'm missing the word. Yeah, the performance of the model, yeah.

Speaker 1 [25:22]

Yeah, and I would say, like, besides just the kind of diminishing returns that we saw in terms of, like, model performance and A-B testing, I think we really like the flexibility of PyTorch as a framework. So it really allows you to kind of have a lot of flexibility in how you configure your model. You can easily set, like, custom losses in pretty much, you have a lot of freedom to do things there. So those things are not so straightforward with our previous framework. So that was another big reason

Speaker 3 [25:54]

So another question that interests a lot of people is which metrics were used to evaluate your rankers?

Speaker 1 [26:02]

Yeah, that's a good question.

Speaker 2 [26:04]

I would prepare it actually let me go

Speaker 1 [26:10]

Yeah, so I would say, so there's a lot of different metrics, actually, that one can look at when measuring relevance. I think there's a ton, really. And I think, so here is an example of how you would go about it. And this is probably the most straightforward metric. It's called the average precision. So essentially, you need to first declare what is relevant as a binary label, 0 or 1. So in our case, it could be the activities that the user clicked or booked, as shown by the two different colors or highlighted by the yellow and orange. And then you go position by position. So this sum k, which signifies the position, you kind of average the precision up to that position. And then you take the total average as the kind of one ranking metric. So, yeah, I mean, the point is to, like, essentially, or the goal of this whole thing is to, like, rank the most relevant things, which, again, is under the learning to rank framework is just used or determined by user logs, so the clicks and bookings, and then take those and try to rank them as high as possible and then serve that model. So here's one example. There's other metrics like NDCG, very common also in the literature.

Speaker 2 [27:30]

So we looked at a lot of metrics. This map is actually the one that showed the biggest differences. For example, the NDCG, we don't see as much delta between different rankings, whereas the map was actually the most relevant one. But actually we looked at a lot by categories, the price of the top activities, etc.

Speaker 1 [27:53]

Yeah, it's very specific to the problem and the data as well. So what works for us might not work for everyone else either.

Speaker 3 [28:01]

So we still have time for one last question for one minute. How did you handle the latency of transformers? So was it an issue in the beginning and how did you try faster implementations?

Speaker 1 [28:12]

Do you want to take this?

Speaker 2 [28:14]

Yeah, I think the transformer in itself was not a major problem in latency. The overall architecture with the dip and cross actually took, it's taking a bit more time, but it's more the introduction of the dip and cross that causes some latency issue. But the transformer in itself was pretty efficient, so we didn't have to do a specific thing.

Speaker 1 [28:39]

to make it work. And I think there's a lot of trial and error. We have some very good MLOps engineers who come up with some really clever tricks to help there. So I think there's no one solution or one answer for everyone, but I think it came down to a lot of trial and error and make the improvements that way.

Speaker 3 [29:01]

So thank you for the presentation and also for answering the questions We were unfortunately not able to answer all the questions But maybe you can come to the front or I can send you the questions and you can answer them on this code So thank you again for the talk. Thank you

Speaker 1 [29:13]

talk. Thank you.