Personalized Restaurant Recommendations at Scale combining Transformer with Gradient-Boosted Ranking

Speaker 1 [00:13]

Please listen to the talk and then submit your question on the website of PyCon Day. Today we have Marcel Groski with the personalized restaurant recommendation at scale combining Transformer with Gradient Boost in writing at the Paladin Room. You are welcome.

Speaker 2 [00:37]

I guess I was never starting a talk before the planned schedule time, so that's good. One minute more to spend on talking, which I have almost wasted by now. Today I actually want to present some of the work that my team has been doing over the course of the past year on ranking personalization and how to improve ranking personalization at Bolt for our venue recommendations particularly. and here is the first like a very bold specific term so we usually talk about restaurants, stores, supermarkets, grocery stores and we all like use the venue as an umbrella term for this so every time you are going to hear venue from now onwards think about restaurants think about grocery stores and so on and so forth and I actually want to share with you how we achieved a totally new model that has already substituted three existing models and is soon hopefully also going to substitute the fourth model with a combination of two models that we joined together which is a transformer of course nowadays everything is about transformers, LLM, DDI, whatever but it also has very useful applications for personalization recommender systems and how we actually put that together with gradient boosted ranking which might sound a bit boring but I guess good old gradient boosting is quite powerful and it was a nice thing to join together with our venue ranking here but before diving into the talk and the specific itself i first want to outline the agenda for the talk so first a little bit of introduction and motivation also what world is i hope most of you have already like come across volt and seen us we are operating in more than 30 countries um and i actually want to continue onwards with like the core part of this talk that is about the model architecture for uvr this is how we refer to our model it's It's a universal venue ranker. And then particularly share a couple of details on how we explicitly pushed for exploration, which is a big problem for us as there's a lot of repetitive behavior and striking the balance between which users might need more of repetitive patterns or have more or less taste for discovery and those users that are more interested in exploring something new and how to strike the balance between these two things within one ranking model. and also share a couple of insights about the production systems. Unfortunately, my colleague Stefan couldn't make it today, so I will share a bit of these things here. He was a main contributor of this project, so a shout-out there to him. That was definitely a core stone in making all of this possible. And then I also want to share a bit about the results that we achieved with this offline and online and like the reason why this model is nowadays powering our venue recommendations at Volt successfully. And also a bit of the follow-up ideas that always come across before, during and after the project, something that you want to improve and extend further. So let's start with introductions. So just to give you a little bit of overview of what Volt does and what we are about. So we are there like to bring joy, simplicity and earnings to the neighborhoods of the world and we basically connect three kinds of people together. For most our customers that we serve, our versions which we also serve and also our career partners by providing them earning opportunities and bringing like all these stakeholders together in one platform and being part of their like large delivery platform called Volt. So Volt being part of DoorDash and we actually operate across more than 30 countries worldwide. So this is only Volt, with more than 300,000 career partners, with 60 million plus users, so basically the people who see these rankings, who see venue recommendations, item recommendations. So quite a large, yeah, a big part of people that are affected by the technology that I have the joy to work on with my team and with more than 15 000 employees worldwide and we are part of doordash so we joined forces in 2022 and recently as you might know uh has also acquired deliveroo so it's now like these three companies that are joining forces together doordash deliveroo and volt and like taken together we are even operating in 40 countries which is a very interesting and cool journey because you get to know what the others have explored or developed so far what we have developed and joining these forces together to make it really an excellent platform for for our stakeholders and of course this comes with a whole lot of recommendation and ranking problems personalization I hope that I don't really need to make the case or convince you that personalization is playing an abundant role in everyday mainly virtual life so So be it social networks, be it e-commerce, be it media and TV streaming. So everywhere you are somewhat affected with recommendations and personalization. As we have learned over the course of the past even more than 30 years, that recommendations play a pivotal role in serving users' need for personalized content. And of course, the platforms also benefit from this quite a lot, like by delivering personalized content we drive up engagement, purchases, whatever which of course is ultimately in the interest of a profit-oriented company. And how we do this at Vault like there are very different means of things where we can personalize and you can basically come from a very high level go to the nitty-gritty details but in the end like we are a platform that is providing these centers our clients, our merchants the opportunity to sell everything they have on our platform, through our platform. So that means, if you think about that, like every kind of venue, a restaurant, a store, a grocery store, a beauty store, whatever, can be ranked, can be ranked in specific categories. We can rank categories themselves, be it food categories or be it also like different product lines, like groceries, fruits, whatever. and then like assortments of stores like if you think about this carousel-like interface that you see quite commonly then this is also another problem like which carousel to put on which position to make it the most engaging and most discoverable experience for the user and in the end like you're not going to buy a restaurant or buy a store you are buying from a restaurant or store so you're mainly interested in the items in the dishes that they are offering. You are interested in buying groceries. You're interested in buying maybe your favorite pizza tonight because you don't like to cook. So this is, in the end, where it boils down to recommending items and also where scalability plays a bigger role than for the venue recommendation case, which I will also outline. And this is actually part of the team that HACE has made this possible. So all our great colleagues, It's a very diverse and multi-country team, and we are joining forces and working on all these kinds of problems, like the venue recommendation ranking, item recommendation ranking, carousel ranking, all kinds of personalization from the world side. This is not the whole team, so I'm very happy that recently we had two new joiners on our team, and actually we are still hiring, and I've also added a slide for this for the later part. It's my first talk where I'm actually able to say that we are hiring, so I'm very happy to say that. And all of these people were actually contributing to the success that we saw with UVR. So shout out there. And UVR was not the very end of this story or the very beginning, to say it more precisely. But there was already a history of a model landscape that we had. Like in 2023, when I joined the company, we had newer collaborative filtering in place. So for those of you that are a bit confident with collaborative filtering, It was basically like an MLP-based version of collaborative filtering. There has been a lot of debate around it, like how good it really is. So in 2019, there was a paper in a REXIS conference debating the usefulness of recent deep learning-based recommendation models and how powerful they really are. But happily, though, their staff applied scientists back at the time, Sascha has put in a lot of work to make that usable for Vault and was very successful with it, making it the first really personalized model for Vault customers. And then shortly after, we also tackled the challenge of the question, what do we do with our so-called cold start users? So with users, we have no knowledge of who come to the platform for the first time, who haven't actually performed any purchases, done any clicks, whatever. so how do we learn what they really like and how do we come up with our recommendations that from the early on of their early on of their journey help them to discover the content that like caters their needs shortly after also in 2024 2025 we then introduced a like re-ranker which is called the second pass record that was working on top of newer collaborative filtering so both of these models were working together and cf and sbr as one ranker for our restaurants business But as I said, we are not just the platform that delivers your pizza or your burger or your sushi. We are a platform that has way more, like, grocery you can use through our platform, buy electronics, whatnot. So the whole, like, universe of what I usually refer to as retail, that in my definition includes groceries. And in this, we also said, hey, we need something there. We also need a rampart for our retail domain. So we went out there and built a very basic and very simple version of a retail ranker that was beating our, by that moment, very naive baseline very much. So also not a big surprise that we rolled it out. And this was our retail ranker, which was a purely CatBoost-based ranking model that we used there. And then it was last year where we also came together and said, yeah, our model landscape is growing. We are joining forces. with DoorDash so how can we simplify things but how can we also make them better at the very same time and this was somewhat how the idea of UVR was born which we um yeah used to at least achieve three different goals one was of course our foremost goal to improve the business top line of course we have to convince our stakeholders and we just can't build things for the sake of fun so they should actually also in the end benefit the business and be like one of the most efficient uses of our costly time so we said if we like invent a new model and if we merge these models somewhat we of course want to improve our conversion rates in the restaurants and overall in the global business but also some problem that we found that we had been dealing with is this problem of are we really catering the needs of users that want to explore something new And we also learned that users who are trying more new content, like trying new restaurants or new venues, are also more likely to retain the platform, which is, of course, also in our interest. So how can we actually strike the balance without showing something totally irrelevant just because it's new versus showing the same thing over and over again? So there's a lot of space in between, and I guess with this project, we were successfully able to explore that space and find a good solution there. Not an optimal solution, I guess, but striving for optimality, I guess, is something that motivates us there. The second point was, of course, I've shown you the landscape, the growing landscape of at least four models that are solely focused around ranking the order of venues. We also have other models that we have to care about, like checkout item recommendation models, carousel ranking models, whatnot. So the complexity is growing. So you should also ask yourself, how can I actually reduce the complexity or limit the complexity that comes with new models, additional infrastructure and systems that you need to build around it, alerting, whatnot. So the idea was also decrease that landscape by unifying it into a single model so that we don't have even more and different and also slightly outdated models in the sense that all these repositories look slightly different or look slightly different. And in the end, if you can make it more efficient, then maybe as a side quest or as a side benefit from this you can actually also reduce the cost that you put into training those models, serving those models. models because you don't need to serve that many models of course in the end you serve a much more complex model so could be the case that in the end you end up not saving there but if that model is at least more powerful then I guess that's a good case so yeah this is when we went out and and started to work on UVR and by that time transformers were already like playing a big role in recommender systems and are actually quite an old technique if you see like how fast this field is progressing. So nowadays everybody talks about fine-tuning LLMs, about, you might have heard about semantic IDs and how to leverage semantic IDs for recommendations. So there's a paper by Google from 2023 on generative retrieval of semantic IDs. And there's been a whole lot of history since then. So first thing that we did is, of course, like, what is kind of the recent trends that I wouldn't even consider two-door models anymore as a big recent trend, it has more become like the de facto architecture that you saw in many recommender systems, like having a user tower, having an item tower, and then having like some kind of interactions between these towers and in the end create embeddings that you can use for like very efficient approximate nearest neighbor search. So this is basically the idea that has been adopted a lot in the industry. Nevertheless, this idea of sequential recommendations has also been lingering around for quite a So especially like with the rise of deep learning applications almost a decade ago in recommender systems You saw a lot of RNN based architectures And then like when this paper which name we all know by heart attention is all you need appeared in 2017 It was no surprise that shortly after this paper has appeared first researchers went there and applied transformers and the attention mechanism actually to sequences of user directions of user purchases and this was actually in 2018 and I will show you just the paper and a little bit of comparison and then shortly after more architectures, more elaborate things appeared like BERT for RIC in 2019. So actually like what we did in this project, one could aggressively say is already like five old year architecture because the field is just progressing so fast and everything that has been invented in one field gets sooner or later applied to the problems of recommender systems. So yeah, let's at least catch up on these five years, and then maybe look into the future of what we can do with general interpretive or semantic IDs that my colleagues are already working on and that we are exploring, especially for item recommendation cases. So that's quite a lot of visualizations for one slide, so please apologize, and I also don't want to go into everything but point out the key message. So I guess the very left image everybody knows so the transformer layer from the original attachments all you need paper what you see like a second thing from the left is already an adoption of transformers for recommendation so this is why it's called BERT4REC so it's the bi-directional encoder representation transformer that is applied to create based on a user's interaction sequence So it could be purchases, clicks, could be both, creates a user representation on the fly that you see at the very top at the projection. And then you use this user representation to solve some ranking or some classification problem. Whose output do you then use to enforce an order on whatever you want to rank in a personalized fashion? And the key difference that there is between the two-tower models is that for two-tower models, you generally had more like static user embeddings, not always. You can also like pair transformers with two-tower models. But what we had here is that the user embedding and the user representation is born far more implicitly. So you usually represent like all kinds of your items with embeddings, but not anymore the user explicitly. However, this is also something that the ZASREC paper, those architectures shown here, So this was kind of one of the very first applications of transformers to recommendations actually did. So they had a causal self-attention mechanism, as you can see there. And they also played around with explicitly injecting additional user embeddings. But what in the end turned out to be very powerful is this transformer-based representation of a user and leveraging this user representation. And you can see it somewhat applied in many recent recommendation papers quite recently. So here to the very right, you for example see a paper by Pinterest Talks from 2025 that was also presented at last year's REXIS in Prague. And there you see that they have this user sequence transformer that is taking into consideration interaction sequences, even though for Pinterest they use far longer sequences than we use right now and also represent different kinds of actions. like if you are dwelling on a certain pin or if you are saving something, if you are just clicking or if you are even converting from a pin that is related to an ad to sell a product and then this actually goes into this two tower architecture and this is what I mean by you can combine both things together and we just did an even simpler thing I would say like we combined this with a cat loose breaker. So there's even far more room for improvement, I guess. and yeah when it comes like to the generative AI applications then this is what you find more and more like using semantic IDs so very worth reading paper recommender systems with generative retrieval you will find it also in the slides and actually combine this with the architecture of transformers So, like, create these tokens that represent items and use it within a transformer and then create recommendations for this. And somewhat a problem was that people first thought this could only be used for retrieval, like to distill a large corpus of items down to just maybe 100 or 1,000 in that domain. but then what you found is that you can also like create a more integrated architecture that does retrieval and ranking together. Okay so a lot of talk about what others have been doing, what our plan was what we have been doing in the past now let's focus a bit more on what we are doing more recently or right now what is in production one of the common problems that we have is find good proxy metrics offline that help us to take them as proxies for what is possibly happening online. Happily, though, I guess our problem is not really, how to say, difficult in terms of the metric, because over the course of many experiments, we have already found that improving the MRR, which is the mean reciprocal rank of the value the user converted from in a list of venues and taking the average over many sessions is a very good proxy metric for our conversion rate of our users. So like the higher on average the relevant venue appears to the top, the better the ranking in general, the more likely it is that the user is going to convert and with conversion then comes what is one of our core metrics, our GOV, which is the gross ordering volume. So basically like what you know from eBay or Amazon as well like the gross merchandising volume and this is just a similar metric to that but the question is like how to improve the mr because the mr is a metric it's not your actual objective that you're optimizing for so they're like one of the common things that you use in recommender systems is bpr patient personalized ranking loss which is kind of a sequence applied to the difference of the activations from a positive and a negative item interaction and yet the idea ideal case it would work that way but sometimes it's more realistic that like your numbers go through the roof and like the AB test shows no uplift and conversion rate or GOV. Happily though we were able to show some uplift since we also learned that we can take this MRR as a very good proxy metric but beyond that we are also thinking about effects like that play or like unfold in a more longer term time frame which means like if the user sees something new how likely are they to be making more orders than users who are exploring less new menus so this is also something like very important but therefore we have our smart analytics folks who can do the math there something that is also maybe a bit easier in our domain is that we for venue recommendation actually don't have to play with that usual like cascading architecture recommender systems where you have retrieval first or candidate generation first and ranking second at least because you have a corpus of millions of items that you need to pick from and you can't rank them all in one pass as it would just be very inefficient and then you have to deal with latencies that you can't fulfill or latency requirements so in our case this whole candidate generation comes way more natural because if you open your app and say like I'm located here at home then of course there's only a limited amount of venues that are delivering to your location because there are delivery radii in place also it comes down if the venue is available if they are not too busy then they sometimes also turn it off and are not delivering also if we have enough careers and sometimes maybe also our streets might be blocked and you can't pass them so we need to reduce areas nevertheless in the end this boils down far more naturally to us having to rank like hundreds or thousands of venues which means that there is no step needed for retrieval and for a large chunk of venues down to a few ones which i then rank so what we already do is we always directly jump into ranking those venues even though this ranking is a two-stage process but it's not like a cascaded architecture that you commonly see in recommender systems where We have retrieval first and scoring and ranking second. So yeah, and as I said, we rank all kinds of things, be it categories, be it carousels, but what we are looking at is basically the ranking of the venues in a carousel or on our restaurants page, as I've already shown you before. And then I said MRR, and there are three kinds of MRRs, or three dimensions where we define it. We define it for new users, and new users might be a bit confusing, so it's more like new purchases. So if the user hasn't made a purchase from a venue in the past because they are not an FTU user, say this is not from ISFOM or Gantt, then this is treated as a new purchase. And then you compute the MRR, like for example here, the user purchased from the fifth position, so the MRR goes down to 0.2. And the same way we apply this to recurring purchases, no, to recurring, to new purchases, and also to FTUs. This gives us, for every session that we track where we know what the ranking was, where the user purchased from, the mean reciprocal rank over lots of sessions, which we take as a criterion here that we want to optimize for. And of course, as you can assume, predicting something recurring is far more easier than predicting something new. But we want to solve the harder problem, which means becoming better at predicting something new, so that the whole model, assuming, but this also plays out, becomes a more discovery-oriented model that guides our users to try actually something new. And we actually broke this problem into two subcomponents. One is predicting the next purchase, and the other one is ranking values and combining them together into two models like constitutes the whole URIOM model. And when I think about or talk about next purchase prediction, then just substitute one word there, which is purchase by token, and then you have next token prediction. And here you are, like transformers are your friends. So this is why we basically generate lots and lots of user sequences, which are basically their purchase histories. And you can do and broaden that to many different actions as well. But let's only focus on purchases. So like a time-sorted sequence of user purchases from the venues. And this, like, we put into a transformer to then take or create a representation of the user and use this representation within a multi-class classification problem to then impose logits for all venues that are available and use that as an additional feature for the subsequent model that is then solving the venue ranking problem. So first step, next purchase prediction. Given a sequence of value purchases, how likely it is the user is purchasing from one of the available next venues. And then the next problem is the pairwise venue ranking, as I already said, for which we use some BPR loss-like function. I mean, in Capital Spranker, it's called pairwise logit or pairlogit, but it's basically just a resemblance of Bayesian personalized ranking. And here's a question of basically you have these pairs of items, so you have a user-positive item interaction and a negative sample where you sample a negative interaction. And then you basically let those, you could say, compete with each other. you want actually the prediction to be as far from each other as possible so that in the end by assigning the scores the positive item will always appear over the negative item and this you do like a lot of times that you can do it with different signals but the purchase signal is like one of the most reliable signals here a click you can say okay the user clicked it because they were interested but maybe they clicked it because they were first interested and then they saw okay this is not something useful for me and then they went back so like interpretability with different signals that users provide needs to be taken with a grain of salt so we are taking the purchase yeah let me skip it but nevertheless what you need to take into account of course just ranking items against each other without any context doesn't make you learn a lot so you want provide a context, like features about the values, features about the user, so that the model can learn, given these features and this context, is this value to be placed over another value? And then, like, in a, yeah, very simple way, this is what we actually do. Like, the overall model is a composition of a predictor and a ranker, where the predictor is actually a transformer model, and the ranker is a cat-boost ranker model. So a gradient-boost decision tree that that's trained for ranking criteria. And then we enter like the venues, like which are the venues that we want to rank, which we are like provided for naturally as we know like which are delivering, which are within the delivery radius. And then we provide like lots of contests to finally be able to assign scores and take those scores to enforce an ordering on the venues. And some like very natural way of how you do that, most of the time just takes the scores and then rank the venues by the sending scores. But you can also go far beyond this. So because this is a deterministic way, you can also use it or do that in a stochastic fashion by using them for sampling and ordering on the venues and use this ordering to then show to the user, which adds some stochasticity to the system, but which is very helpful when it comes to some advanced techniques like counterfactual evaluation and learning for recommendation systems, which can sometimes lead to even fancy terms, possible that is called offline A-B testing. And yeah, here's basically the architecture that I said. So in terms of what we have, there are purchases, there are venue clusters. The venue clusters are basically sometimes made me think about a McDonald's. All McDonald's are treated the same way, so this is why we reduce them to a specific cluster of venues that are identified by belonging to the same brand or same franchise. and then you have the user sequence transformer and the sequence transformer scores that it assigns to the different venues are then one of the features for the cat boost breaker so it's very straightforward taking those scores that are assigned to the venues that are available which are basically we call this the user net score and then accompany this with lots of different features about user venue interactions like for example the click count You will definitely find a detailed list of the features in the slides, and I guess I will go over about this quite more quickly. But you also have venue features, so think about conversion rates or the venue's popularity or the cuisine that they offer, and also user features, as I said, like the historical click counts for venues, but also things like how well their taste matches the cuisine and deriving different affinity scores from that. and supplying this to the rater as well. And then we basically feed that in and use something that we call our shadowing data. So the shadowing data is basically a capture of a user's request of the ranking that they saw and then combine that with a purchase that we observed for this session so that we know how did the ranking look like at the time the user made the purchase so that we can evaluate this ranking in terms of where do you put the actually relevant value for the user. Yeah, the sequence transformer doesn't only get access to the venues, but to different signals, and I guess a bit better for this to show might be here. So what we have, we have a sequence of user purchases at the beginning that are feeded into an embedding layer because we do not only use the venue, but we use other signals as well. So for example, the timestamp and also the location of the user. So in terms of the embeddings, like we have a venue embedding for each venue that is available but we also embed certain time features like the calendar week, the day of the week, hour of day and also we map user locations to their corresponding hexagons on a certain level and embed those hexagons so that the model is basically also able to learn like similar taste within the similar neighborhood, for example, that also might help with cold start quite a lot. And then also we have kind of a day difference to tell the model how much the past an actual venue was or an actual purchase was. And then what we put on top is very much resembling this architecture that you have in the BERT4REC paper. So you basically use an encoder-only transformer that in the end basically gives us a user representation that is not particularly shown here, but that we use to project it into the space of all venues of the country to then assign scores to every venue. And these scores, as I said, will then become an input for the ranker that is then trained on pairwise ranking. And this you actually see here. So it's basically the standard CatBoost ranker model. And as I said, one of the unsurprisingly most important features is this user net score that we get from the transformer. and then lots of different features on the user, on the venue side but also on the context side like for example degree time is also something that matters to users especially if you are hungry. And yeah, then of course there is a lot of pipeline around how we evaluate it and then push all the results and the models into MLflow for later analysis and also we have the deployment criterion in between but maybe more to this in the engineering part. and what we do is like we train these two components separately so first the transformer and then the ranker and the transformer basically gets lots and lots of user sequences so there's one sequence for every user containing all the purchases of that user with some filters over a certain amount of time and then we split this data like into a hold-up group of sequences and into a training group of sequences and use them to train the transformer model and once this has converged We use it to score the data for the ranker, which then also gets these additional contextual data that I just showed there, and use the last 30 days for training the actual ranking model, so that it has four weeks, one day for validation, and one day for testing. And we only, so we evaluate this on sessions that have all values to not like have any bias in our, or not like a bias of selecting already values that the user has seen because it would be like information leakage from the future. However, when it comes to the training data, we only focus on seen venues because then we can have harder negative samples, which means like a venue that we know the user has seen, if that was sampled as a negative, then this is a more informative negative sample than the venue that the user never saw. Because for the venues the user never saw, we cannot really very nicely assume that the user didn't like it because maybe it is just missing because the user didn't scroll there. And then, as I said, we were also pushing a lot for exploration. And the cool thing here is there are several levers to push for exploration within the transformer, but also, like, within the ranker. And, like, we find some nice ways to really find a way of where we wanted to put our model to be at and that were showing us some really nice trade-offs between, like, recurring MRR. So the MRR computed on all sessions where the user made a purchase from a venue they have purchased from before or new MRR so the MRR for sessions where the user actually purchased from a new venue and yeah so what you can come up with like you can like tweak around the negative sampling which is also quite a common lever in recommender systems to do that but of course you can also like create weights for your samples for the transformer for the raker, play around with regularization, additional features for exploration. So, for example, we also quite recently introduced a feature that is basically summarizing the user's appetite for diverse recommendations or how diverse they taste. So think about like you made 20 purchases, and out of these 20 purchases, there were just two distinct venues. Or think about you made 20 purchases, but they were from 10 distinct venues. So this is, for example, one very simple way of how you could represent this user taste for more variety as a feature to the model. And you can use it for the short, mid, and longer term in all kinds of dimensions. And one thing that we found that was quite interesting, though, is here on the x-axis, you basically see the new MRR, and on the y-axis, you see the recurring MRR. And like every point in this more in this image is one specific model for a specific country So it's one country in that sense It's actually comparable But it's one model training and then you can play around with the max pairs. The max pairs is basically nothing else than the negative sampling ratio, but what you find is the higher your negatives are like first you play around with the weights and what you see is like like the more balanced it is, like the closer new and recurring weight are. So in the top left corner, we put a 1.5 sample weight on new purchases and 1.0 on recurring, so they are pretty close. Like we say, not literally translated, but a new purchase is 50% more valuable than a recurring purchase, for example. And then like you end up in the top left corner, seeing like a huge uplift over the baseline for recurring MRR, but even like a small regression of 5% to 10% for new MRR. However, if we push it a lot with the sample weights, then what we can see is that the whole thing moves far more to the right, meaning like we trade off performance on recurring orders for an uplift on new orders. And then like within each box, you can further play with the negative sampling ratio. And there, something that we found is the higher the negative sampling ratio is, the more negatives you sample, the more you are putting it into predicting recurring more correctly versus new correctly, which actually made sense because there is this observation of so-called overconfidence that also some researchers found and that they used for developing some additional models on top of these transformer-based architectures. that there's kind of a debate whether causal attention or bidirectional attention are better. So there's ZUSREC and PERFOREC. And they actually investigated this effect, which makes sense, like, for every pair that you sample, so for every negative, you also sample the same positive again. So the model keeps seeing the same positive, positive over and over and over again, which, like, in the end, leads to some kind of overconfidence on the positive value or on the positive item, Which is something that you want to limit because then in the end also with the popularity bias in the data You end up being very good at recurring and showing nothing new to the users, which would not be very useful In terms of the production system I added two slides there as I said, unfortunately My colleague who would actually be very mad at me at this time for my poor time management again nevertheless we are also sharing a bit like how we actually put this whole thing into production and how the production model looks like so take a look at the slides and if you like want to know something there I'll like ask me after the talk however I need to celebrate with you and also skipping this data leakage what we found with this model is that we have really achieved some really really great gains on the new MRR so this is finally like showing models for several countries like sort of by some top countries that we had and we were really like seeing great results and the cool thing was that the transformer alone its performance was almost on par with the ncf and sbr performance so only the transformer that had access to far less contextual features about venues which ncf and especially sbr had was already able to substitute this was already like a very good milestone in this overall project because we saw only with the transformer we are already on par with two in-production models. So if we now add this additional component that could be a two-tower model or in our case a cat-boost ranker, we are definitely way above. And this is what you see here. So like this comparison to the baseline or our original ranking, you see the blue bars, these are the transformer ones for new. And then you see all those purple ones which are actually the final RankNet, which is the composed effectiveness of the overall model. So to no surprise, we rolled that model out. And on August 20th, it was time to roll it out, and it was up and running. And we had some global conversion rate uplift. We were able to boost the conversion rate for daily version trials, so for new venue purchase also quite substantially. And And what was interesting, though, is this was like a bit of a, not really a bad thing, but a downer of this. We saw that we actually decreased the ads revenue. And to no surprise, nowadays, the ads team is adopting the same model. Why? Because the actual organic recommendations were becoming more relevant as compared to the ads. So the ads model actually had to catch up on ours, which is then not surprising that there was a slight regression in the ads revenue. because like the recommendation model was just comparatively better so yeah the decision was to launch it in uvr uh worldwide and i'm very happy uh like to to have been part of this project and contributed to because like seeing this model rolled out in more than 30 countries and being there is quite a cool thing and like more to come so more discovery for the models doing cross-domain uvr what we are currently heavily working on but also adding category awareness, so there's never really an end. And who wants to know more about this or actually work on it, as I said, we are hiring. So go to walt.com jobs. And as I forgot to say by now, I also need to self-promote myself always. I'm the host and founder of the Rexperts podcast, so the only worldwide podcast on recommender systems where I frequently interview experts from academia and industry. So if you're interested in learning more or learning more about Transformers Rexes, then tune in. And with that, thank you.

Speaker 1 [43:48]

If you have questions, please write on the talks back in the day. We go with the first one. Do you also consider the item data for value ranking? User buying all kinds of dishes from two restaurants versus user always buying pizza but from 10 different restaurants?

Speaker 2 [44:08]

So I have thought about that specific feature, so how the variety of item purchases from different venues is, but yes, we are. So for example, the user or venues have a selection, especially like our grocery venues, and we take into account this selection data, these selection features, because we learn that users are interested in broader, higher selection. so yeah not in that way good idea we'll take that with me but we do

Speaker 1 [44:41]

The second one is similar for me. How do you pass the predictor's result to the ranker? Any processing of the probability?

Speaker 2 [44:51]

it's actually something that I've been thinking about the most recent days because I would say like we do it very naively so as I have shown the transformer is finally trained on minimizing the cross entropy loss so it basically solves a classification problem which is like a multi-class classification problem about like thousands of venues and yeah you could basically use the softmax activation of it across the venues but then you need to boil down the venues to those that are only available, so what we really do is we only use the actual logit before the activation and this logit is actually provided to the ranker as a feature. I think there is definitely room for improvement, especially when it comes to calibration, which is something that is commonly done in the app problem because there you really want to predict the probability of conversion and not just supply a logit, but here, the problem that we are solving is not having exact predictions but we have to impose a relevancy max ordering of venues so it does not have this criticality of what we need to supply but I think normalizing it properly and supplying it to the rank

Speaker 1 [46:06]

How do you perform the validation of your model? Is the validation performed live at inference, timed with all the features about the user's feedback? How do you determine the weight for each parameter according to the user or the area? How do you weight which nature of the model?

Speaker 2 [46:23]

person like if that person is around like to come to me afterwards but maybe validation i can i can show so um like we use validation data that we use early stopping and we use testing data for the final rollout decision that is not used for any kind of training and we usually like uh test the model on the last day as we are retraining these models for our countries currently on a daily basis which might also be another like possible efficiency game so all these models are freshly rolled out every day so that we evaluate and or test them on yesterday but taking like a full window at least for the rancor of four full weeks and then like we have a benchmark criterion that their rancor needs to strike in terms of the MRRs and if it does so then it basically passes on to

Speaker 1 [47:20]

not when falling into deep learning instead of having deep learning plus getting boosted, those are different stacks.

Speaker 2 [47:26]

This was actually the preliminary idea in the very beginning, where we were splitting the work a bit and developing the transformer. I was actually starting with a two-tower model for this, and I think, looking backwards, it was more a matter of adapting the objective criterion for the two-tower model correctly. But we had a bit of problems that it was not working right away, and I think it would still be superior. So going like full deep learning using a two-tar model like for example the architecture that I showed with the Pinterest reference is definitely Or I believe it's superior to our current solution. Nevertheless at that point Sometimes it's not about taking as much time as possible to make your solution perfect But sometimes you also need to like satisfy your stakeholders who are waiting for something, especially if it's a bit more like exploratory So we were a bit pressed for time and I said wait let me just check out if simple cat-booth breaker does the job so I did this within one or two days and I saw like oh shit it's currently working better than my two tower model which I tried to somewhat couple with the transformer in the end I think the two tower model's possible effectiveness would have been better but when it comes to this efficiency at that point the cat-booth breaker was working right off the box very successfully so i said like let's go with it it's simple it's something that we already use that we have experience with so why not so sometimes simplicity strikes like the more advanced models

Speaker 1 [49:02]

How do you create the embeddings for the tokens that the Purchase Predictor transformers

Speaker 2 [49:07]

This is created when we like initialize the transformer model, so it carries an embedding layer. This embedding layer has values or has embeddings for all values that we observed in the training data. This is also why the transformer has a little bit of a problem with cold start values, but this is also a reason why we train it so frequently. So there might be better solutions, for example, the dense black paper. And then like we create embeddings in the embedding layer of the overall transformer model so not only like of the transformer layers so that at the very beginning we get a sequence we embed those sequences and they will be trained alongside the transformer models also with regards to minimizing the cross