Personalizing Carousel Ranking on Wolt's Discovery Page: A Hierarchical Multi-Armed Bandit Approach

Speaker 1 [00:05]

Cool, yeah, thanks to be here. It's great to begin again at PyCon in Berlin and now in a different role. Today we are going to present to you the work that the venue recommendation team at Vault has been working on for the past couple of months and which is now luckily in A-B testing worldwide. And the topic we are going to talk about is how we actually personalize the discovery page, which is Vault's main entry point when you enter our app or our website that you see first, and how we make that more personalized and more context-aware. In the following 40 minutes, we're going to talk about these topics. So first, we want to introduce to all of those who are not yet aware of Vault, what Vault is all about, and actually how we do personalization at Vault already, and how we plan to do it in the future and to do it even better as we are doing it right now. Second point is actually, since this is a more specific problem and not about how to impose just a standard ranking of items like you see in all those many examples like I have applied matrix factorization, collaborative filtering to some movie lens data set, but rather what we want to talk about is how do you learn a personalized homepage? So really the assortment of carousels that you see when you enter, for example, like Netflix, Amazon, or in our case, for our app, Vault. And the third thing is actually about sharing not only other approaches, but our approach. So how we vertically rank our content on Vault's discovery page. And lastly, we are, of course, sharing not only the approach with you, but also the architecture and the implementation that powers that solution under the hood. And with that, I'm just doing something that you should not normally do, but first introduce myself before handing over to Stefan. So I'm Marcel. I've been working as a data scientist for the past seven to eight years. And from the very beginning of my career, I found personalization and recommender systems really as my passion and desire. and I'm now very happy and lucky to work for Vault and do personalization there. I'm also running my own podcast on recommendations and talking to recommender systems experts from Spotify, Netflix, Amazon, also to your previous speaker, to Mirza, so he was also on my podcast. So tune in and check for Spotify on Rackspurts if you want to know more about this. And with that shameless self-promotion, I will hand over to Stefan.

Speaker 2 [02:44]

That's a bit early for the applause. Let's see if we want to do that again later. My name is Stefan. I'm 28 years old, working for Volt as well. I'm working as a machine learning engineer in the same team as Marcel. Working there for like nine months. And also there's some self-promotion here. I'm also active in the Berlin community space. I'm part of the organizers of the MLOps community. So if you're interested about MLOps, ML Engineering, India from Berlin, we have regular meetups. you can find us on meetup.com, Berlin MLOps community. And with that, let's jump into personalization at Vaults. I'm first going to introduce what Vault is all about. Vault's mission is to make cities a better place by empowering local communities. And we do so by making the lives of our customers easier and more convenient, helping our merchants, which are store owners and restaurant owners, to grow their business and offering flexible earning opportunities to our courier partners. And to enable this, Volt is developing software products, financial solutions, and also operating grocery stores. Volt is available in 27 countries and more than 500 cities. On our platform, we have more than 140K merchant partners and more than 230K courier partners. A number we are particularly proud of is the 36 million plus registered users, which is steadily growing, and we have more than 10K people working for Vault. This is about recommendations at Vault. You can see there are different typical applications for personalization at Vault. The first is to rank restaurants. This is basically our bread and butter. Here we are answering the question, at which restaurants are the users most likely to order at, based on their purchase history and also the current context. The second application is ranking stores. This is basically the same, but for retail stores, and retail stores can be anything or any store that is selling some retail products. For example, bakeries, grocery stores, or florists. And the third application is item recommendations. We also recommend items to our users. This is an example for a basket-aware recommendation where we have an item in a basket, and then based on the similarities between items and the user's purchase history, we recommend items that would go well with that. Personalization is typically a very hard problem to master, and if you listened to the last presentation in this room as well, then you already know there are a lot of problems and challenges. A couple of them are listed here. And especially in the food delivery domain, the first one is very interesting because the user can have different preferences or intents when they open the app. So every time they open the app, we don't know if they're looking for restaurants, if they're looking for food to order, if they're looking for groceries to order, or they just want to buy flowers or whatnot. But based on the context, that can help us, but there can also be a challenge for us as well. For example, based on the date, for example, on Valentine's Day, people are probably more likely to order flowers. That could be a good hint for an intent. However, the contextual change also may make a bigger challenge for us. For example, in the morning, a user wants to order something else than they want to order in the evening. Then we have also multiple stakeholders, cold start problems a lot, and diverse biases. For example, the popularity bias is pretty interesting. Imagine a restaurant or a venue that you order a lot from, for example, McDonald's. That will appear a lot in the purchase history, and so it will also appear a lot in our recommendations. There are much more challenges, but I'm not going to talk about all of them right now. one thing to remember the project we are introducing to you today we didn't work on by ourselves we are part of the venue recommendation team and the leadership team also influenced the vision of this product and with that I'm handing over to you

Speaker 1 [07:08]

Yeah, thanks for the first part. And with that, let's pull up our sleeves and introduce you to what our discovery page is actually about. So we talked about that we want to show you a personalized ranking of our restaurants, of our stores, and also show you relevant items on the venue pages. But as I already said, our discovery page that you see to the left here is the main entrance point for all our users. So if you open your app, if you go to world.com, this is the first thing that you're going to see. And it's very simple, basically an assortment, a vertical list of different carousels. So a concept that you might know from other platforms as well. And those carousels are, normally those carousels have a common topic or a common goal. So for example, there is a carousel that features order generals. If your intent is, for example, to just quickly order from the same venue that you have ordered again, then this is the right carousel for you. But sometimes you might be in a different mood or might search for some promotions and discounts and prioritize affordability higher. So in that sense, maybe our promotions row might be the right one for you. And so in that manner, we feature these assortments of restaurants, of stores, of banners, of categories. so different kinds of actual carousels on our discovery page there. And, of course, every user is different, the context is different, so we also want this to reflect the difference in the people that search for something they need, whether they search for groceries, whether they search for some retail items, or whether they search to just not be hungry anymore. how we are doing this so far is basically quite a standard but something that you can do even better so what we do right now is we can attribute purchases to all of these carousels and we also do track the impressions that users generate when they scroll through the discovery page and what we do is basically based on this estimate the performance that every carousel has based on the conversion rates they generate. And then we basically rank those carousels by their city-specific conversion rates. So disregarding the fact that you have different carousels for like Cologne or Berlin, the page might look the same way for every user in a single city or maybe even across cities, which is something that you could do much better. And this was basically the starting point for this project where we said, okay, let's go and improve it because it's not only the main entrance point, it's also a significant lever of our business. So let's, of course, personalize it because we have great evidence across the industry that with personalized solutions you can improve your business. And let's also contextualize it. So we already do this, of course, because we need your location to understand which venues are available in your area, but also let's take time into account and let's make it more dynamic to reflect rather sooner than later changes in preferences and have them properly anticipated in the vertical ranking of those carousels. And even though it's not the focus yet, let's add some controllability in terms of going for multiple objectives and not maybe for relevance only. And when you approach such a project, and it might not be quite new for you, what is the first thing you do? you might start ideating about the problem yourself or you might also check out how have others solved that problem how is that problem actually structured so and maybe also how has industry solved that problem so far because of course this is not a unique pattern so carousels and a vertical assortment of carousels is not something that is unique to world but you see it across many platforms so many other platforms many other smart data science teams have or must have been tasked with the same challenge here and we already had it in the previous talk with the netflix prize that was going from 2006 to 2009 with the best team doing kind of rating prediction not really ranking but from there on netflix has evolved over the couple of years and contributing on a constant basis great content among other players and one of the greatest blog posts that this almost 10 years old, is actually this one. So when Netflix was writing about how they actually learn a personalized homepage. So Netflix has many, many potential rows they can show to the users. So tens of thousands that are mostly automaticially generated. And of course, I'm not going to show a user those 10,000 rows. So personalization should play a role in which rows I pick from that whole corpus of available rows but also in how I rank and display them to the user and this is also again like we do have in recommender systems a multi-stage process and finally also needs to anticipate how users actually interact with your home page especially regarding position biases that you have interacting here or that users are affected by when they interact with a page like for example by selecting what is at the top left corner with a higher likelihood than what appears at the bottom of a page. And this was kind of an introduction and to show what are different categories of how you can approach this. So from going the rather simple way of let's have a rule-based system in which we, for example, have content curators or content editors that manually fix rows at certain positions by their own judgment, not really being based on data. But how can we go further from this and take data into account? So evidence from the user that tells us something about their preferences and also something about in which context might the content be right for the user. And one of the more simpler approaches is the so-called category of role-wise approaches. In those role-wise approaches I take my assortment of carousels that I have like these carousels a b and c here and I do have a scoring function that takes into account a single or multiple objectives like for example the relevance those carousels might have for a given user and then I basically assign scores to those carousels and then finally rank those scores exploit only or adding some way of exploration in that descending order and display them to the user or maybe the list of the top k rows. So quite a simple problem approach and basically treats the rows as the items in a standard ranking problem, but, and this is the more important point, independently from each other, which limits a bit the goals that I can pursue when taking that approach. So this is why there is a bit of a more complex approach, which is the so-called stage-wise approach, because as in the talk before we have heard with the Q&A, is relevance the only thing? No, it's definitely not, because you also want to diversify your page. You want to offer your users the ability to discover new content. Of course, there might be users who are only interested in seeing or interacting with what they have interacted with before, or maybe going very goal-oriented and just reordering something again, but sometimes I want to find out something new. I want to try out a new restaurant. I want to maybe try out a new supermarket or order flowers from somewhere else because I have better flowers or fresher or whatever. And this is maybe when you want to incorporate additional goals, like for example, the diversity and explore something. And in the category of stage-wise approaches, this is better enabled because what stage-wise does, it not only selects a carousel, but it also takes into account what has been selected before. And based on the selection before it is better able to diversify what comes next and then scoring might be different and hence the next rows might be different as well. The third approach is even more complex and it goes into a direction of really trying to score all the different permutations of a page that you can have because all of those carousels or most of them are basically a ranked list of items. So that can grow very, very complex. But there are also papers out there, for example, by Amazon, where they try to score basically the whole combinations of assortments that you can have. And with that goal, enable even better optimization into the different directions that there are. To give you a bit more of an example, this is from a paper by Walmart Labs from 2020 where they showed how they basically rank carousels on their home page for their online grocery store and all of these carousels are basically featuring items so maybe some some fruits vegetables or something different and in this framework they adopted that row wise approach so that they come up with a score for a of a user for a carousel k and the score is basically composed of two components. The first is the actual affinity of a user for a specific carousel which they estimate basically just using the preference or the aggregated preference of the user for the items that appear within the carousel. So think about of the nominator that you can see there as the dot product of your user and item embedding. And now I just basically aggregate those scores, those dot products, apply some position weighting to account for position bias here, and I get something like a user carousel affinity, which is basically the product or the sum of the user item affinities. And since this is not the only thing, so this is more like relevance focused, but there's also discovery, they also take into account to the right side the user's propensity towards discovering content from other categories which means that for every item there's basically categories and I know which categories users have already explored and by this I basically aggregate again position weighted the different categories for the user and then what we do is basically we weight these two components and thereby trade off like for example relevance versus diversity in those carousels so basically a rather more like simple approach even though it doesn't seem simple at first sight to rank those carousels for a user a more complex approach is like i've already talked about the stagewise category and here's an example by ebay where they apply this framework of i do not only want to go for the very next totally disregarding what I have selected before but they use that approach here to personalize basically the carousels which you see at the bottom of the item detail page and I'm not sure whether they are still doing this but at least by that time this was something that they reported to do and that was getting them gains of about seven percent in increase of conversion rates by taking into account what is the item that appears at an item detail page and what do I know about the user. So this is basically the context that you see to the left. And then what they do, you select the very first module. So basically what is appearing just underneath the hero item. And after they have done this, they use the information of what has been selected along with the contextual features to select the next item so in this sense basically the green module that you see below the first selected module and this basically enables them to diversify the carousels and what those carousels feature further down the road and then also enable users to explore new content and not only be able to engage with what is like relevant or similar so and you can go even further and this was kind of the stuff that we first did so we checked okay what is out there but then of course we asked ourselves okay so how can we actually improve our discovery page and this is how discovery vertical content ranking short dvcr has started where we watched out and said okay how can we apply those approaches how can we adapt them to our domain and this is how started. And as I already mentioned, our goals for this project were to introduce personalization, so not go with the city-wide conversion rates, which basically disregards that users are individuals and have different tastes and preferences, so that there also should be a different ranking for every user of the carousels appearing there, but also to make it context aware and to understand based on the data we have on purchases, impressions, attributions, at which times of the day which carousels perform best and especially in light with the users we have. And make it dynamic and this will be definitely something that Stefan is going to tell you more about, how we enable a solution that enables us to change our content and to account for the fact that users might have certain preference for certain rows but what does it actually mean so we are basically focused in this first stage of the project more on the relevance part which doesn't mean that it's going to end there but it's just the first stage of this overall project where we said okay our goal is to increase the conversion rates on vault's discovery page and of course this is something that you can do by introducing more diversity to your content but also by making it easier for users to find what is relevant for them and for relevancy a great signal is the purchases and we already talked about what is implicit feedback what is explicit feedback but a purchase is basically the user's way of strongly showing preference for a product of course it could also go wrong so you might make an order from a restaurant you don't like it you don't buy again but it's basically the way of how our users can tell us with high confidence that they like something instead of just relying on clicks or something else. So this is why we focused on the purchase history here. When talking about context awareness we went for the hour of the day because I mean for purchases from restaurants you're not doing that many across the day so maybe something like the hour of the day or the day of the week might be the right aggregate for it but this is still under consideration whether we do it more fine ingrained or add some additional signals. So first thing, go with the hour of the day. Third thing, and this is what I already mentioned, is actually to make it dynamic. So to reflect user purchases, to reflect also the introduction of new carousels and enable them to get attention by basically having daily updates so that new user data, new carousel performance is being taken into account rather quickly. of course you can go even further, but it's then actually a matter whether this is really worth the effort. And then, last but not least, controllability. So introduce the optimization for multiple goals beyond just relevance, but this was for that first solution out of scope for the moment, but it's something that we seek to address with further iterations of that approach. And what you can see here is basically non-mathematical trial to compose what we have done in that project. As I said, what is our goal? In the end, we want to have a personalized ranking of waltz carousels and show them according to that ranking, which is easier said than done because it's not only up to us to steer that ranking. There are also kind of manual interventions like pinning certain carousels at certain positions nevertheless we as the venue recommendation team are trying to do our best to introduce the reflection of users preferences here where does it stem from so we basically take two kinds of signals we take the city wide purchases that were made or attributed to our carousels on the one hand side as a quite dense and less noisy signal because we have kind of 20, 30, 40 different carousels, sometimes more, sometimes less, depending on the city, and we have lots of purchases that are being made through these carousels or attributed to them. And then we have the user signal, which is, of course, for us, the more, or for the personalization, a more important signal, but however, it's noisier, it's less dense, it's a quite sparse signal, because, of course, think about yourself, how often you purchase food or groceries somewhere, and this is then actually kind of a problem. However, what we can do is we can take the city as a prior. So we can think about that the common set of users that is causing the purchases for carousels in the city as a prior information for what might be relevant for the user and soon take the information that we get from the user through their purchases to adapt that prior. So to collect that evidence, update the prior and get a posterior and take that posterior information to create the rankings that you see on the right. And then you can reflect user preferences by, for example, seeing that shopping deals, which might be the fourth most relevant thing for the city are the most important thing for the user because we have observed the user frequently purchasing from that very carousel so that it should be the one that should appear to the top here. This is more like intuitively speaking so you get the gist of it but of course we needed to model it and I've already been alluding to that kind of Bayesian way of thinking about that problem and there's many great literature where you also see that Bayesian is being applied here. So think about that in a bandit framework. So every carousel is basically the arm of your bandit. And a user pulls and receives a reward if we are having a purchase there. And you could, for example, take an impression as something that is not a success. And this is very similar to what Walmart did. So you might have observed that lambda term in the formula that I've been showing before. or also some approach that you saw at Deezer and we basically tried it out but we were more successful by modeling the final scores that you can see here which is not the very same as a multinomial distribution. So by basically modeling the amount of purchases a user made within a certain time window as a multinomial distribution over K arms so basically K carousels that we had. And we did this in a hierarchical manner by first taking the city information and second updating it with user information. But we did also apply updates on the city and on the user level. And how we did this, I'm going to show you in the next slide. So first thing on the top of our hierarchy is actually the city level. And we went pretty basic and said, okay, let's take a uniform prior. So let's assume that every carousel in the city receives the same number of purchases within a given amount of time, which is our alpha city. And this alpha city is basically the concentration parameter of that prior Dirichlet distribution. And what we take into account, so the data generating mechanism is basically following that multinomial distribution, which we see with the age. So this is basically the evidence that we got within a certain same time window where we said, okay, these are the number of purchases for every carousel that we observed within that city. And we take it to update that parameter so that we get as a posterior distribution for that city these counts, which then could reflect the concentration parameters of that city directly posterior. So this is basically our city performance. So having some uniform prior assumption about how the carousels behave for the city, update this with the purchases I observed, and then take those parameters to reflect the city after having seen the information or the purchases. then as a next step what we do and we are definitely not paying attention to every mathematical correctness here but what we are interested in in the end is not really correctly estimating the amount of the purchases but rather to impose a ranking which enables you to lift some of the assumptions because in the end you want to just have some proportionality and to impose the ranking properly so we scale it down to limit the influence of the city on the user level which then makes us arrive with a new prior but a prior that is no the prior for the user level which is our alpha user here and this is basically the linking point where we have now some from the city informed information that initializes our information on the user level where we basically do the same so we take that prior, we update it with the user information to again collect parameters, the concentration parameters of the user directly posterior and this one or these parameters we then use to impose a ranking. Currently in an exploit only framework with few exploration but then soon based on top of it also adding more and more exploration to increase the diversification of the page. And this approach has shown to be quite successful and I'm just showing a couple of the results that you can see here. So what we are mainly interested in an offline evaluation manner is the mean reciprocal rank of the row from which we made a purchase in the corresponding session. And you can see the different results across different cities here and the big difference and what I want to highlight here is the difference between light blue and dark blue which is kind of showing you the difference between non-personalized and personalized solutions. And it's different across different countries, of course, but it shows you that personalization matters and that taking into account user purchases is adding some benefit here. Before handing over, this is kind of a rough schedule of what are still the many things that we have maybe ignored so far to start easy first, but that we want to approach. So something that is missing now is really exploration, which you see in many of those frameworks and which is a great argument for adopting Bayesian methods because it's quite easy and natural to introduce exploration there. Another thing is, of course, debiasing. So, of course, the vertical ranking is heavily impacted by position bias which is something that you need to take into account. Of course, make it content aware which is currently something that we are totally ignoring. Like, for example, in the Walmart example, you take into account the ranked list of items and the user's preference for them to infuse a preference for the carousel. But so far, we only really look at the aggregate level of the carousel, but not at its individual items, which make it easier to start, but it's definitely not the end. And then, of course, introduce multi-objectives. So go for, like, diversification, novelty, fairness, beyond just pure relevance, which is a great discussion within the Rexxas community. But this is just our plan for the future, and I will now hand over or hand back to Stefan to tell you more about the architecture and the implementation.

Speaker 2 [32:10]

Thank you myself like for a second rounds have to hurry up a bit because those 40 minutes here are shorter than the one We had at home Let's talk about architecture and implementation first start with architecture requirements. This is of course the first things that you think about when you want to build an application and We had a couple of those in mind when we started our thoughts first of all serve in real time We wanted to be able to serve the requests as fast as possible possible. This is especially important for our service being part of the very first services that are called when the app opens. That means adding latency to our application also adds latency to the app's startup time, which obviously is a problem for the recommendation team because we don't want our users to churn before they see the very first recommendation. That made latency a primary concern for our solution. Second requirement, high availability. We want to be able to serve as many requests as possible with our solution. And that had just some implications on the solution. For example, it had to be replicated over different regions, had to be distributed, yada, yada, yada. Update regularly, and that's what Marcel mentioned or meant when he mentioned updating the data on a regular basis or on a daily basis in this case. We wanted to be able to update the data without updating the whole software of our application or redeploying our whole application because obviously those have different life cycles. But we still wanted to be able to use that updated data within the inference. So we had to make it detached but somehow available to the rest of the application. Easy to use and company standard. We didn't want to build a highly polished, super sophisticated application just to show our engineering knowledge, but rather build something that is easy to maintain and that developers within the country can easily adapt to. And supported by our ML platform team means that we want to use as many services that are not the core of our application or that somewhere are in the periphery that are supported by the ML platform so we can focus on our core strength, which is personalization. Taking those into consideration, we came up with an offline architecture and later on I'm also gonna show you the online architecture. The offline architecture here has three major components from the left to the right. The first component is Snowflake, which is our internal data warehouse. Most of the data at Vault is stored and processed inside of Snowflake. And the data we were especially focused on or that we wanted to use was purchase data and purchase attributions to pages of views in our app. Second application is, or second component is Flight. We use Flight for our machine learning pipelines. It's supported by our ML platform. We use it to build DAG, so Directed Acyclic Graphs, of which an example you can see here in this slide. This has three steps. The first step is creating a temporary table, second step is validating the data, and the third step is storing it in an output table. During the whole process, the data stays in Snowflake. This was particularly important to us because Snowflake is really good at data processing and we don't want to send data around for no reason. For this, we're using Snowpark, which allows us to use data frames but keeping the data within Snowflake. And we don't have to use raw SQL, which obviously you can't test. Then, when the data is stored in the output table, it is registered in our feature store system, which we used Feast for, and this makes it available later on for the inference. It is, there's a cron job that is regularly running to materialize the data from the offline store to the online store. Talking about online, this is our online architecture and here we are depicting how an incoming request is processed by our application. First it is obviously you open the app and then some internal services are called and one of those internal services is calling our personalization API using the getSections values gRPC method. The personalization API is an API that we share with another team, the item recommendations team, and this enriches the request and forwards it to our application, DVCR, the DVCR service. DVCR service is deployed using Selden Core, which is the application that we use, or the service that we use, deploy most of our machine learning applications to a Kubernetes cluster. It's also supported by the ML platform team, and that DVCR service then retrieves the additional data from the feature store, uses that to compute section values, and returns it to the sender. Example request and example response are shown here, but because of time I'm not going to talk about them too much, but rather show you a bit of code. This is an implementation of our DVCR scoring server. It is based on ML Server and the V2 protocol. I hope you can see it. We have two major methods here, the one is load and the other one is predict. Load is called during the pod initialization, the pod startup, and what it does, or the most important thing that it does is it initializes the connection to our feature store so we can reuse that connection later and don't always have to update, which helps us just being faster, having lower latency. method is used to handle an incoming request by passing it into a Pydantic model, validating it thus, then getting extra data from the feature store, as I've shown in the online architecture, getting extra data from the feature store, computing section values and returning it to the sender. All right, let's come to the learnings of our implementation. The very first learning is building a microservice over building a monolith. At the beginning, we decided to build our own service, build a microservice, and that implied that we had to build a couple of things from scratch, for example, hosting the application, having an API, exposing the API, adding it to, like, or creating a proper, like, creating proper tests and everything. So we had to build a lot of things from scratch. We could use some templates, internal templates for that, but there were still some building blocks that we had to create ourselves. And we could also have added it to the personalization API that I showed you earlier. That would be the modular monolith. However, now we're really happy that we didn't do it because now we have the responsibility of our service on ourselves. And we can update it, roll it back, and everything that we want to do with it on our own. And we don't have to align with the other team all the time. Second learning is about dependencies to other teams. And, if you go back to the architecture requirements in mind, I said that we wanted to use as many services that are supported by the ML platform, which is great, because we can focus on our core strength, it takes weight off our shoulders, and it helps us not to spend so much time on monitoring and maintaining other services that are not at the core of our application. However, we also learned that it can slow us down if we have to wait for a service to be released or the platform team makes us use that service in a particular way. And the last learning is about data quality and that probably appears on every machine learning related presentation here at PyCon. And I mean, the thing that I can tell you about it that you probably all know is that data can always be very unexpectedly funny in many possible ways and even if like we at the beginning we started mapping out the data trying to see what is available and how we can use it it still surprised us with challenges with changes especially in online data especially with attributions there's a lot that can that can change even if the ui is changed in the app somehow that can change the whole like a lot of the logic that you have in your attributions and whatnot and I think the main learnings here is just don't be surprised if the data doesn't met match your expectations talk to the data owners and validate the data and that was it thank you so much for your attention

Speaker 1 [41:17]

Thank you so much.

Speaker 2 [41:18]

both very, very

Speaker 1 [41:19]

A very, very inspiring session. So, some questions.

Speaker 2 [41:23]

Some questions were already answered.

Speaker 1 [41:25]

So, but let's ask you, how do you split users for A-B testing versus current approach?

Speaker 2 [41:33]

and in which proportions do you use A-B testing?

Speaker 1 [41:39]

Yeah, I can take it. So it's basically that we split them randomly, the users in the different countries we are running the AAB test in, and basically currently in three groups. So we do have our control group, and they are evenly distributed. And the first group is actually testing the non-personalized solution, and the second treatment group, so the third variant, is actually testing the personalized solution to also understand in online fashion the effect that personalization has here and not only the total effect compared to control. Thank you. And if you get a posterior...

Speaker 2 [42:19]

posterior distribution, can you

Speaker 1 [42:21]

Can you simplify, incorporate, exploit?

Speaker 2 [42:21]

simply Corporate Exploitation via Sampling the Ranking.

Speaker 1 [42:24]

sampling the ranking for cargo cells, or did I miss something? Yeah, so good question. That would be actually the goal of our next project, because currently what we do is we simply exploit, which means that we rank by the final concentration parameters of the posterior distribution. But of course what you can do, you can take that posterior to sample from it, and by that actually also introduce then exploration to your system, which is definitely one way. You could also think about others, like for example, introducing some soft Macs, but possible, yeah.

Speaker 2 [43:02]

Okay, I have one more question because...

Speaker 1 [43:04]

because time is running out.

Speaker 2 [43:05]

And what has a stronger right? Context versus user. For example, when the user never...

Speaker 1 [43:11]

never bought something

Speaker 2 [43:12]

both something in the evening.

Speaker 1 [43:18]

I can't tell you right now, but good question.

Speaker 2 [43:22]

Then you get another one. Do you use the same model for all the cities?

Speaker 1 [43:25]

For all the cities. It's the same model, but of course it's differently parametrized. So in that sense, it's not the same model. But somewhat yes, somewhat no. Thank you very much again.

Speaker 2 [43:38]

much again. Thanks a lot, Paul.