Better search relevance using Learning to Rank at mobile.de

Speaker 1 [00:05]

Hi, everyone. Good morning. Really excited to be here. Today in this talk, I will present the progress we have made on improving our search relevance using Python. Okay, so to start with, a quick introduction about myself. My name is Manish Saraswath. I'm currently working as a senior data scientist, machine learning engineer, mostly responsible for not just building models, but just making sure that how do we ship this model to the end consumer. On this eve of PyCon, I thought of also mentioning some of the code contributions or the open source contributions that I've been able to do in the last few years. I have contributed code to some of the open source projects like Gensim, Data.Table, if you're into data science and machine learning. I've also contributed on Stack Overflow, So this is something which I'm quite proud of. I have currently over 20,000 reputations. And I'm also author and maintainer of a SuperML R package. The first language which I learned was R. And then I moved to Python. And this picture I put there, besides Python, I also love football. So this is the first ever match which I went to watch in Munchengladbach. It was between Germany and Italy, and Germany won. So money well spent from my side. And as a data scientist at Mobile, so we have several use cases where we are using Python. And most of the use cases are enabled using Python, like I said. So something like if we are trying to improve our product discovery using recommender systems, we are also trying to help our users find vehicles faster using learning to rank. Also, we are trying to make Mobile a trustworthy place using fraud prevention models. We are also trying to help users to continue their search journey when they don't have enough results to see using search expansion models. And we are also trying to understand user intent, what they want, if they want to buy a vehicle or if they want to lease a vehicle and just show them vehicles according to their needs. And besides that, we are also helping dealers on the other side to let them know how much price should they put for the vehicles which they are trying to sell. So these are some of the problems that we are solving. And this is completely using Python. So as you can see, Python plays a quite important role and is well integrated in our product. okay the outline of this talk is going to be quite straightforward so i will first start with introducing mobile in case if you don't know and then i will talk about the search engine we have how we how it works and after that we will talk about the problem that we have solved then the steps of how we solved it then after that how we measured it did we even solve it or we made it So using some performance metrics. And finally I will share some of the learnings and key takeaways. These are probably some of the lessons which I've learned the hard way and in case if somebody would have taught me, I would have done things much faster. So just a quick show of hands. How many of you don't know about Mobileye? Okay, okay, okay. So as I expected, it's quite popular in Germany. But for the ones who don't know, I fetched some numbers. So Mobileye started in 1995. It's almost 30 years old now. We have at any point in time around 1.5 million listings. At every day we have more than 50 million searches happening. We have more than 40,000 dealers plus private sellers. And we have around 40 million monthly active users. So the whole point of giving these numbers on this slide is just to give you an insight of the volume of data that we are handling in the search team in order to build these models. Besides these numbers, we are also a two-sided marketplace which I've already figured out. So on one side we have buyers, on one side we have sellers. Sellers can sell cars, bikes, trucks, and now from last week onwards they can also say e-bikes as well so if you can see this two-sided marketplace nature of our business it also gives some kind of challenges to the data science side because we have two objectives to follow the sellers want more visibility on the platform and buyers wants to see the most relevant vehicles that they can find so how we do this match how we do this matchmaking happen between the buyer and seller. This is the responsibility of the search team. This is how the homepage of mobile.de looks like. So on the homepage, so there is this small red box. This red box is called the quick search. So it means this is the gateway to enter all the vehicles we have on the platform. So in this snapshot, you can see the user looks for BMW 120 and for this particular search query we have around 2,000 vehicles at that point in time in that user location where the user made the search. And also from the internal analytics we found that out of 100 clicks which happens 84% of them goes into the quick search. So it means that as a search team it's a quite important responsibility on us to make sure that users get a good experience. So what happens after the user clicks on this angebot, which means offers, the next page which comes is called as SRP, which means search results page. This is where the user gets to see the best 20 vehicles for the search query which they have made. And this is where our role comes into picture, our models, all the algorithms that we are using comes into picture to make sure that how do we find these best 20 taking into consideration what the user wants, what search query they have made, how they have interacted in the past and all the data that we have been storing for a user. So this was just to introduce you to the platform because I will continue using these terms in the next few slides. So I hope you can connect these pages with the metrics which I'm going to talk about in the following slides. So how does it work? So I have shown you the home page, I've shown you the SRP page. So on the backend, let me try to explain it in two simple stages of how we are trying to make this matchmaking work. So search for us works in two stages. I call it as a multi-stage search process where stage one is called as a candidate retrieval step and stage two is called as a re-ranking stage. So candidate retrieval stage is basically from all the tens of thousands of vehicles, The job is to find the top 1,000 vehicles which match the search query. And stage two is from those 1,000 vehicles, how do we find the top 20 to show on the page? The first stage is highly powered by Java and Elasticsearch. Okay, I should stand near the mic. Cool. Yeah, so the first stage is powered using Java and Elasticsearch. And second stage is also powered using Java. but right now we are using something called as a learning to rank plugin. So this is a Java-based plugin which gets integrated into the Elasticsearch instance and there you can run your learning to rank models. In order to provide the features, so if I'm talking about a model, the model needs to have some features in order to make some prediction value. So these features need to be transformed using something called as a painless script. This is very specific to this plugin, and data scientists usually don't have this skill. They need to learn it, they need to understand the context, so it provides some kind of overhead on that point. So this worked for us for a while, and then slowly we found that the stage two became a bottleneck for us. So the team started complaining both on the site engineering side, who were maintaining the Elasticsearch instances, and also on the data science side, who were trying to improve the relevance, but they were not able to do it because of this bottleneck. So what exactly was this bottleneck? In the next slide, I've tried to segment the problems into two different categories. So one is from the site engineering perspective, and second is from the data science perspective. On the site engineering side, we got lots of complaints that the LTR plugin offers some kind of incompatibility issues because of which they are not able to upgrade the elastic search versions, and nobody was also interested in taking care of this plugin. So it just worked. And how long it will work, so nobody knows. And as a result, since search is the core of our, the core gateway where users interact, so the overall system maintenance was also getting affected. On the data science side, the biggest problem, in fact, one of the biggest problems was writing feature pipeline in painless script, although they call it painless, but it was not painless, it was quite difficult. So it took some time. Every time there is a new data scientist joining the team, we had to get them on board. It takes weeks. And also because of this, the model was deployed in Java, but the model was built in Python, There was this difference in languages, so because of this, the model deployment had to be done manually. There was, I think, one or two times where we wanted to deploy a new model, but somehow by mistake the old model got pushed. So you can imagine if the model deployment happens manually, it is prone to human errors, and then now we know whom to blame, humans. And the last but not the least, so this plugin was also not supportive in terms of supporting the new advanced ML models that we wanted to build. So the data scientists in our team, they were able to build models offline. They can do all the offline evaluations of the best models as possible, but we cannot deploy them. So it means overall testing a new ranking algorithm in the team, it looks like it's a work of several weeks. So we had a clear problem in front of us. After we had a decision in the team, we had discussions with the site engineering team. So the next was, so until now we have defined two clear problems that we are talking about. So just in case if there is some confusion, so we are trying to solve two different problems from two different perspectives. So first is to fix the bottleneck. So this bottleneck is caused by this Java application because of which we are not able to deploy the models. Second is to enable the data scientists in our team so that they can deploy and they can test the new models which they want to. So there are two sides that I will talk about in the following slides as well. And also I will go into the implementation details of each of these parts. So let's start with the first part when we started redesigning the stage two. I still remember. So this step, actually, this step, the first two weeks in this step were only spent in writing documentation. I think we started writing the implementation part. I mean, the first line of code after several weeks, because especially in case of machine learning, if you're trying to build an API to serve the models, The first step is to get the thought process right on what you are trying to implement. So in our case, we had these discussions with the principal site engineers in our company, and then they posed us several questions. But here are the three main questions which I would like to also share with you in this slide. So the first is the choice of language. So in which language would you like to build this API or ranking service or whatever you call? And some of the questions that we should ask in this step is, is the language matured enough? So whichever language you choose, you might want to choose a language which is memory efficient, which is fast to run, which is fast to deploy, which is typing enabled or so on. But we also need to think about how long would the hiring take for this particular language? Do we even have enough candidates in the market who knows this language? and second is we since we are since we are thinking long term we also need to think in terms of the current state of the team and the future state of the team right now who is going to build it right now we are excited we want to solve the problem we can build the service but tomorrow who is going to take care of it so these kind of so these kind of questions needs to be asked at the initial stages and also like is the learning curve steep so for example if you pick up a language which is quite fast but if it is difficult to learn and if you have a big team already if you want to bring them on board then it will take some time they will have to spend lots of time into training into learning something new and they won't be able to contribute to the business metrics or some business goals and last but not the least uh actually this is where i personally discuss most time is to have uh to have a clear requirements of the functional and the non-functional requirements. So the API which you're implementing, what exactly is it going to do and what exactly is it not going to do? Having these kind of requirements at the initial stages of, at the initial phase of when you're implementing an API, it is extremely fruitful. And as it goes for the choice of language, surprise, surprise, we chose Python, because reason being, so because reason being for data, so data scientists were already developing And their models in Python, they were doing all the data analytics and model building, model transformation, even data pipelines as well, and everything was in Python. And also in terms of now API, so this is where some of the members in the team, they had some skills and they also had some experience that Python scales if implemented using the right set of tools. And also Python is a mature language. it is not so difficult to find a data scientist in the market who does not know Python. It is quite popular these days. So it kind of gave us some confidence that if we use Python, we kind of allow our data scientists in the company to own the stage 2 ranking end-to-end. So as a result, they kind of reduce the support from the backend engineers that we had before. and now we can change things we can build a new models ourselves we can run experiments we can eventually iterate much faster so i would like to take some time and spend and share some more details on the functional and non-functional requirements part so functional requirements talks about the behavior of the system how the system is supposed to behave so in our case since it was the machine learning api so our requirements were quite limited but it was good to to mention them explicitly so we had requirements as follows that the service must return a ranked list of 20 items it must support business logic in case if you want to add some additional boost so the api should support that it should support pagination if the users wants to get a list for page 2 or page 10 it should be supportive and last but not the least it must not respond with the stale dealer information so here in case here the last point is extremely important for our case because here defining this requirement also helps us think okay we need some kind of data pipeline which gives us some the latest events and something to be done on on that side so just having this line also also helped us to define how much complexity is involved and on the non-functional side so non-functional requirements talks about the qualities of the system right so in our case the latencies the latency were quite strong i mean they were quite low the requirements were so it must respond within 30 milliseconds and this time and this time as well it gave us some idea okay so we want the service to be fast what options do we have left now and in terms of we also wanted the service to be highly available and we and we are okay with the eventual consistency so eventual consistency means if a if a seller has updated the price of their car from 15 000 to 12 to 12 000 so we are fine if this 12 000 value shows up after five minutes we don't want it to show up immediately and finally the last point is the system was the system should be stateless so there so there should not be any memory attached to the service so that if the load increases, we can scale it horizontally and vertically if we have to. And finally, we also defined some business metrics early in the process. Okay, after this service works, so after everything works, how do we know if we succeeded or not? So in that case, on the business side, some of the important metrics are the user conversion metrics. We would like that the users have sent more conversion, that the buyers and sellers have interacted more if this uh if they are able to interact more then we can attribute that this uh project which we have done this project has added some value and on the dealer impression side like i said we are a two-sided marketplace where sellers want more availability on the the platform so it means if they have if their visibility has also increased so it means both side of the marketplace they are happy so until now we have uh so until now we have talked about the problem that we are solving and the requirements and we have also defined the requirements. Now let's talk about the implementation. So how we implemented this ranking service. So I will talk about the main points. So one thing is we knew that the service has had to be quite fast. So that immediately gave us an idea that we need to use some cached approach because since this is the first time we are solving something, let's go with a safe approach. We know that it works. Let's not start with some complicated model which takes long time to respond. So cache approach is basically we use a bad job where we update the predictions every X hours. So in this case, with this in mind, so on the service side, So when we started implementing this API, we went with the RESTful service. Simple, it works. And in Python, we used the FastAPI web framework because in the past I've used Django, I've used AIO HTTP as well. But now with FastAPI, everything is quite simple. Their documentation is quite extensive. You can spin up a web server in like five minutes. And also, since we know that we are going with a cache approach, it means we need to have some, our service will need to have some I.O. operations. So in case of I.O. operations, it helps if they are async in nature, and FastAPI gives this inbuilt support for using async. And it also allows easy middleware integration in case if you want to send logs to Graphite or any service which you're using, you can use that. And finally, we deployed this fast API behind a G-Unicorn server using UVCon workers to make sure that your service is parallel and the requests can be handled concurrently. On the database side, we did some benchmarking. I had past experience of using Redis, but this time I started comparing it with Mongo. so we preferred Mongo over Redis because first of all it has a native support for pagination so this was one of the functional requirements for our site and it checks right there and second is so MongoDB has quite fast aggregation pipelines so for example if you have some kind of prediction score and on top of it you want to add some logic if you want some if else statement if the prediction value is smaller than something or bigger It is quite easy to do it in Mongo. And then finally, also it has support for vector embeddings. So for example, if you want to use some embedding-based features, so in MongoDB, you can just simply add a plugin and make it work. So after we have implemented this, now let's see how it got integrated in our service architecture framework. I will try to explain it in a quite simple way. So this phone, so this phone is like the client. The user who is coming to MobileAid, they make a search query. The search goes to the backend service. The backend service then goes to Elasticsearch and gets the top 1,000 results. And then it sends it to this newly built ranking service in Python, which we have worked on. and this ranking service talks to the Mongo, gets features, and then sends back the top 20 results. The bottom part is where the data pipeline that we also had to implement. So when a user makes a, so when a dealer makes some changes, or if private seller makes some change in the vehicles they are selling, they have changed the title, they have changed the price or something, so we get those, we get those events using a Kafka queue, And then finally we had to implement this consumer. So this consumer takes only the features which are useful for the model, and then it takes those features, it does some transformation, and then it updates them into the BigQuery. And then in the end we have a simple Airflow job. This Airflow job just reads from BigQuery and then dumps the latest features into Mongo. Okay, so we have integrated the service. So next, I would also like to show you how the workflow looks like for us. So it's quite straightforward, actually. We build the code using VS Code in Python, then using Jenkins and GitHub Actions. We package the code, we do all the testing, and then we package the code using a Docker container, and then we push it to a registry. And then from the registry, the image gets pulled into a Kubernetes cluster, and the service gets launched. So the benefit of using this workflow for us was, we can test at least this, we can also test the Docker image to Kubernetes integration locally if everything works. And this, it can also, it can be done using a library called as Kind. So Kind you can easily install, Kind means Kubernetes in Docker. You can launch a local Kubernetes cluster on your laptop and you can test this integration part. Similarly, since our service also had this database integration using Mongo, it can also be tested locally easily. You can use a Docker compose, you can spin up a simple MongoDB image offline, and you can test this. Finally, to check if the service scales or not, for this, Python has an amazing package called Locust, which you can also deploy offline. You can also deploy it on the cloud using Helm charts. there are already charts which have everything packaged. You just have to run it and it works. Excuse me. Yeah. So until now, we have mostly talked about the, the service part, but there is one more problem that we are trying to solve. So that is for the data scientists. So how are they able to, uh, how can they, so how can they deploy a more advanced models than what they have? so i try to so actually there is so much to share so in this slide i will try to explain each of these points in a brief way so we started with a so we already had a pairwise ranking model and this time we we make some tweaks in that model so basically in the past we we had already tested pointwise ranking this time we have pairwise ranking but we all we also make sure that when you so whenever you are training a pairwise ranking model make sure that the pairs which is made so these pairs should be sampled equally so for example let's say let's say in our case as well we use a multi-relevance model where you can so where you can think of you give conversion a higher a higher score than i don't know clicks or impression so in this case you see the contact gets a value of three it means you are waiting contacts more important than park so parking is parking is like favorite and then you are waiting parking more you are waiting parking higher than clicks so in this kind of case make sure that the that the training data set which you have for this pairwise ranking it contains sample equally from each of these class because if you are training a pairwise ranking model and if you see you have lots of clicks and you have very few contacts so it means the model will mostly learn patterns which drive the clicks. So this is something that we also had to learn the hard way because when you check the feature importance plot you will see the features who are ranking at the top they are only those features who are driving the clicks, they are not driving the contacts. So this is something that you need to be careful of. Second is in our case we wanted to optimize NDCG but since NDCG is a rank based metric we cannot optimize NDCG directly because all the machine learning models that we built, so machine learning models give out a score and NDCG is built based on ranks. So you need to have some kind of extra work, which converts the score into rank. So in order to fix this problem, there is something called the lambda-mart algorithm, which is an approximation of NDCG. And this is something that we did not implement ourselves. This is already there in implementations like XGBoost, like GBM CatBoost. But in our case, we used XGBoost. We did offline validation using NDCG at K. And then we also handled the position bias. So position bias in ranking is quite evident. So position bias is basically the clicks which are happening at the top. They are not because of your showing something relevant, but it is because it is at the top spot, right? So there are several techniques to handle position bias. So one of the approaches basically you can give, so you can pass the position of that item as a feature in the model. And then when you test, you can simply pass that value as 1, because you want to predict what will happen if this particular item will be shown at the first position. But in our case, so the recent ExyBoost version, which is 2.0, it also handles position bias internally. You can also try that. And so that is something which we used. And after that, we also made some changes by introducing something called monotonic constraints. So monotonic constraints, again, you don't have to do it manually. This is something which is already implemented. So monotonic constraints are if you also want to add some kind of business logic into your model. So business logic is something like, so for example, if I'm building a ranking model, I know I would like to rank the items higher where the trust score is high or where the fraud score is low. And this is something which I don't want the model to figure out. this is something which I want the model to learn because I am telling it to learn that and this behavior can be enforced along with all the features you have you can enforce this behavior using monotonic constraints and finally since we have this two-sided marketplace so we also try to solve this problem using a multi-objective problem actually so multi-objective explaining this is it in itself a presentation but right now i will just try to explain i will just give away the objective function that we are using to implement this multi-objective ranking so if you are familiar so as compared to gbm as compared to the traditional gbm implementations which we have this xg boost light gbm and cat boost model the key difference between that the traditional gradient boosting and these boosting is in traditional boosting the objective function is only based on the gradient which is the first order derivative but in these new implementation it also includes the second order derivative in the objective function. So the fancy name for the second order and first order is the first order is called as lambda and the second order is called as a Hessian. So in this equation so in this equation this lambda one and Hessian one these are like the first order and second order derivative of the first objective, where users wants to see the most relevant item. And lambda2 and hacian2 is the first and second order derivative of the second objective, where dealers want to be shown more. So how do we balance between these two objectives is using a value called alpha. This is something which is set by the user. So for example, in this case, if I said alpha is one, it means my objective function is completely waiting the buyer interest. But if my alpha is one, so it means this lambda two and hessian two, these value will be zero. It means the objective function is completely driven by the buyer's interest. And if your alpha is zero, it means the objective function is completely driven by the seller experience. And this is something which is quite easy to implement in light GBM and XGBoost. So this is something which I've tried myself. You can create custom objectives, just make sure that the output of the function is just these lambdas and Hessian values, and it works out of the box. So finally, after we made these changes, we implemented the service, we made changes in the model we build the data pipelines we put all of it together so finally let's see how does the performance matrix look like so like i mentioned that for us this effort needed to have a significant impact on the business business matrix so that it is quite important so in order to measure that impact so we did it using an ab test so we ran the ab test for three weeks to make sure that we have enough sample size. In the A-B test, we had two variants. One is control, one is variant one. In control, we had the existing light GBM model, which was served using a Java application. And in the variant one, we had the new Exibus model served using this Python application, which we built. And after we ran the test, we got quite positive results, especially on the unique conversion matrix because we want every new iteration of a model or a service to impact more and more users. So that's why we also look at the unique user conversion. How many more users did we enable to make a contact with the seller? And to support that, we also look at the clicks. The clicks improved by 2% and the overall likes or the parkings also improved. So this is on the machine learning side and second is the service which we made and the if you if you remember we had this requirements on the latency side the service should be fast enough so for that this is a screenshot from grafana apologies if it's too small so but the point is by by by using this cache approach so cache approach was basically this lookup approach where we are we were using a bad job to update the predictions so for us the this the service time was six milliseconds it was way it was way under what was expected so this is so this gives us the confidence that this is a good way to start maybe in the next next iteration we can use some model which predicts in the live in the live environment but having this pre-computed predictions is also works quite fast and in terms of successful requests our successful rate was quite high 99.3 percent of the requests they were successful they were not throwing any errors and and also the service was able to scale quite well so it could handle 1500 requests per minute of the peak load okay so actually so let's talk about the learnings and key takeaways this is probably the most favorite part of this slide so i have i have i have segmented the learnings in two different segments one is on the serving part one is on the model building part so on the serving part i so i would say if you're building a machine learning api start with that rfc document so rfc is request for comments you can call it a design document or something else but just but just having a document where you just where you just write down what the service is going to do and how you're going to implement implemented, it helps lots of problems in the few weeks down the line when you're implementing the project. Second is also try to discuss the integration challenges which you might have. So for example, if your service wants to talk to something else and if your service wants to talk to service B and service B is not open to talking, let's say, because it's in another cluster or there is something else. So having this kind of conversation helps a lot. And fast API framework works just fine. It's quite easy. I love their documentation, the examples which they have given. It works. And then MongoDB works better when your model predictions are more than just a key value pair. So, for example, if you are building a rank, if you're building a recommendations service API where for each product you have a score, in that case, Redis works just fine. But if you want to add some kind of logic on top of the model predictions, or if you want to add more features tomorrow, if you want some kind of complexity in terms of features and in terms of the final score which you want to calculate, MongoDB works very good. And having a solid unit test is quite helpful also, especially if you're building a service which has to do with data, data is messy, you might find null values, empty values, It can fail. It will always fail, in fact. So we also spend some time to try to make sure that at least on the service side, each function is covered under a unit test. And finally, try to keep the Docker size small. It helps because if something fails in production and if you want to push a quick fix and if your Docker image size is small, it starts fast. So for this, what we did was, instead of using conda or mini conda approach we are so we are using base python images and then you install the dependencies which you want to make your service works it works quite nice and all the and on the model building site and i wish somebody had told these things before to me so when you are uh so when you are running an ab test and if you are and and if you have to retrain your model when your ab test is live make sure make sure the the data does not include make sure the data which which you are using to retrain your model is does not include data from control or and any other variants because if that happens you you might see the uplift in the matrix which you which you have seen in the first week of the model and after you have retrained your model that uplift is gone because the model starts to learn patterns from the other variants as well second is think about feature availability make sure the features which you are using in the model so far so for example if you have if you have 100 rows in your data some some particular feature is is available for how many rows because if it is only two rows then the feature availability is only two percent then also think about the sorry so feature availability in production is more about if there is a feature which you can use in the offline training and if that feature is not available in the production so that feature might be completely with that might feature be very important if you use the feature importance plot but that feature if it is not there in production it means you need to implement it yourself second yes and third point is the feature coverage part where you also so you also need to see if that feature is also available for the whole subset of data or not so for example if you are deploying a model which is let's say for markets in germany and france there is some feature which is only available for german markets and those feature will be empty for france it so this at this point you need to ask a question do i need one model for the markets or do i need a separate model for the markets and finally gradient model so gradient boosting models are quite strong baselines they predict quite fast so i think we have moved from the days when you start with a logistic regression models because it's easier to it is easier to deploy but I think gradient boosting works fine okay I was rushing the last few slides because I only have one minute left that's all okay

Speaker 2 [39:53]

Thank you, Manish, for your talk. And we have some questions I would like to remind. We have a slide. You can find the link in our chats in Discord. And you can upvote or ask a question for the speaker. So I will go through the questions. And the first one is the retrieval step could be your next bottleneck. Do data scientists plan to take over that part 2 in the future?

Speaker 1 [40:26]

Sorry, can you repeat the retrieval is the first?

Speaker 2 [40:28]

The retrieval step, so data retrieval step, how you currently retrieve the data and do you agree the retrieval step might be a problem in the future?

Speaker 1 [40:40]

So just to clarify, this retrieval step points to the stage one, I think, right? So in this case, yeah, so this stage one is... So in stage one, I would say there is very less data science involved because we need to find from tens of thousands, just a few thousands, and this is something which is highly scalable in Elasticsearch. So currently, we are at least not trying to fix something which is not broken. in future if we see that there is some optimizations we can do then definitely we might add some i don't know some demand based features to find the top x but right now that logic stays with the back-end

Speaker 2 [41:23]

Okay, thank you. The next question is, how is similarity being defined when searching relevant cars?

Speaker 1 [41:32]

Yeah, that's a good question. So actually, in case of search ranking, so this concept of similarity goes hand-to-hand with the search context. So what the user wants to find, we show the user exactly that. So for example, so this means that the similarity is defined in terms of search query parameters. If the user wants to find a BMW within 5,000, so we make sure that everything matches in that criteria. So this similarity is basically, in case of elastic search, this is a similarity score. And this similarity score is a combination of the different boosts that we have given to the elastic search query. So for example, it goes something like this. If make matches exactly the same, give a high score. If the model does not match, give it a lower score. If the price falls in this range, then give it a high score, so on and so forth.

Speaker 2 [42:29]

Okay, good. The next question is, can you elaborate a bit more why you choose MongoDB over Redis?

Speaker 1 [42:38]

So, actually, Redis works quite well in case of if you have a key value usage, if you want some value. But in our case, we also wanted to add some kind of business logic on top of this model predictions which we have. And in Mongo, you can easily write this aggregation queries where if your model score is 0.5, and on top of that, if you want to add some business logic, this is something which is quite easy to be, This is quite easily implemented in Mongo, but in radius. I was not able to find that

Speaker 2 [43:12]

Okay, thank you. The next question is, can you expand on what exactly you are caching for fast inference, or how you achieve such fast results?

Speaker 1 [43:23]

Yeah, thanks. That's again a good question. So in our case, what we are caching is basically two things. We are caching the feature set for each vehicle. We are caching the feature set and we are also caching the score on which they're going to be ranked.

Speaker 2 [43:23]

Yeah.

Speaker 1 [43:39]

And the score and the feature set, it gets updated every hour.

Speaker 2 [43:45]

Okay, good. And the last question here is for you people who are interested in how do your pipeline have an A, B testing ability? And if yes, how it's implemented?

Speaker 1 [44:01]

Okay. So do you mean the data pipelines or?

Speaker 2 [44:06]

No, a be testing for your models in the production. So yes here

Speaker 1 [44:10]

Yes, yes. So for this, what we do is, so in case of A-B testing, so we have different, so first of all, we have different versioning of the models that we build. So, and each version is mapped to what variant or what test it is being used in. and this is some kind of mapping that we are currently maintaining manually just to make sure that a particular version that has been built and that version is not being overwritten by some pipeline because it is being used in an A-B test. So we do have it, but currently it's a manual process for us.

Speaker 2 [44:50]

Okay, good. Thank you, Manish, for your presentation. My pleasure.

Speaker 1 [44:53]

Thank you.