When LLMs Are Too Big: Building Cost-Efficient High-Throughput ML Systems for E-Commerce Cataloging

Speaker 1 [00:00]

Hi all, last day of the conference, so enjoy the last talks as much as you can, ask questions, the questions will be answered at the end of the session, the questions can be asked through talks.pycon.de, the session will be 45 minutes long, and today we have when LLMs are two big building cost-efficient high-throughput machine learning systems for e-commerce cataloging and presented by two senior machine learning engineers at Idealo Internet Gimbiha by Tobias Sens and Bastian Wand. Yeah, give a warm applause.

Speaker 2 [00:50]

Hello. Yeah, and thank you. Welcome to our talk. Yeah, in the next half an hour, 40 minutes, we want to present our solution and practical experiences in the area of high-throughput text-based classification where we see that LLMs are too big. Maybe before, let me share some details about our company because I think it's important to see what is our background and what are our challenges. EDL is an internet comparison website. Maybe the most important thing here is that we have our offers from over 50,000 shops. That means we are a full-line retailer, and we have over 5 million products online in over 2,000 categories, and each product has a lot of offers in there. We are active in six countries. That means we have to deal with five languages in our models. and Bastian and I we are working in an offer recognition and product association team it's called cataloging and we have to deal with over 2 billion offers and we have to process them and these come from the 50,000 partnerships and to give you a glance of what kind of data streams we have to handle so our imports processing 150,000 offers per second but of course not all of the offers have changes which are important for us, but we have on peak times to deal with around about 80,000 offers per second and classify them. And in our team, machine learning plays an important role, but it's, and I think you know it, machine learning is only a small part into a bigger system, and this is like the system of our team. So we have a lot of other services. We have Kafka queues. We have MongoDBs. We have controllers, which are calling our classifiers. I'm showing this diagram to explain that we are only part of a complex domain, which is event-driven, and we have also strict requirements in terms of scalability, efficiency, cost efficiency, and of course, the highest priority in our case is reliability of our services. We have such a team structure because in Idealo we are following the principle of the product end-to-end responsibility along with the way of you build it, you run it. I personally really like it because we are not only developing, we are also operating our services. We can do this in our team because we have an interdisciplinary team stack. That means, besides our machine learning engineers, we have also very experienced software developers. They have a Kotlin stack, we have a Python stack. In our teams, we are meeting both worlds, and this is very interesting. So we learned a lot of the software developer world, and also software developers are learning a lot from us. And this makes fun, at the end. I'm telling you this again because I found it's important to keep in mind why we did some design decisions in our systems. This can depend from company to company, to structure to structure. This talk will be focused on category classification. In our case at Idealo, the category classifier is a service that assigns offers we get from the merchant to our own internal category taxonomy. The input we are using is our text fields. This can be title, descriptions, and sometimes you also get kind of category paths, which is not fitting to our category paths. And with that, we try to classify a category within Idealo. We have different challenges. One is that we have to deal with five different languages, so our models are multilingual. The other is that we have to process a huge amount of data in a quite short time, because when a job gives us an offer, the offer should be online as fast as possible. And the third is that the data we are handling with, as well as the category structure, they are evolving over time, so categories can be changing. They can be splitted, they can be appearing new categories, and this is something we have to handle in our system. From a historical perspective, we have relatively successfully deployed a support vector machine. Maybe you know this is quite an old approach, but it's very efficient and it's also very cheap, I would say. The support vector machine we are using, or we have used, is integrated into a fully automated continuous integration, continuous deployment, continuous training pipeline, which is shown here. We have simple steps of data acquisition, where we collect training data, evaluation data, test data. Of course, we have the model training and we have the endpoint, which is automatically deployed. All of these steps have their own code bases. The code needs to be up-to-date for security reasons, so we use Renuate for that. And of course, the most important part here is the category classifier, because this is where the actual business value is done, I would say. And this classifier needs to process on peak times over 4.8 million offers per minute. And so far, support vector machines were very cost-efficient and we use them very, or we like to use them, But we wanted to take the next step to explore transformer-based models. As you know, transformer is actually the state-of-the-art in encoder classification. They achieve high accuracies, they could better use a context in the text. The problem in our setup is, and that holds for most deep learning approaches, that they are so much computationally expensive. And in our case, latency is not that big a problem. we could scale up, but we have a really key constraint on operational costs. Spending five or ten times more on the same service is not an option for us. On the other side, the decision we made to explore deep learning is not only driven by cost and performance, it's also because most of innovation happens in the transformer side. I think this is a kind of missed opportunity, because the traditional approaches have also some very interesting concepts and properties, but it is as it is, and when we change our stack to a transformer stack, then we can more easily improve our approaches, then we can also benefit from the rapid progress in the communities, so this is also an argument where we want to step in. But using transformers needs a different amount of engineering and research, and that's what we want to talk about here. I would like to take a step back. On this slide, what I wanted to show is how we, doing research, how we're doing development and production, this is the normal machine learning lifecycle pipeline, the important parts here is that you have a research phase and a product development phase and fast research is very important especially in the field of ai because there are a lot of claims that are made and the evaluation of these models are mostly done on their data sets and in machine learning performance is very related to data and you don't know if your data corresponds to the the data, the models have to be related, so you have to do it by yourself over and over again. So that's our impression. And the take I would like to give here, or the claim, is also that this is very important to have separate code bases for production and for research, so that you can keep the production code clean and maintainable, and in research you can do wipe coding, you can messy coding, you have fast proof of concepts, and then you do a refactoring and put it into production, so this is how we are working. Now I would like to hand over to Bastian, who will tell you more about the research side.

Speaker 3 [09:37]

Thank you, Tobias, and hello everyone. I'm Bastian. I'm also a senior machine learning engineer at Idealo. I have a bit longer research background than Tobias. I also did my PhD in Germany, but then I moved to Canada to do my postdoc there. And after that, I was a professor for three years in Sweden on machine learning and 3D computer vision. And every time I tell that, people expect not very fancy models from me that I'm showing, very researchy stuff. And I hope you won't be disappointed, I will show some models here, but first I want to talk about something that is really, really important, and that's often more important than your models that everyone is interested in, and that is data. Everything starts with your data. Your machine learning pipeline starts with your data, and my part of the presentation also starts with data now. So that's why I'm going to talk about for, let's say, 10 minutes about how we design our data sets and how our data sets look. So as Tobias said, we have a huge data set. We have around 2 to 3 billion offers in our data set, and that is very uncurated. It's very messy data. If you don't believe me, look at the random offer at eBay and read the text. So then you will believe me that this is not the greatest type of data you can have. And that's why we decided to build a production data set. So a production data set is the sub-sample data set of all our data we have. Second reason that we have a smaller production data set is that we have to train on regular intervals because our database changes. I will talk about that a bit more on the next slide. But all our data is too massive to train regularly on it, so we have a sub-sampled production data set that we crafted for training. We also split into our deployment part and the experimentation part, and for the experimentation part as Tobias introduced it, we also have our experimentation data sets where we can use all our messy code on and where we can evaluate really quickly. So our production data set must have some properties. Of course, it should resemble our real data, because we need to train and test on the data set, and that should more or less reflect the real data, and that's the major part of the data discussion I will give here. Yeah, we made it suitable for training and testing in our pipelines, and the important factor here is that the database changes over time. That doesn't mean our input text, for example, for an offer might change. Yes, that does change, but also our classes may change. So we are in the classification domain, and our classes might change if our content team says, hey, this category has to be split in two, or these two categories need to be merged. So that changes over time, and we need to assemble a new production data set over and over again. The experimentation data set is very different in that regard. That experimentation data set should stay consistent, Because if we have regular updates on our models, on our training regimes, we really want to see how the experiments we do reflect how the performance changes over time. And for that, you have to keep the data constant. We also have three different data sets, one that you can normally train on in a few hours, one that you can train on overnight, and one that you can train on in three to four days. We have shown how we sample these datasets and evaluate a prototype on this dataset that translates very well to our production dataset. So let's look at some real data, and I really have to apologize for this slide, because I did something I never do, that I leave out the numbers on the axis, which is terrible, but I'm not allowed to show these numbers, so please excuse that, but anyway, this is real business data here. So, what you do when you have a classification problem and you're a data engineer or whatever, or data scientist, you do histograms. So, you do histograms like here on the top. We have the categories on the X axis and each little bar here, remember we have 2,000 categories, so we have many, many little bars here, shows how many samples are in that category and how many samples we have. And at first glance, you see, well, That's what we call a long tail distribution. So we have a lot of offers here and a very long tail over here. And what you can also see is that there are many, many categories that almost seem to have no offers. Believe me, they are there. But here, they are so little compared to the front part here that you can't even see them when you plot them in relation to the others. So what we do here is we subsample a data set to have an even distribution to allow the classifier to also focus on these formerly minority categories. So, we do a sub-sampling, we reduce the number of data, but we have a nicer uniform or almost uniform distribution here except for these few classes on the right where we don't have enough samples here. It's important when you do that that you don't do that completely randomly. You don't want to have the same offer over and over here for one category, you of course have to increase the diversity somehow of your offers. And fortunately, we can rely on subcategories. We have 30,000 subcategories, and then we have a few million sub-subcategories. So we can look into these subcategories, try to have a uniform distribution over them as well, then randomly sample from those, and that ensures maximum diversity here for our training. And that's the important part here. That's for training. We want the classifier to train on all categories more or less equally. But now we need to test, and we want to test in reality. And in reality, we have billions of offers coming to our pipeline. We have two, three, four million offers that can come in within a minute that we need to classify. And we don't know what their classes are. That's what happens in production. But we are able to somehow estimate how the distribution of these offers look. And that is what we have in reality. Or that's our estimation on the offer distribution we have in reality. And you see that diagram is very different to the one I showed before. Reason one is that's in logarithmic scale. The first one was in linear scale. So that looks very different. two is, and that's also the reason why it is in logarithmic scale, this is even more long tail. If I would plot that in linear scale, you wouldn't see anything. You would just see a huge bar here and then nothing all the way here. So logarithmic scale. So how do we evaluate on that real data we have, on that real data distribution we have? Well, we have people who label that data more or less continuously, but they are not really guided in a way, please match that distribution. They have their own business goals. So, we have labelled data that looks somehow like this if I order it in the same way as here on the left-hand side. And you can see that it's vastly different. It has nothing to do with the shape on the left. And it's also very sparse. That means we don't have labels for every category. It just has to do with different business goals here that these categories are not labelled. So if you would evaluate on this labelled data where we have really, really good labels, you wouldn't know what's happening in your real production data set. So what we do here is we develop a sub-sampling scheme again that tries to match this distribution to that distribution. That's what you're seeing here at the bottom. It's not perfect, but But it's still following more or less the same shape as you see on the top left here. And if we evaluate on that data set now, we have a lot better estimate of what is happening in reality. Okay. So much for data. Now let's talk about the more exciting stuff for many people. And that's how we select models. For models, there are millions around. And we have two major considerations here. And that is why we do classification. Of course, we need classification performance. But Tobias said it several times, we also have inference speed. So we want a fast model to process our millions of us per minute. And that influences all the model architecture considerations. Yet another slide with no numbers on the axis. And these are not the real data points, even that's just a schematic made up by me. But that's something you get. I cannot show the real numbers here. But that's something I made up. And that's something you will always see when you evaluate your models on your data. You will always have the trade-off between inference time and classification performance. And you will see a curve that is starting quite quickly and then settling down after some time. So for us, inference time is the most important thing. So we decided on that mini-LM L12 for now. We are still considering switching to this E5 small model, but we have to evaluate that a little bit more. So for now, we decided on the mini-LM L12 model. So who in the room has ever trained a classification language model? Anyone do that? Oh, awesome. So you will be very bored on this slide. So what you normally do, you download your mini-LM weights that is pre-trained, for example from Hugging Phase. This is the blue box here on the left. You have your input data that goes into the mini-LM. The mini-LM creates an embedding. An embedding is just a vector of numbers, in this case 378 or so numbers, that represent your input text. And on that you put a classification layer that's just a linear layer that is upscaling or linearly projecting to a large layer of our 2000 plus output classes. You normally train that with a cross entropy loss. You can do that. We do that. It works well. We were quite happy. So you can put that into production. Awesome. But can we do better? So if you tell or if you show this to business people or to managers, they say, well, let's take this one. Just make it faster. An engineer says, no, we can't. But there are things we can do. We can take that mini-LM and make it better. And maybe we can even use our best model in our case here, this E5 base model, to make our mini-LM stronger. And some people might know where this is going. This is going to what we call knowledge distillation. The idea is that you have a large model, like your E5 base model, and train that. You can't put that into production because it's too slow, but we train that model now and it achieves great performance, so let's use it. And then we take our small model, our mini-LM model, and train it on what the teacher has learned. The intuition here might not be too intuitive, but my intuition here is, and many researchers share that intuition, is that these stronger models that have more parameters are better to find this embedding space and are stronger in clustering this embedding space to be really, really representative. They can do it better than the smaller models because they don't have that many parameters. They're better in creating that. So, we now train our teacher model. We assume we have trained that, and the way I showed before, just that I switched this mini-alarm here to an E5 base, we trained that with our cross-entropy loss. Great. Now we freeze that model. Freezing means we don't train it anymore. It's in the training pipeline, but we don't update the weights anymore. This is not trained. It just gives us the embedding. Remember, this teacher model produces a stronger embedding, a more meaningful embedding than the student can ever do. It's arguable if that is a good teacher in the sense how humans would do it, but the teacher just tells the student model, yeah, don't care about the things you predict here. I tell you what you have to predict here, you have to structure your data here. You don't need to care internally why that is, but I show you what the output should be. And that's exactly what you're doing here. So you train your student model in the way you did before with the cross-entropy loss here on that classification layer. But you have a loss between those two embeddings. And if you've seen Hilde Kuehner's talk yesterday, this might look very familiar, right? This multimodal matching and so on. We do a similar thing here. We have an L2 loss between these two embeddings. The teacher tells the student, that is how your embedding should look. Based on that, you can train your classifier. That is the student-teacher training. That massively improved our performance here. What you do in the end, you throw away the teacher. Please don't do that in real life with real people, but you throw away the teacher, you just keep the student, and you keep this classifier here, so we are as fast as before with the mini-LM model, but with much improved performance from the teacher model. In fact, in some of our experiments, in some metrics, the student even outperformed the teacher slightly, but it still did. Okay, let me quickly wrap up our research findings before I hand back over to Tobias. I hope I could convince you that carefully building our datasets is super important. It's super important for training and also for meaningful evaluation, so you always have to match the distributions of your real data and your training and test data. I also showed you the student-teacher approach that massively improved our classification performance while So we are keeping the inference speed, the stuff we just discussed. And then just three points here that are findings, let's say findings on the way. Or takeaways on the way. We have now two losses. Let me go back one slide. We have two losses here. Now we have a cross entropy loss and then two losses. And whenever you have two losses, you need to balance those losses carefully. So you have to do ablation studies here, parameter searches, whatever, to balance them. Another finding is something that might look unfamiliar if you have been in the teacher-student domain before, is that I use an L2 loss here. It has just shown to be more efficient than the losses used in the literature. So, yeah, takeaway might be, hey, if you have a simple idea that you can quickly test, go for it. Sometimes it is better than what you read online. And the last thing, we also played a bit with quantization. So we used PyTorch's bfloat16 quantizations are just a way to compress some of the model weights to 16 bits instead of 32 bits, which makes matrix multiplications within these networks a lot faster, and that also sped up our training and inference. So now we finished the research phase in just a little over 18 minutes. So we did the research phase and Tobias will now talk about how we get that into production. At that point, since we trust our data and how we sampled our experimentation data, we can really trust our results here and we are very confident that these results really translate to production. Tobias is talking about that now, how we translate it to production and all the considerations you have to make there.

Speaker 2 [26:12]

The most important part is the endpoint and the first thing you have to decide is on which kind of hardware you deploy this kind of service. In the cloud you have several options. You have CPUs, you have GPUs and as we are on AWS you have also custom accelerator hardware that's called Inferentia. We are selecting this hardware based on three criteria. One is high throughput, low cost and also high reliability. And these are the categories we are looking for. Of course speed, we are looking for price and this could be a little bit tricky because faster and sensors could be more expensive but altogether it could be less expensive. So we are looking at customer sample. We are also looking into integration effort because you have to maintain the code, you have to integrate the code, and this can be a factor. And something which is also very important is availability. That means, as we are at the time in the age of AI, lots of people are using GPUs and using these kind of services. And even in the large data centers, we discovered that there can be a bottleneck of availability of resources, so this is a factor. And last but not least, but also very interesting, is that these custom chips, they are not so flexible because they are, for example, only designed for inference, but they are often less power-consuming, and this can be an option to reduce your carbon footprint. On this slide, I show you a table of the comparison we did. The red line is the reference implementation of the support vector machine, which runs on a graviton. Then I brought you some older GPUs and newer GPUs. A10 is older, A4 is newer, and then you have these two versions of inferential instances. Interesting what you see is that with the newer GPUs and with the new influential instances you get around about two times faster than the old endpoint but they are more expensive so you have to pay about 60% more. An interesting thing here also for us was that these old Inferentia instances, they were slower, but not that cheap. So you only pay like 18% more, and that was very interesting for us. Nothing is what we experienced during an experimentation phase that we run out of GPU resources and also inferential resources. There were times when something busy happens in the cluster where we were not able to allocate them. This is something which should not happen if you are before Black Friday and you could not upscale your approaches because there are no GPUs available. So we selected the Inferentia instances and this may be also interesting to use this more older approach because we have small models and Inferentia 2 is designed for deploying large language models and so on. And so they are more needed in the cloud. However, so this is what we selected. And however, if you work with this kind of optimized specialized hardware, This comes all the time with additional implementation effort. Especially in our automated setup, where we deploy automatically, the deployment pipeline has now become crucial. I would like to show you some concepts that we apply to our deployment pipeline. This is a really simplified illustration of our pipeline, but it shows the most important step. When we do model training, we train the model, and we do evaluation, and after then we deploy. The important step here is the evaluation step, because at this point we compute the accuracy of the model, and then we are sure that for us this kind of safeguard, with the metrics we achieve from the evaluation, we select if the model should be deployed or not, if the training is successful or not. And now, if you want to work on inferential two instances, you have to compile the model with a new package. This could be a possible solution where you say, okay, after evaluation, I decided the model is good enough. I compile it, and then I deploy it. The problem here is that during compilation, there could be optimization errors, and also it could be the case that you add a bug to the compilation code. So, of course, you have unit tests on all this, but you never know. The problem would be that you deploy a model which is not good, which has errors, and detecting this in production is really, really difficult, and also the bug fixing. What we are doing is that we have a pattern where we say evaluation needs every time done on the most production-ready endpoint. That means we train the model, we compile the model, and with this compiled model, we launch a classifier that's the actual production classifier same code as in the production environment and we test on this classifier so the advantage is that we can detect errors immediately and we have a real measurement of the performance of the classifier the disadvantage is more in the coding style because you have now relations between the code of the classifier and the code of the training and this is sometimes a little bit more complicated to handle to keep everything up to date if changes in the classifier, if there are changes in the model compilation. And there was another challenge which kept us quite busy. This is called dependency management. Especially in Python, this is not very easy. And my Kotlin colleagues ask me every time, do we really need to do this in Python? Yes, we do. So, unfortunately, the older version of Inferentia has... So what you have to do if you run this model in Inferentia, you need to compile it with a Neuron compiler, and there's a Neuron runtime. These compilation version and runtime version need to be totally the same. So in this place, you need to have the same version of the Neuron package. The drawback of the Neuron package is that it depends on PyTorch 1. And this depends on an older Python library. So every package you have here needs to be a little bit more older. Because we want to have our packages as up-to-date as possible. This has security issues. But another issue is also that the model training, in the model training we want to have the newest possible package because there's so much evolving, there's so much going on. For that, we need a way where we could, on this point, decouple the model from the compilation code. I think the best solution would be an O-N-N-X model, but we started with a hacking-phase model, and this was, for us until now, a good enough solution, so we stayed by this. But because HuggingFace has a high compatibility among different kinds of versions, so this was good enough, and I think if this breaks, we have to move to O- and N-X models. The last thing I would like to show you is the compilation code, because the funny thing is that this is the easiest part in that way. So, the compilation code is really easy. The more complicated part is setting up the Dockerfiles with all the setups. What you see here is that we load our model from the sagging phase. Then you have to compile the model. There you need an example text that we put in, an example feature. Interesting here is that you compile it in respect to batch size. We found out in our case that batch size from one is perfect, so a larger batch size doesn't improve our performance. This is also interesting because on GPUs, you have normally that you need a batch size from 16 to 33 to be really efficient on the GPU. And this is for us a huge advantage on the inferior instances. And then you JIT compile it and then you compile it with the Neuron compiler. You save it and then you can load it on your endpoint and it's ready to go.

Speaker 3 [35:29]

Okay, so I'm super happy because finally I can give you some real numbers. So our goal was to replace that old support vector machine with a newer transformer-based model, in this case the mini-LM-L12, but we built our pipelines in that way that we can easily change that model to another version that might come out in the future. So what we achieved here with the student-teacher training and all the tweaks in the research is a classification performance of about 30%, which is quite massive, considering that the old outdated support vector machine is not that outdated in terms of performance, because it already performed quite well. But we could increase the performance by about 30%, and with all the things we did on the deployment side, selecting the correct instances, selecting the correct production environment. Correct, of course, for our specific case. We could keep the cost down, so we had about an 18% increase in cost, but everyone agreed that it is worth it if we have that much more classification performance. So, we are super happy with these results. That brings me to the takeaways of this presentation. We showed you in the beginning that we decouple research and product development, so we let it run in parallel. It's not as decoupled as it might look here, so we both are researchers. We both do our research. We both do a bit of infrastructure work, but that's mostly on Tobias. With that, it really enables quick innovation. If something new comes out, if we have some new ideas, we can really quickly implement that into research, because we have our running production stack. Our production is stable, and we are very quick in innovating. Especially the student-teacher thing, we did that from research to production within two to three months, I guess, with all the experiments that were needed, actually, including that dataset sampling thing that I told you about. Speaking of data, I repeated again that I hope I could make clear that data quality is extremely crucial for your ML system and I would say the most determining factor for your machine learning system to run correctly and to actually do what you're doing. Often way more important than the model architecture. So, data is important. I also talked a bit about knowledge distillation, so how we can still use our smaller model but run it with the performance of a much stronger model. Tobias, in the end, talked about the ideal production environment and that it depends on multiple factors that need to be balanced, like speed, price, integration, and availability of these instances, and, of course, on our business metrics. With that, I am at the end of our presentation. We left some more time for discussion, and yeah, I hope we got some nice questions. Thank you.

Speaker 1 [38:50]

Thank you very much, Tobias and Bastian. We have 24 questions. Awesome. Yeah, but we still have like seven minutes, and I hope we will try. Yeah, I will start with the most voted questions. How do you monitor the quality of results per category over time?

Speaker 2 [39:19]

Yes, so we have two kinds of monitoring. We have like, when we are training the evaluation, we write kind of evaluation report, which is monitored over time per category. It's really fine-grained evaluation, but we also have online monitoring. That means in production, we are monitoring stuff like prediction score, and this is also metric where you can see, okay, there is something strange happens. So if you deploy a new model and the prediction score is going up or down, then you know there's also a bug which you haven't detected in your pipeline.

Speaker 1 [39:51]

Thank you. How do you handle new categories? Do you re-evaluate old products daily?

Speaker 3 [40:00]

Even worse, we retrain. So categories can change. Yes, we can get new categories. It happens the categories are split because our content team decided they need to be split for business reasons or the categories are merged. So that happens all the time. And we have a regular retraining schedule. So as you might know, we can't just change the model and add a category to that model. We really need to retrain at least at the moment. There's some ongoing research that might make this retraining bit faster that we can reuse some older weights. But yeah, so we have to retrain and redeploy and we do that on a regular basis. That's also the reason why we have a smaller data set than our full data set that we're trying on.

Speaker 1 [40:43]

Okay, thank you. You mentioned keeping production and experimentation codebase different. How costly is your productionization process?

Speaker 2 [40:58]

Good question.

Speaker 3 [41:00]

You do that all the time.

Speaker 2 [41:01]

Yeah, I mean, the thing is that once, if you have set up a good production environment and a good experimentation environment, they are somehow similar. So, like, the initial part is really time consuming, but then we are more like playing around, trying different kind of models, different kind of sampling. And of course, this takes some amount of time to put it into production, but it also helps to clean up the code. It also helps to take another look over the code. So when Bastian is doing some research, I take over the code. There's also kind of quality assurance. I couldn't say numbers, but of course, there are sometimes a week or two weeks in a two-month project where I would just sit there and make unit tests and all this stuff. But it's really worth it.

Speaker 3 [41:48]

And it depends a bit on what you're changing. If you're completely changing your data structure and how the data propagates through the model and change the model architecture, sure, that takes a bit longer. But, for example, just changing the model to a newer model takes us, what, 10 minutes, 15 minutes? So it really depends on what you're actually changing.

Speaker 1 [42:10]

Is the production model a multilingual model or do you handle language specific setup models processes separately?

Speaker 3 [42:18]

No, it's a multilingual model. Yeah, and we refine it or we fine-tune it on our specific languages.

Speaker 2 [42:27]

So we threw everything in the model. The complexity is more how we prepare the data.

Speaker 3 [42:33]

And the idea is that we also have high-level features among the languages, like sentences are formed in quite similar ways in English and in German, and words relate in similar ways in English and German and French and whatever. So all these high-level features on how words relate and what that gives us classification output, that's in all the languages. So it makes a lot of sense to have a multilingual model, to have just more data that you train on to find these relationships between words.

Speaker 2 [43:00]

Especially in our domain, because we are dealing with lots of brands and stuff which are really constant over different kinds of languages, and that's what I said before, that our data is different than data you saw on the research datasets, and that's something we have to figure out.

Speaker 1 [43:18]

If you have so many data, why do you use a pre-trained model instead of training a custom small model from scratch?

Speaker 2 [43:27]

Good question.

Speaker 3 [43:29]

We're still somehow limited. As we discussed before, our categories change over time, and sometimes we can't afford to train on a data set of the size of two billion samples. It's just not possible if we retrain a model, say, every two weeks. If the training takes, like, four weeks, well, that doesn't work, right? So, we also have to keep the training time quite low. And also, we need kind of curated data, as I showed before, with the distributions. You need nice, more or less uniform distributions to train a model that performs equally well on all the categories.

Speaker 1 [44:07]

Thank you. Did you consider using LLM to label the data for evaluation?

Speaker 3 [44:14]

Yes, ongoing.

Speaker 2 [44:17]

Yeah, the problem with LLMs at this kind, I mean, for validation, evaluation, it may make sense, but we don't get an accuracy where we say, okay, this is good enough for us. Because we not only want to measure, we also want to use the fine-tuned, like the labeling to retrain our model, and then there we have to be sure. And at the moment, LLMs doesn't give us this accuracy.

Speaker 3 [44:45]

Maybe we can give a smaller insight in our current research there. So we actually tried that out using a large model like ChatGPT or whatever, and asked, hey, please categorize in these 2,000 categories. And that's really, really terrible compared to...