Beyond Deployment: Exploring Machine Learning Inference Architectures and Patterns

Speaker 1 [00:08]

I'm Tim Alfring, I'm from the Netherlands, and I'm a Machinery Engineer at StepStone. For the people that are not really from Germany, StepStone is a job marketplace, so we try to find the right job for everyone, and we have different machine learning algorithms that are deployed on websites, and I want to show you how we do this, and what kind of patterns we use for this. So as I mentioned, we have different machinery models, and one of the models is salary estimations. So we have a model that if you give some input about yourself, you can get some estimations so it makes you easy if you find new jobs, if you get offers, if it's a fair offer, yes or no. And you have to fill in a small questionnaire, and when you do this, we have a prediction. So how does this work? You give some information about yourself, so your location, what's your job title, what's your experience level, our model will do the magic, and then there's some kind of output coming out of this. And how this works on the website, really basic is, okay, we have a user, interacts with the front end, there's a back end service there that is querying our model, or sending requests to our model, and then we send this information back. Or most of the things are just handled by the backend server, only like this couple of values that we are trying to get for the specific user. And we have within our company, it's not super strict, but we have different responsibilities. So we have software engineers that are more for the front and the backend. We have machine learning engineers, data engineers that are really for the serving part. And we have the data scientists that are more focused really on the output of the model, what's the quality of it. And today I won't really talk about the data science part and the software engineer side, but I will really focus on the model serving part. And a couple of things that I will go through when talking about different patterns. I will talk about the latency, what that means is how long does a request take, the throughput, How many requests can the service handle in a given time unit? So let's say our throughput for certain services is like 10,000 requests per hour. Availability. So is our service always available? Or maybe sometimes it's not available because there's some scaling going on. Resilience. So what happens when there's like a spike in traffic? Can it still handle this? And also a big factor is like cost. So, of course, we can just deploy our model with a really big server, and then we can handle all the requests, but probably then the company will go bankrupt. So I won't focus on security aspects, like alerting, monitoring, like the model updates, and also I won't really focus machine learning inference on edge devices, because we don't really do this and I'm not really an expert in that. So back to our first example. This is an example of real-time inference. So we sent the request and we got something back. And how this is good for use cases where you need low latency. So it is like front-end service, so the user interface, so people expect when they submit their survey that they get immediately like something back. It's also in general low throughput use case like if we have like 10,000 people clicking on the button at the same time getting some request, probably our model surfer will go down. And we have a couple of those examples within StepStone that we have like a candidate fit score that you get, job title recommendations for recruiters when they create like jobs and the salary estimation planner. And how can you deploy this? So really this model serving part, for example, just like scikit-learn in fast API, you just build a Docker around it, and you just have your service. And how do you actually scale this? So as I mentioned, there's one easy way to do this, it's just get a bigger machine. So you just calculate what will be your peak traffic. You just estimate, okay, then I need this machine, and you just deploy it, and then you will be always safe. One more cost-effective way is to actually deploy it in smaller machines and have a load balancer together with an autoscaling group, like pick, okay, which surfaces are now idle and I can use. So this is classical vertical versus horizontal scaling. and the first one is like yeah you over provision and the other one is like really based on traffic so if we see there's like more load on our services we can just spin up more machines in the cloud so that was like one of the things that we can just do is like if we get a lot of requests we have just auto scaling groups there that we can when there's a lot of traffic we scale out when it's like in the night like there's not so much traffic so we scale in again and we can in our service we can also add some caching the nice thing about this like real-time inference endpoint is that you can also add some other services so not only like your backend service but also internal use cases can use this prediction service so what we usually do we have our services and before we actually go live we have like i don't know some kind of streamlit apis that we can have some playground for product owners where you can actually put in some values and you can actually get some answers. And we also supply this with, I don't know, some SHAP values or other things, but really that people early on in the project before we actually deploy this also can interact with our model. So this looks super similar to the first slide, but this is from a different project where we also do salary estimations, but these are salary estimations just for a job location and a job title. And we have that in a lot of different locations on our website, that we have those estimations. And what we here just can do is we can have all combinations that we want to put on our website. We can already put it in a big file. We can make those predictions, and we can just store this in the database. So we don't need any real-time inference. What we just do, we just have a Docker container, or maybe it can even be a notebook that It reads all the different combinations, loads the model that the data scientist made, and then you do a prediction, and you store this somewhere. And then you can just store this in some kind of database. So as I said, this can be like a notebook, can be like a dog container. This can be also like a GPU cluster if you have large language models. So you can make this as complex as you want. And one example where we use it with StepStone is every night we have a job that is calculating what will, predicting how many applications will we get for every specific job. So we have just like a trigger every night. We have all the different jobs with their features and then we make the predictions and then we store this in our database. And how do the services then query this? just the application server, like our backend, can just read from this database. So then we just pre-populate this database every night, and this works really well. And also this database, if we just have like a key value storage database, nowadays they're like super fast, so you have like, they scale really well, and you have like really sub-milliseconds response times in the cloud. So the nice thing is that the application really doesn't know actually anything about the model, so you kind of decouple this from your application. So how do we actually also scale this? So let's say we have to handle like 100,000 of listings in a night, we can actually just partition the input files, maybe in ten different files, and we have a processing container that we also partition, so we have ten containers running there that creates the files in the right input. Then we have the models that also are different containers, like we can also just scale that, and in that way, let's say one model can handle 10,000 items in an hour, then you can just have 10 machines running in parallel and you just do like 100 000 in an hour so like this this kills just like horizontally um so that works really nice um in that way um the only issue is that you have like a cold start problem so what i mean with this is like we have also products that we for example recommend um jobs for specific jobs so you have a listing there's like oh maybe you also like those listings and if you do this every night this can be an issue because we have a new job that is just being published it is not being found in the night yet so we don't have any predictions there so we need to have some like tailor-made solutions there maybe some business rules maybe looking at similar listings and then just give the same recommendations there so there There are some gap fits there, but it's really good if you want to do high throughput predictions. Another use case, again, salary. We also have salary estimations on jobs, so if you are looking for a job, and we try to make it transparent for people that search jobs, okay, what kind of salary can you expect here? And this is something that not all companies, especially in Germany, are really transparent about and we try to push this with some estimations here. But what this means is that every time there's a new job being posted, we need to also have some trigger that we say, okay, well, what is then the salary? So we have to use our model again. And how we do this is just in an asynchronous inference way, so like an event driven way. So we have like incoming events, we are reading from like a queue, and our model is like, it can be like also just this fast API endpoint again, that is instead of getting HTTP requests, it is just pulling from some kind of queue, it's one by one grabbing all the events, and And then making predictions, and then put it in another queue for another application to pick this up. And within StepStorm, we have a lot of use cases where we actually have this event-driven way. If there are new listings, or new candidates sign up, or maybe someone applied for something. So we have a lot of different use cases for this event-driven way. And some of our use cases have issues. For example, the real-time endpoints, it can take quite a time. So maybe for large language models, it's now, I don't know, it depends what model you have and what kind of container or instance type you're putting it, but it can be like 20 seconds before a response is coming. And that can be an issue if you have this real-time endpoint and you actually have to keep this connection on all the time, and in this way, there's no connection, there's just like pulling and pushing all the time. so again how do we make this in scale so we have some data producer so i don't know some kind of application within our system is is notifying us okay there's new listing coming so at that moment it's being populated and let's say we have 10 000 items maybe there's like a new crm system that is pushing all their jobs to us so we have 10,000 events in there then our model can pick them up one by one but you can also again scale the model so you can have like multiple instances that are consuming from the same queue so you can again compute in parallel so this queue is going way faster and then you put it again in your prediction outputs queue and then there's some kind of surface that like picks the picks us up maybe there needs to be some post-processing post-processing being done, and then it is being populated to the database again. And as you can see here, again, we have the same pattern that actually our backend server has no idea actually how the models are there, they are just like querying our prediction service and we are just reading from our database. So again, this decoupling, which is for our backend team, I think quite nice. the other nice thing here is that in a night, again, there's not so many recruiters that are publishing new jobs for us. So sometimes there are just no requests coming in. What you then can do, you can just shut down all the instances from your model, and you don't pay for those GPUs anymore, and then at the moment there are, again, events in the queue, you can just spin up a new machine. It doesn't really matter, there can be a million events still in this queue, and it doesn't really matter if it's being picked up now or maybe in a half hour, so in this way we can also optimise a little bit on cost. The disadvantage here is that maybe it's a little bit more complex than just a simple real-time inference endpoints because you have those queues and maybe also the consumers and the producers for the events, and just like the batch process, the backend needs to know that there might not be a solution there, a prediction there. Maybe the prediction, the request is still in the input queue, and people are already going on the website to request the salary estimations, but they're not there yet. So the user interface needs to just be aware that this can be empty, that we send requests back, no, it's not there yet. One thing, when I like this queuing system, this asynchronous way, can scale really well. You can put one million events in there and the model will pick it up, you can scale. Why would you need to have this batch process still? How this works is that most of the models, if you do one individual request 100 times, the time that it takes is longer than if you just do it in one batch. So like this dot predict and then you just do it in a batch, because there's a lot of optimisation happening already in the code. And how to solve that is like with this asynchronous way, it's like instead of getting one by one all the events, you just grab them in batches and then you put them together and you make the prediction and then you split it again for the prediction outputs so this works well and it comes close but it doesn't really like still if you want to optimize on cost or on gpu power i would go for the for like the batch process because you will save quite some cost because the other one has quite some overhead in that and will just be in the end slower for the same compute. One other thing that we have is serverless compute, like it's kind of a combination, like if you're aware of the cloud providers, like you have lambdas in AWS, you have Azure Google Cloud functions and they all work kind of similar so like this function is not really like a container that you pay for per minute but you kind of pay like for the request times so it will scale like super good so you can have so many concurrent requests you don't really have to care about any containers anymore so this can also be like a good way to deploy your models the only thing is that yeah it doesn't really work like there's no at least in the big cloud providers there's no GPU serverless instance at the moment so for quite some use cases it doesn't really work also the memory in the CPU are a little bit restricted but if your model fits on it, it can be really good and also cost effective way to deploy it and it can be triggered by HTTP requests but it can also just by events like with asynchronous way so when to use what so like this this is just like summarisation what I just had, I hope all the values are correct let's see about it later if I get some comments about it the only caveat is it might be a real time endpoint, like the costs are high because you always have it live, but it might be the case that if you compare, for example, with batch, and you want to pre-compute a lot of combinations, and you need to have a lot of compute for that, it might actually be just beneficial to have just one real-time endpoint running, if you just don't have so much traffic. Like everything, it just depends. So, what we actually do with some of our projects is combining our different architecture or patterns. We have a project that we are summarizing job listings. This can be super useful if you are on a mobile phone and you have this long job listing that you just see a couple of lines or we do it for email campaigns or for chatbots. The way it works is that every time we have a new listing so we use the asynchronous way every time we have a new listing we just process them we uh we have like a large language model behind it so we do some prompting then we send it to the model then we are doing something with output and then we store it again in our prediction database and this works really well because there's a little bit of an issue with those large language models as i mentioned before there can be a little bit slow so to put it somewhere else might not really be that good, and also we want to have it actually as soon as possible. But what we also have is we noticed that because they are completely automated, we also want the end users to have a little bit of control with it. So we want them to also be able to change them or maybe to regenerate them if they want to change it. So we still also have our real-time inference endpoints, But then it's in a certain way facing that the users know that they have to wait a little bit. So they are pre-populated, but they can be regenerated on the fly if necessary. So in this way, we can combine them. Okay, so now we have actually all our different summaries always in our database. but now we have something that we, or data scientists have new iterations of their model and we actually want to improve our model and we might have like a different output, so then we have all the listings that are now online we want to reprocess with their new model and make sure that they are now also being shown to the end user then we use our batch process, so then because it's the most cost effective way for us because we have to use our GPUs. We are using the batch process there. We just get all the information, all the listening text there. We put them in some kind of big job, and then we process the output and put them in the database. So I'm a little bit at the end already. So what I would advise, if you start with your projects, start simple and really look at the different patterns that are out there. How do you want to serve it? are people expecting to have an answer immediately are they able to wait do you actually need to have some endpoints or can you just keep it with some kind of batch process that you just run in the night and everything will be fine um yeah so actually that was it and i hope you learned something and thanks all for your time and if you have questions come to me after the talk or now or find me on linkedin

Speaker 2 [22:18]

Thank you for the great talk. We have a lot of questions, so we should start with the first one. That's the most upvoted one. Could you add 10K by standard to all estimations so we get more base salary?

Speaker 1 [22:37]

I don't know, we can change a little bit.

Speaker 2 [22:40]

So second one can you share details about the data set size you use make predictions and whether it's typical developer who submits data

Speaker 1 [22:51]

Yeah so I covered quite some use cases now so yeah we like it really depends so like for example or recommendations systems they are really dependent dependent on like who is actually applying for the jobs like what which jobs did they visit did they get hired yes or no so that's all a click data we also have other data that we just collect from our users by filling in some information

Speaker 2 [23:19]

Okay, next question. Which technology do you use for the message queue?

Speaker 1 [23:25]

um we use kafka but like it doesn't really matter um yeah like like the idea is like it's more here not really uh focused on technologies but more like yeah on the pattern stuff we also used the rabbit mq in the past

Speaker 2 [23:43]

Does queuing your input not make the end-to-end user latency higher? Does queuing your input not make the end-to-end user latency higher?

Speaker 1 [23:56]

Yes, so that's why we are really beforehand thinking, okay, when are the users going to see it? And for us, it doesn't really matter. So with the listing predictions for those salaries, we don't really care if those listings, the estimations are on there right at the moment that the listing is online or maybe half hour later. The end users don't really care. listing will be there online for like a month so that delay is okay but as i mentioned in the beginning there are some use case that this delay is like uh yeah we should care about and that's also why we then choose other solutions

Speaker 2 [24:40]

Have you considered deploying and running models into the front-end with frameworks like ONNX, Web Front-end?

Speaker 1 [24:47]

Yeah, there's for us no need for that, I think our front end is already pretty complex with a lot of different surfaces there, so it doesn't really fit in our stack I would say, but it can be a possibility, yes.

Speaker 2 [25:04]

What did you take on managed API serving with, for example, AWS SageMaker or HTML?

Speaker 1 [25:11]

Yeah, it can work great. So we have services running just in those fast API endpoints, but we also have it in SageMaker. The nice thing about those managed services is that they orchestrate also a lot, so it's really easy to do some auto-scaling there, for example. It's really built in. And also they are coming with some model registry, which is pretty nice because we deploy it in different patterns. And then you just pick, like you reference the same model. So that makes it nice to not kind of reinvent the wheel that you have to do your own model versioning there.

Speaker 2 [25:49]

In your experience with asynchronous inference, does FastAPI able to deal with tons of events driven requests? Do you apply any optimization on the endpoint?

Speaker 1 [26:01]

don't really have experience with that so like we are now using um ourself a sage maker asynchronous endpoints and there we actually send sns topics now it actually works is uh yeah pretty simple like you send the topics uh with an attachment where the payload is with the location to an s3 bucket so like then in the like SageMaker is kind of kind of fixing it for you so you don't really know it as end user but that's how then works in SageMaker so like you always have like reference to like what data with in our use case like the prompts and then you also when it is done there's a message back with also reference to the output in like some kind of S3 bucket

Speaker 2 [26:49]

Okay. Do you have country or even region-specific models or a global one?

Speaker 1 [26:54]

So, our salary models, they are all country-specific. We don't combine any data there. We did it in the past, and we found out specific models work best, and I think that is for most of our models that we have or language-specific or country-specific. We have one model that I helped develop, it's like the summarizations with the large language models, and actually there we have just one model, and it can pick it up really good if If you have like an English prompt and you actually ask for an English summary, or if you have a German prompt and you ask for a German summary, that works just fine in this model. So yeah, that's it.

Speaker 2 [27:36]

Based on which triggers are models retrained?

Speaker 1 [27:40]

depends so like our salary models they are retrained every couple of months because you can imagine that if you have your salary estimation that we give you now maybe in one year that's changed it a little bit some models they are more static than others so like yeah it depends on the use case and it's like also more like a data science questions they have to know the business needs and then they can kind of tell us like okay when do we need to retrain it

Speaker 2 [28:10]

Where do we deploy these different architectures?

Speaker 1 [28:13]

um yeah so we do it in sage maker we have some lambdas there so that's like the serverless components um for example in the um if if we have like the in the asynchronous one all those services they're all lambda based so that's like the surface uh components there so like if there's actually nothing happening then we also don't pay for anything there um yeah and also the database is the dynamo the b so we also only pay for like the consumption of it so that makes it I think cost-efficient.

Speaker 2 [28:46]

Is it vanilla scaling as with Kubernetes or is it custom?

Speaker 1 [28:51]

Yeah, so not Kubernetes, but yeah, it's like I think they have behind SageMaker like some similar technique as ECS

Speaker 2 [29:00]

Why serverless compute only works on low CPU and memory? Could you elaborate on that?

Speaker 1 [29:06]

Well, there are some limitations with the amount of data that can be in those containers. And it's also really product specific. So maybe I know a little bit about AWS and if you have load just or big models, it doesn't work. If they're a little bit smaller, they work.

Speaker 2 [29:30]

How do you handle cold start during scaling?

Speaker 1 [29:36]

Yeah, so if it's like usually we don't really care that much and it depends again per use case. So if the use case Is really critical. We just over proficient a little bit so we make sure that we never have like Time once there

Speaker 2 [29:53]

So the time is quite over, but if you like we can do some more minutes into the coffee break

Speaker 1 [29:59]

I think people can come to me I don't know if it's relevant for everyone but I will be here still alright then