From Research Models to SLAs: Operationalizing TSFMs with Python

Speaker 1 [00:18]

Welcome to our session today. I would like to introduce our speakers today, J.K. Christian and Catalina Felipe. They'll be speaking on from research models to SLS, operationizing TSFMs with Python. We don't have the QR code for the Q&As, so we'll have the Q&As at the end. We'll just be passing the mic. Yeah, give it up for our speakers today.

Speaker 2 [00:57]

Thanks, Agata, for the introduction, and thanks, guys, for dropping in. Super cool that we can get around here today and talk about this topic. It's very special to our heart. We have worked on this for quite some time. And as you might have guessed from the talk description, the idea today is that we have been developing a time series forecasting product of which the foundation model is a part of it. And we have spent quite a bit of effort Pythonizing these models into APIs and also expanding it to multiple use cases beyond forecasting. So we thought we will use this session to kind of walk you through it and then show you a demo of how the API works and how it has kind of improved our day-to-day forecasting work. So you guys might have been in the TAP PFN talk yesterday and things are changing rapidly, not just for the LLM folks, but also for us who work on time series. And, you know, back in the days, the forecasting problem used to be months and months of data collection, model development, and then you need domain expertise for each use case. There's always a model that you have to set up, that you have to improve, that you have to maintain, and then retrain and whatnot. And then we always used to run into these pain points that you don't either have enough data, the data is not clean, or each use case requires a different forecasting model, or the forecast themselves are outdated or you need a different kind of forecast because you're looking at a new market or product and then boom two or three years back we entered into the foundation model let's say era of our machine learning development and you know that quite a few open source models have come out since then that are in some sense making our lives easier and of course there are challenges with that but we really found that adopting to it has given advantages to us so far but the main challenge is given that we are sitting in a kind of an engineering team where a lot of our customers don't want to have this challenge of like taking each foundation model separately setting it up and using and so on we wanted to encapsulate into one single API wrapper and that's basically the story of this talk. So now having set the stage up what we did as I mentioned we started with this whole enterprise forecasting platform that I'm talking about today it's a product that's got different kind of things but we started with the classical forecasting services the the the style that we used to work formally you know you you used to have like batch pipelines meaning that you had one forecasting problem and then you used to run through a pipeline with a bunch of models in our case we were able to integrate 30 plus classical ML and deep learning models, and then you will run it on a weekly, daily, monthly basis, however it works. We started appending a second capability, which is the time series foundation model services. And as I previously mentioned, we didn't want to restrict ourselves to one foundation model. We wanted to have the opportunity to have multiple of these, which we could give a supply internally to our company within Siemens, and also have different kinds of tools associated with this, zero-short inference, fine-tuning, fine-tune inference, any kind of real-time predictions on demand, with no pain of rerunning or doing the recurring pipeline, which is really a big con from the first box here. And all of this under one shared platform layer, right? So you have one API wrapper, one access control, and one subscription which could ease in all the friction in onboarding someone who wants to do forecasting without conventional methods. And that's what we did. So we got all of these forecasting with no training priors, or training the model, the pre-trained model on your specific data, deploying this via our API without any, let's say, maintenance effort associated with that and more importantly doing all of this in hours instead of days and months which is typically what it needs if you are setting up any foundation model from scratch along with this you get all sorts of other advantages you have one model across multiple use cases so let's say tomorrow you come with a forecasting problem you're not restricted to one foundation model you can try all of them and benchmark which best best for you you also have the easy deployment aspect and of course the compute cost is much more savvy in this case. So the first question that you might have asked, of course, while I was telling is what are these foundation models and why we picked them. So we have so far hosted four foundation models, Kronos, LagLama, Times FM and GTT. So as you guys might know, the first three are open source. It's from Amazon Meta and Google Research respectively. They have very similar architecture, and they have some pros and cons. So, for example, Kronos is, from our experience, a very good general-purpose time series foundation model, which can do zero-short inference, fine-tuning, clustering, any kind of uncertainty-based forecasting. LagLama is very quick to set up, but does not have some of the features that Kronos has. And the same with GTT. Perhaps you guys might not have heard in the past. It's a proprietary model developed by Siemens that is almost exclusively trained on Siemens data, has a lot of features that Kronos has, and on top of that, it can also do classification and a couple of other things. It's super optimized for enterprise workflows and has a lot of particular industrial IoT context baked into it. And all of these models are included in this API. So this platform abstraction lets users choose by capability and not necessarily by the challenges of the backend implementation. That's one side of the story. The other side of the story is we get different profile customers, customers who are fully developers and who are very happy and comfortable working with direct API, and that's something that we started. That was our foundation. But we also started realizing that there is a bit of an adoption issue, so we started developing a front end app that is using or sending the query back to this API to retrieve results that you can just click and play with. And more recently, we have also developed a MCP server from this API so you can talk with agents and run your pipelines using the server directly for a couple of these models. Plus, we also have a custom plugin that we have included in a couple of applications that's used widely within Siemens. And these plugins also increase the adoption of forecasting using foundation models and, of course, using classical forecasting methods as well. With this, I give it to my colleague, Katharina.

Speaker 3 [07:55]

Thank you, JK

Speaker 2 [07:55]

Thank you.

Speaker 3 [07:58]

So as my colleague JK was saying we are Putting these foundation models into api's But our product didn't start like this and to show you the architecture Maybe it's good to take a step back and say where we were at the beginning So basically we had a Python package that we call ML pipeline builder here and with this package we could enable to trigger pipelines in Azure ML Studio, and in these pipelines you have your data, multiple time series machine learning models would run against each other, best one would win, predictions out. The thing about this is that a package would not be so user-friendly, so we thought, okay, let's put this behind an API. So an Azure web app. We installed the package there and right now the user would only need to fill up a config file, send it to this URL and the same pipeline would be triggered just like that. So of course we need security, authentication, we had OAuth authentication in this endpoint and with this we had our classical forecasting services. The thing about this is that you need to train the models. The predictions take a while to run, so minutes to hours. So when foundation models came along, we thought, okay, why not, and with the experience we have already with this endpoint, why not put the foundation models also inside some APIs? So we did that, and right now we have four foundation models, each behind one Azure web app, and so they have all of these different routes where you can do the zero shots, the fine-tuning, so the fine-tuning will trigger also a job in Azure where a user sends their data, some hyper-parameter tuning, and a model will be fine-tuned and then saved on a model registry in Azure. Of course, these endpoints are also protected with the OAuth 2.0 authentication. When building these endpoints, try to make this so the most similar possible so that we are not reproducing codes. And one other thing we did is we have kind of a central API layer, API central where all of the shared logic leaves, so billing, logging, and whatever could be repeated between these endpoints, including the ML pipeline one. One other thing we did was, since we now have five endpoints, we kind of need a way to manage it. We put this inside APIM, so an Azure API management service that will help us with authorization, so each of these endpoints we can decide which users have access to what, some rate limiting, and a good thing that APIM brings, it's a single gateway, so the users will only see one single URL, and they only interact with this one. And so, as Jacob was saying, we have four different consumption channels, so direct API access, front-end app, MCP server, a custom plug-in, all behind this architecture. And now for a bit of a more practical example for you to see how easy it would be to just change a model, you have here a custom payload where you have Kronos as the model and the mode zero shot. If you wanted to trigger the LaGlama one, you just have to change the model here to LaGlama and it would trigger another endpoint. And the response is as well standardised for each model. So what we have here is something that, for us, it's easier to develop. So if you want to put up a new foundation model in a new web app, we already have a standardised structure, and for the customer is also better, because they don't need to worry about the different foundation models that are around. If a new one comes along, they just want to put, okay, model is this new model. So these are the good things about this architecture. Then what we also did was put the endpoints behind the MCP server, so the tools that are available in the endpoints are now also on the MCP server with the forecast fine-tuning, the inference on the fine-tuning, also the resources that we created and prompted to make it easier. So with this, we can have users that are not so technical to talk with natural language with the MCP server and they can access our endpoints without too much technical knowledge. And now I'm going to show you a video about how this MCP server could be used. So in this interface, we are connecting to the Chronos MCP, and it's asking to load all of the Chronos time series forecasting tools. So it's saying it has inference pre-trained, the fine-tuned, and it has two architectures for Kronos, and embeddings. So those would be the tools available for this endpoint. And then you interact with it and ask it, okay, let me go a bit further ahead. Please you give some time series. Please forecast me the dates, the values for the next three days. It would go to the tool set, it would take up these time series data that you sent it and put it in the correct format, send it to the MCP server, and then it would return back the results in natural language. So it was thinking, but it's fast, that's fair. And then it would turn you with a graph. And what's nice about these clients is that they produce the graphs without any effort. So maybe going ahead, just a brief mention to what else can we do with these foundation models, because we always talk about forecasting here, but what's really great is that these these foundation models could also power other cool stuff here. So just as an example, we have here these four things. So anomaly detection, you can do it multiple ways. So with the confidence intervals, reconstruction-based, and then you compare the actuals to the predictions of this model. And then if it's too far off, then it might be an anomaly. You could actually also do it with clustering. So if you transform this time series into embeddings, and if it's too far off into the cluster embedding space, it might be an anomaly. So clustering always works with these embeddings logic. Could also do the same for classification. So if you already have some labeled patterns, you produce embeddings, put up an ML classifier on top of it, and you have labeled time series. And last but not least, imputation. So back to the logic of reconstruction. If you have missing values with the foundation models, you can kind of reconstruct this time series and fill in these missing values. So it's good for historical data repair, for example, and all of these things have practical examples. We're going ahead. Maybe JK.

Speaker 2 [16:59]

So, of course, while you guys might have seen there are multiple models, then obviously we come to the model benchmarks as well. So, of course, you can imagine that there is no one-size-fits-all best model, right? It depends heavily on your use case and so on. But for some of the use cases that we heavily got exposed to in working with folks within Siemens, for example, on the finance domain, we kind of did a bunch of different things. First, took the foundation models, tried to compare them with each other. compare the zero-shot inference outputs with that of fine-tuned ones, and, of course, compare it to a classical ML benchmark. So on this graph that you see, that's an error metric on the x-axis and execution time on the y-axis. Ideally, you want the best model should be on the lower left quadrant. And the auto-ARIMA benchmark, which is the classic ML benchmark that we set up, is right up the top. So it's very low error but relatively high execution time. and you see more or less the best models that do well are the Kronos models, Lag-Lama and then like Lag-Lama fine-tune is very close to the zero-shot inference one. So our general observation so far is that as a whole foundation models are typically faster particularly when you try zero-shot. More accurate, they are reusable and generally more versatile so you can literally drag-drop them from one domain to the other. And going forward, But comparing how zero-shot in short-term horizon compares with that of long-term horizon, we didn't find too much difference in terms of the model performance there. As you could see from the graph here, Chronos, LaGlama, and Times FM came out to be one of the good ones in both cases, whether you're looking at a couple of months or a year. I think zero-shot is a very good, fast, and useful baseline to start with, and then fine-tuning is worthwhile for when you want to really optimise it for your domain-specific use cases. I would still throw it here that the classical models are still important, and they are very good benchmarks to compare with, and maybe even optimal for some use cases. However, overall, we found that foundation models are a very good place to start with, at least for folks who haven't explored this in the past. And as I mentioned before, this is like, it's per use case, the results that we are sharing, and so model choice should always encourage our customers to try out different things and then benchmark based on that, which is quite possible in our setting because as you saw, we offer multiple models and people could try out which is best for them. So with that, we would like to shortly show you a demo of our API. Let me just bring it up. So time permitting, we hope that we can show you four or five things that we are quite excited about. So the first level, we want to show you a little bit about the authentication and how the API integration works, and then quickly do a zero-shot forecasting here, and also look at kind of the payload of the request and the response, and then maybe we could trigger a fine-tuning workflow as well together. And then Catharina will walk us through a couple of very specific use cases that she talked about, including anomaly detection and embeddings. So the data set that we took here for illustrating some of these stuff to you is IoT machine sensor data that we also typically encounter within the Siemens environment. So there are multiple machines here in the setup. So it's a structured data. And each of them has a temperature and vibration property associated with it. So here at the beginning, we are just importing the libraries. And then we have some helper functions that do the plotting. And then now if we go quickly down here, you can see here that we are authenticating the connection. So it's basically you just pass your API credentials and then it works. And then here we are loading the sensor data. So you get a kind of a general idea of the data set overall. So M1, M2, M3 are like sensors or machines within this data set, and each of them have two properties that you have. And then if you actually analyze how this time series data looks like, it's very highly fluctuating, as you can see. And some of these machines also have these extreme outliers, which is also a very good test case for us to try out our foundation models. Now, coming to the zero-shot part, we have set here a forecast horizon of 10, and M1 as a target column. And then here is where the piece that Katharina talked about as well comes into play. So you basically pass the model, and then automatically the respective endpoint gets triggered. So for example, in this case, we have previously set laglama, and now we can try this again. It calls the laglama API and then returns the execution time. Now you could, for example, change this to chronos, and then it would call the API and then return the respective execution time. So it's basically a swap of the model name, then you get the zero shot because all models offer you this feature. And now if we rerun the model forecast plot here you see along with the original historical data you also see the forecast and the actual values and most importantly the confidence interval which gives you a kind of idea if there are anomalies within the prediction. Now going on to the request response inspector. So we also wanted to show you a bit like how the request payload looks like, so you have the parameters that we passed which includes the forecast horizon, date column, key column and so on and so forth, the data shape that we passed and then the response structure has some metadata associated with it, so the execution time and then something about our API credentials, the timestamp and the results and so on. Now this is the forecast values that we caught, but you could also quite literally fine tune in like five lines of code so for example let me trigger this pipeline here right now and then we do have like overview of pipeline on our Azure AutoML setup and here once this pipeline kicks off we will go back to it and check out. You see now we triggered a pipeline right now, and then if we go into it, there will be a component which is now storing all the steps of the pipeline, which is basically one because we are working with a foundation model which wraps every step. Now with this, I also have one short point to add. We also have a model registry where you can get the overview of all models that you ran in the past, which gets collapsed across the use case that you're running. I would like to give it to Katrina to go over the...

Speaker 3 [24:30]

We're running a bit out of time, JK, so I'll just be really quick. So with the anomaly detection, you also have a route for detect anomalies. And in the machine number three, we knew that it was running into an anomaly because the temperature was rising really very much. And so we forecasted, we went 10 data points back, forecasted, and then comparing to the actuals, something's off because it's very far off from our confidence intervals that you can see here. And so this is a way to detect anomalies. And then here you also have some cool stuff with what you can do with embeddings. So you call embedding, zero-shot embedding, routes. Every time series is translated into embeddings. And then if you do a simple PCA analysis, you can clearly see that the temperature is separated from the vibration and you can kind of cluster them like this. So if you want to go for the conclusion, JK.

Speaker 2 [25:32]

So just to wrap up what we discussed so far, it took quite a while for us to get here, but we realised that operationalising TSFMs are a good task to pursue. We are able to now offer at least to our direct customers within Siemens and outside a unified API interface, which can offer all of these foundation models, and the coolest part is adding the fifth one is going to be hopefully way easier than when we added the first one. And we have a standardised payload and response schema, and automatic lifecycle support, and what we have coming next is that containerised deployment will come to it, maybe if you guys want to chat after the talk, happy to do so, monitoring, load testing, and more broader branch markings that we can display to you. With that, I would like to thank all my colleagues for the support in developing this product, and we have another talk tomorrow on surviving AI fatigue. Please drop in if you are interested, and thanks a lot for your time and attention.

Speaker 1 [26:49]

Yeah, it's not time for question

Speaker 2 [26:50]

time for questions

Speaker 1 [26:53]

The public QR code?

Speaker 2 [26:54]

Yeah, let me get it, just a second, please.

Speaker 1 [26:57]

We do have a few in the chat so I can start with that. Do you support Madhvari time series or time series with?

Speaker 2 [27:11]

Kajus.

Speaker 1 [27:13]

in many domains, e.g. demand prediction, those are unavoidable.

Speaker 3 [27:19]

Yeah, so for the GTT model, we already allow for covariates to be used. For the Kronos, we're working on the migration to the version 2.0, so in short, yes, we already allow it, not for all of the models.

Speaker 1 [27:38]

Okay, how much does hosting and using the TSFM models cost?

Speaker 3 [27:50]

Hosting is the biggest resource that takes up the most costing. It's the APM, actually, to manage all of the endpoints. But everything is under 1,000 euros.

Speaker 1 [28:12]

Do you store the context of the call? So one might query something akin to the trained model endpoint later?

Speaker 3 [28:24]

If you store, can you repeat again? So for the zero shot, we don't store the context. We only log the accesses that have been done. So we don't, we are not storing the data that the user sends us as of the moment.

Speaker 2 [28:43]

Now we have the QR code, guys, if you want to put in your questions. Yeah.

Speaker 1 [28:47]

What is the difference between your platform and NextLab client that also supports managed endpoints for the same set of the models?

Speaker 3 [28:58]

So this basically is to be an internal Siemens product, so even in terms of security and we are hosting our own models, GTT could not be out there and it's a great asset because it's also trained on Siemens data and it's one of our best performing models, so I hope it answers it.

Speaker 1 [29:32]

What's the typical data volume you support in a single call?

Speaker 2 [29:39]

So you guys might remember that in the last slide, we have the load and stress testing as one of the activities that we are working on right now. I think we were able to do at least 100 concurrent requests, right, Katrina? And volume of data, I would say the maximum that we have tried.