Build a personalized Commute agent in Python with Hopsworks, LangGraph and LLM Function Calling

Speaker 1 [00:07]

Hello, everyone. First of all, thank you for coming. And thank you for the introduction as well. So my name is Javier, and I'm going to talk about how to build personalized commute agent in Python, only in Python, with HopSorg, Slangraph, and Function Calling. So as you know, world models are becoming more and more capable. Recently, we got a couple of releases from OpenAI and also Meta's LAMA. So with these new models, they basically improved efficiency for low context reasoning and also lower latencies. And especially, they increased the context window. So they allow for more tokens to be handled for the user queries, which something that we will talk about a little bit later. But the thing is, so these models are trained normally on the information from the public internet, and this means that in the end they have some understanding on the human language, so they can interpret what we mean and our queries, and they have also a broad knowledge about the world, but this also means that they are somehow limited. The knowledge that they have is static and is fixed, so they don't learn. And also, yeah, we can see it in every release, there is like a cutoff, a knowledge cutoff date, which basically sets the limit or the boundaries of the knowledge they have. So for example, with these models, the ones that were released recently, they have like a knowledge cutoff date of June and August last year. And also, as you can see, they improve quite a bit, the context window. So this helps, as we will see later, this helps add more context to the prompts. So basically, if we want to ask these models what are the chances of Real Madrid winning the Champions League this year, unfortunately, we know that they are already out because this happened more or less recently. But if you ask one of these models, they will answer you that, yeah, Real Madrid has a great history on that, on the Champions League, but they don't really know that they have been disqualified already. So how can we fix this in the end? There is a technique called retrieval augmenting generation, or RAC, that helps you add contextual information to the user prompts. So basically you augment the user queries with some contextual information. It can also be personalized information based on the user that is using the system and also can be real-time information. So you might be consuming some real-time information, maybe computing some aggregates or something like that and then add it to the user prompt to add more context. So the way it works normally is there is a user that interacts with a virtual assistant. let's call it, and it sends a prompt, and then the virtual assistant will get this context from external data sources that can be vector DBs, if you have some documents indexed or a feature store or some other data source, and then it will augment the prompt and send it to the LLM. So it has some knowledge. So when it comes to our previous question, to solve the problem in this case, we would need to basically index Wikipedia all the time as it gets updated, and try to retrieve this information to enrich the prompt so the LLM knows that Real Madrid is not classified. But the problem here is that you don't really know which question is coming from the user, so it might be a different question, so it means that in the end you need the whole Wikipedia to be in the context just in case the prompt is different, or you need some tools to basically find the relevant information and not bring all of it into the context. So in the end, this is more or less like whispering to an LLM, like cheating, or adding more information so it's able to answer the user prompt. So with the latest models that we saw, they increased considerably the context window. Bringing the whole Wikipedia into the context, it's closer to be, like, feasible, because you have, like, up to 10 million tokens, for example, for Lama 4. But yeah, it's hard if you want to deploy these top models, because they are a kind of resource hungry, and they have some GPU requirements. I think for Lama 4 it was like 800 GPUs, so that's pretty expensive as well. So it doesn't work always. So how can we fix this? There is another technique called function calling, in which in the user prompt, when you augment the user prompt, you can specify some functions that are available in the system. You can describe which functions are available in the system with the name of the function and some parameters and pass it together with a prompt to the LLM. So then the LLM will answer, will decide if some information is needed and is missing in the context, and the answer will be the actual name of the function and the parameters that need to be called to retrieve this additional context. So in the case that we had before, So, in this one, we get the prompt, then we ask the LLM what to do next, and then we get the answer will be you need to execute this specific function because I'm missing some information, and the functions that can be implemented are basically as you wish. You can make predictions with another model by querying another endpoint, or you can interact with another LLM system, you can generate some Python code, you can execute half a function that executes Python code, and so on. So in the end, there are multiple functions that you can implement to extend the capabilities of this retrieval of context information, and this is normally a tedious task if you want to implement multiple functions. So here is where LanChain and LanGraph comes into play as well, because they help you build this kind of workflows with multiple functions in an easier way, as we will see later. So basically, LanGraph, if you're not familiar with it, is a tool that helps you define workflows as directed graphs, as we can see at the top. And there is some state that is shared across the nodes, every node is a unit of computation, and then the edges define basically the flow between the nodes that can be, like, a sequential, let's say, flow, or it can be conditional. So maybe based on one state, you want to reach one of the nodes or a different node. Yeah, then basically the nodes are specific tasks, and that would be it. With these tools, these are the main concepts, basically, you can build like a graph where each of the nodes is a different function and then add some logic to it. So coming back to our function calling, we said that we can enrich the user query with some information about available functions in the system, which is basically methods that you can call within the system. So it can be done, for example, in that way, by serializing the metadata or definition of the function. We can say there is a function with name, get historical departures with these parameters of this type, and also add some specific information of each parameter so the LLM knows. And then the output would be, so, for example, if we are asking for average delays in departures, then first we need to retrieve the historical information of the departure, so the LLM will answer something like you need to call this method, and then once we execute that function, we will retrieve that data and add it again to the context for the user query and send it back to the LLM again. And then the LLM will have enough information to answer the query. But the LLM will receive a batch of data with the historical information in that period of time, together with the user query. But when it comes to computing the average, it's somehow like an hallucination of the LLM, because the LLM cannot actually compute something, an average. So it will answer something that I'm still not sure how it's possible, but a lot of times it's actually correct. But it's answering based on the knowledge that it was trained. So there is one step further that we can go, which is instead of having multiple functions with the specific name, and then information about the parameters. We can actually just, since we have these context windows that are bigger and bigger, we can just pass all the information, and then to make these computations, we can ask the LLM to generate code that will run with this data as an input, and then we'll provide, for example, in this case, the average that we were looking for, and then pass it again to the LLM, and the LLM will just write a beautiful answer, a user-friendly answer with that value which should be correct, because it was actually computed by code. So when we ask the LLM to generate code, we need to decide which programming language is best for this thing, and in this case, it's actually Python, because of several reasons. One is that it's concise and it's lightweight in syntax, it's very simple, it doesn't have a lot of different symbols that can distract the LLM, and also it's very popular, which is in fact an advantage because there is a lot of code, example code in GitHub that these LLMs are trained with, so they are pretty knowledgeable of how Python is coded. So an example of this code snippet that is generated by the LLM could be like this. I actually took it from the demo that I will show later. And in this case, yeah, it defines a function that takes a Pandas data frame, and then it It gets the day of today, then filters based on, yeah, like filtering the data of today, and then make some computations there, and returns a dictionary in this case. This is decided by the LLM. And then returns this JSON with expected lay departures and lay departures count. And then this context, this will be used as context for the next question to the LLM. So it will know the actual details. So once we get the code snippet, after the first query, basically we need to load the data to run the function, this code, the Python code that was generated. And then, as I said, the result of the code snippet will be added to the user query and send it back to the LLM. So this is more or less how we can, like, enrich or make the answers of the LLMs more accurate by adding contextual information and also adding more accurate computations via code generation. But when it comes to building LLM systems, in Hopworks, we try always to recommend to follow what is called FDI pipeline architecture, which is composed of three different pipelines, the feature training and inference pipeline. The first one is for processing data, so you read from an input, you read raw data from a data source, you compute your features for machine learning, and then that's basically the output of the pipeline. Then you have another one which reads these features and labels, which is what you want to predict, trains a model, and outputs a trained model. And then the last one takes this trained model, takes some input features, and then make the predictions. And it outputs the prediction or the answer. And in HopSource, we have different abstractions that help users with this process of writing and reading and keeping organized these features and labels and also the models. as we will see a bit later as well. So the demo that I was building is focused on building a community virtual assistant that will use public data from the APIs of the public transport in Stockholm. And it will follow also this architecture. So in the feature pipeline, there is a streaming pipeline that is running continuously and is reading events from this API that contains basically departures, so when a bus is expected to come to a specific bus stop. So there is a scheduled time, expected time, if there are delays, if it was canceled, and these kind of things. So this data will come in and be processed in the feature pipeline, also computing aggregations with time windows. And then it outputs features to the feature groups, which is basically this is like a table in a database you can think of where you write this data. And then we have a training pipeline that will train a small XGBoost model to try to predict future delays based on the current status of the departures and the buses. And then the inference pipeline where we have some, so we have deployed the Exibus model and also first I was trying to use Lama 4 because there is a big context window so I so I can send more information, but given the GPU requirements, I had to change to LAMA 3.1, which is the context window is much smaller, so let's see how it goes later on. And then there is some, like, virtual assistant with a land graph, defining a specific graph to basically orchestrate the different functions. So the goal is that you can interact with the assistant and ask, so there are different capabilities in the graph to retrieve historical information of the departures and also to try to estimate if there are delays in the future. So you could ask things like how many late departures there were in the last month, and And then the LLM will generate some Python code to actually compute the count in the last month. So it will filter by month and then compute the count. Also count late departures and it's not here, but also, like, average of delay time and these kind of things. So you basically ask something to the LLM. The LLM generates Python code that satisfies basically the data or the analysis that you want to reach, and then it answers you. So let's see if it can show you here. What's the time? So this is the HubSource UI. I don't need to go too much into detail, but, yes, as I said, I have different feature groups for departures, for example, where we have here different features, like schedule, time, expected time, if it was late or not, deviations, and these kind of things, right? And then the feature view that basically reads these specific features from the feature group. And it will be used later on. But let's go into the different notebooks that they have here. So the first one is just using the Hop Torx Python API that helps you interact with the platform from Python. So it's a Python centric platform. And you can practically do everything from the Python API. So in this case, we are just logging in, getting a handle of the feature store, and creating here different feature groups for users and also for sites. This one is very simple. And then this is the actual streaming pipeline that I was mentioning. So we get the feature groups that we created. And in this case, because I wanted to just use Python, I'm using Quickstreams, which is a Python library for streaming processing. And here I am basically reading from a Kafka topic that I'm sending all the departure events, and I'm computing aggregations, which are based on three different time windows. Let me check here. The first one is 30 minutes, the second one is one hour, and the third one is six hours. And these aggregations basically count the delays that were in the departures, how many buses were late, how many canceled ones, and some other information in different windows. And this information will be used later on for a different model that is trained on these specific features, as we can see here, the different aggregates. And it tries to predict feature delays based on the current status of these aggregations. All right, so, yeah, basically here, I'm just reading the data here, we don't need to go into it, but, yeah, basically train an XGBoost model, I get a handle of the model registry, and then I create the model in the platform, and I save it, and that's pretty much it. And then, on the other hand, I have here another notebook that, again, in this case, stores Lama 3.1 in the platform, and then log in, register the model itself, the model files, and then deploy it using VLLM, which VLLM is a serving engine that is optimized for this type of models. And basically here I deploy the model. So that's basically it in that case. Here I have two models running. The first one is the llama, and then the other one is for predicting late events, late departures. But if we go to the last one, this one is where the graph is created. So I'm going to start running this here. So for the graph, I basically have four different files. Each of them represents one node, and just let's look at the names. One is for asking the LLM to generate code, another one is for evaluating if the context is enough or not. If something fails, because the Python code might be buggy, so it might fail when we try to execute it, then it will go through the query refinement and note, which basically iterates again, asking the LLM to improve the code. And then the final one, which is to actually wrap it in a user-friendly answer and send it back to the user. So those are the different nodes here. And here's where basically we create states and we add the different nodes and, yeah, compile the workflow. So I can't run this one. I don't know if I run it already. Yeah, this one. First, we will try to use the LLM, the Lama 3.1 that is running on VLLM in Hopsworks. We can ask, are there expected lay departures today? So it will ask three times the LLM in this case. So it depends. Sometimes it turns into multiple iterations if it tries to improve the code and so on. But in this case, it was three times. It basically says that I've got the latest information, unfortunately it appears that there are expected delays, and we get the context here that was computed by the Python code, and also imprinting here the actual Python code that was generated, and it's basically computing the percentage of the late departures and expected late departures. And then we can keep going with more questions, like, for example, this one. What is the average? In this case, it should try to compute the average of the data. Let's see if it takes... Yeah. So I would be happy to help you with that. And then some information about the data that we are sending it, the historical data. Then the context, which is the average delays in departures today, and this code. So every time you ask, even if I ask this again, this Python code might change. The context might change. It's based on what the LLM will actually expect. That is needed to answer the user query. So I think I'm running a little bit out of time. So I think I will wrap up here. If you want to learn more about how to build BATs real time on LLM systems, you can scan this code and get this first chapter for free, which is from a book that our CEO is writing on the topic. And it will be released soon. So you can get the first chapter here. And that's pretty much all. Thank you very much. Thank you, everyone, for listening. And if you have any questions, just feel free to ask or join the public Slack for follow-ups and so on. Thank you. So thank you very much. We have one question. Does LengGraph support automatic retries if generated code fails? For example, if it has a syntax error. Yeah. Yeah. So that's when you build a graph, basically, you can I saw one of the notes that was query refinement. So when the code is generated, then another node will try to execute the code. And if it fails, it's actually interesting because when it fails, you can actually cast the error. And then you ask back the LLM to improve the code. But you can provide context with the error itself. So then it might see that there is an import that is missing, for example, the import pandas line. Maybe it was missing. So then it will see, like, import error, or library not found, or whatever. And then it will improve the code based on this feedback as well, based on the exception. So it's just a matter of reiteration. And then at some point you have to say, okay, maximum 15 times or something like that, because Because otherwise it might be an infinite loop. So at some point you want to stop and just say I couldn't manage or something like that. Yeah. Okay. Thank you. Other questions? From the audience. Okay. So you mentioned your CEO wrote, like, a very good book from O'Reilly. Are there other advantages of Hobbes' works? So what they do better than others? So there are, in the website, we have, for example, here in examples, you can go to examples and we have different tutorials on how you can build ML systems or LLM systems using like these best practices that I was mentioning before, depending on your use case, like real time or recommendations or things like that. Or yeah, I think that's one of the sources that you can reach if you're interested in it. Yep. Okay. So thanks again for this nice talk.