Demystifying Agentic AI Using Small Language Models

Speaker 1 [00:00]

Good morning and welcome to this session. I would like to introduce our speaker today, Sehi Sokolenko. He's the co-founder and the CEO of Toa to Dave. He'll be speaking on demystifying agentic AI using small language models. Let's give it up to our speaker.

Speaker 2 [00:27]

Thank you, everyone. I am based in Berlin. Tower is a startup in the Berlin area. We got founded about 18 months ago, but this talk is not about Tower. This talk is about a passion project of mine, understanding all the hype about agentic AI and trying to make it, you know, understanding what we can do with this. About eight months ago, I came across this concept of a small language model, and started experimenting and using Tower for this, and it turns out there's pretty interesting stuff happening in this space. Who of you have heard of small language models? Raise your hand. I would say maybe 60-ish percent. Who of you have actually tried to use it for agentic work? percentage, but maybe 25%. Okay. Nice. There will be some new ideas here for you. Some of the stuff you probably already know. For the rest of the audience, you will learn entirely a new concept of small versus large. Yes, there's a thing called small language models. If you traveled 150 years ago in the United States, if you're one of the travelers during the gold rush, you would see a sign like this advertising Clipper routes to California, promising you riches during the gold rush. If you go to the Bay Area today, you will see different kinds of advertisements advertising another gold rush, the Argenti gold rush. Salesforce will try to sell you the agent force. Some consumer service company will try to sell you AI agents for customer service. Even Postman, the API company, will ask you, are your APIs ready for agents? A smart person, Satya Nadella, CEO of Microsoft, has recently said that AI agents will become the primary way of interacting with computers in the future. So how do we survive this hype, this temporary craziness and madness? And if you take anything away from this talk, it's this phrase of getting your hands dirty. Use the tools from the talk, use the ideas that we'll present on your laptop today, and lean on small language models as a way of experimenting. By the way, if you have questions, I think there is a website to submit your questions to. Please use that if you know the link. I think it's talks.pycon.de. We also have microphones in the audience. And at the end of the talk, I'll try to take several questions. And we'll also talk to you after the talk at the table. Tower has a booth where I will be after this talk. So if you want to talk more about SLMs, about what I do, come downstairs in the sponsor area. A little bit about myself. I worked for several big tech companies, including Google. I worked on a product called Cloud Dataflow in GCP. It's a data processing service, high-scale streaming and batch analytics. I then went to Snowflake, where I broke a few things, including search optimization and metadata. And then I went to Databricks and broke more things, including shipping serverless filtering for Spark clusters and dedicated clusters that teams can use to share GPUs. All very exciting stuff. So as you can see, I spend most of my career either in databases or in data processing. Now, nowadays, I work on Tower with several co-founders and engineers, founding engineers. We're building a Python-native data backbone for your data pipelines and agents. so now that we've got the motivational example out of the picture the temporary madness of humanity the agentic madness let's i wanted to spend the rest of the time motivating how llms became equated with autonomous intelligence and then i wanted to compare large models with small models and see if we can use the smaller ones better for the tasks tasks at hand and then And I'll give you a few tips for using SLMs, how to experiment with them, and how to run agentic flows. In a sense, the reason why we are here where we are with this hype about agents is because of the age-long quest by humanity to create something that is more human than humans. To quote Dr. Eldren Tyrell from the Tyrell Corporation. Who of you have heard talks by Mr. Tyrell? He was, right, I see maybe two hands here. It's a reference to Blade Runner. Okay, so it's a bit of a, it was a test. In 1995, so almost more than 30 years ago, the concept of an agent was already explained in a seminal book by Russell and Norvig. They defined an agent. So it's not a new concept. Agents were defined and existed for a while now, three decades. So Russell Norvig defined an agent as an entity that perceives its environment, makes decisions, and takes actions, and why does it do it? To achieve goals. So there's a goal, there's a goal, and the agent perceives the environment makes decisions, and takes actions. It's very simple, really. Agents are not new. They were classic agents in the 90s, 2000s, 2010s. The most important reasoning or decisioning frameworks for agents are the utility theory and reinforcement learning. If you think of elevators in our buildings, the Odysseus and the Schindler's, They all used some sort of a reinforcement learning mechanism to decide which floor to go first, how to optimize the routes between different elevators, and using a common resource in order to bring the passenger the fastest to the destination. So it's not an entirely new concept. But recently, more recently, I would say five-ish years ago, language models started becoming decisioning engines. So that is new. And this process started with GPT-3. Who of you has used GPT-3 in their lives? All right, maybe 60 or 70 percent. So when GPT-3 was trained, the interesting thing that happened was it was trained on data that contained reasoning-like patterns. So it was trained on Stack Overflow, on coding tutorials, on forum answers. And if you have ever visited Stack Overflow, it's a site where you ask a question and then people start responding to it, explaining their thinking process and giving you answers. So it's a tree-like structure. It's a reasoning structure. Then, a couple of years later, they and other authors have published another very important paper on chain of thought prompting. And their innovation and new idea that they brought in was using intermediate reasoning steps and embedding it into training data. so the way they trained models was they had a prompt and the final answer but they also had intermediate reasoning steps the user wants me to do x in order for me to do x i need to do abc and to do a i need to do one two and three so these are the intermediate reasoning steps and they were part of the training data and so now they ran they use expensive hardware to run these tuples of prompts, intermediate reasoning steps, and final answers to create new models. And this is how reasoning got injected into large language models. So they became our decisioning engines because we used reasoning patterns from GPT-3 training area and intermediate reasoning steps. So now, how does it all work in a simplified form in an agent that uses an LLM to make decisions? A prompt comes in. This is the question from the user. The agent enters a reasoning loop. The reasoning loop divides the prompt into a plan. The plan will contain an execution of multiple tools. The tools can be API calls, they can be analytical database data access patterns, or operational database access patterns. And this loop continues until a token is emitted in one of the outputs. The token is final answer, it's literally final answer, final underscore answer. This is the token that the agent is looking for, at which point the agent will stop, create the final textual output, It will store the input and the output into a thing called memory so that it can use later on, and the loop begins again. Now, some smart folks have realized that large language models contain a lot of useless data to operate this reasoning loop. Does it really matter to know who the Queen of England was in the 1980s in order to make a decision whether to call a data access API? Probably not. The problem with large language models is it contains compressed information from training data. These models are large, which means inference is expensive. And because inference is expensive and the models are large, you cannot experiment with them as easily as you would like. Because they don't fit on the hardware that is readily available to you. A couple of months ago, some researchers from NVIDIA and Georgia Tech published a paper on small language models as the future of agentic AI. what is small there are a couple of definitions small versus large i and a few others prefer a definition that is time-based so in 26 2026 small is well the definition is independent of time definition is whatever fits into a memory of a regular consumer device and this year this happens to be maybe 16 or 32 gigabytes of virtual memory. With this amount of memory, a model that will fit into the virtual RAM will probably have somewhere between 10 to 30 billion parameters. And the consequences of using these models, typically the strengths of small language models are you get more privacy because you can run them on your private hardware, the cost is much lower. I can literally run many of these models on this laptop, which is already two years old. I already depreciated it. It costs me zero. However, the problem with the SLMs is it doesn't know who the queen of England was in 1800s. So there's some hallucination. If I do ask who was the queen of England, it will probably fake me some name out of its waiting model. Now, the authors of the paper have made three statements. Those were the three main statements of the paper. Statement number one is that today SLMs are now sufficiently powerful to handle the demands of agentic workflows. which is a qualified statement, it means they didn't claim that they were as good as Opus 4.6 for generic tasks. They only said for agentic workflows, these SLMs are good enough. And because they are small, they are more flexible in where I deploy them, and they're more economical. I can deploy them on local hardware. I can deploy them on really cheap NVIDIA GPUs in the cloud. cost becomes less of an issue. So what is important for agentic workflows? There are four real factors that are important. Four important capabilities of language models as they relate to use in agents. Number one is the ability to break down a task into subtasks. This is what reasoning is. The first task is always the prompt. You break it down into subtasks, and then you continue and, yeah, you break it down in subtasks. And you keep going the subtasks into smaller tasks. The second important quality is tool calling, the ability to initiate pass parameters and format your output. Quality number three is for the use cases where you don't have the tools to solve a problem, you need to generate some glue code so the ability to generate glue code is important and lastly the ability to follow guardrails instruction following that's also important now there's a table comparison table in the paper which i'm not going to read you line by line but the main idea of this table is to say that for agentic the the four tasks that are important for agents, the small language models of a size X are now as good as large language models of 10 times that size. Which is pretty awesome, if you ask me. Who of you has heard of the Berkeley function calling leaderboard? A few of you. So this is a leaderboard by UC Berkeley. They keep track of about 110 different model families. All the important ones are there. GLM, Clot, Quan, Meta, everything. Vendors kind of supply the execution and benchmark runs, and there's verification going on as well. As of December 25, so last year, December 25, I need to check if there's a newer version. They typically do updates every three months. As of December, the top models for these four important qualities were still the usual suspects, Claude Opus 4.5, Gemini 3, GLM 4.6, but they're all proprietary. So you cannot use them for local experimentation. You cannot use them for experimentation in your environment. now in positions kind of 10 to 20 you're beginning to see open source models so you see things like kimi 2 deep seek 3.2 and a model from salesforce on position 18 xlam 2 and this is the first model that is not only open source but also smallish so it has 32 billion parameters and it provides overall accuracy that is comparable to state of the art. Remember, there are 110 models in this leaderboard, and in the top 20, you have an open source small model. This is actually the model that we're also going to use for our experimentation here in our little demo recording. Those of you who want to understand how this model was created, two things. I'm going to speed up a little bit my talk they used a synthetic data set with function calls and they they use the process called supervised fine tuning to create the the final model so you can actually do it yourself if you generate synthetic data and use this process so i will now ready to learn how to survive the agentic ai hype you'll have to science the shit out of it as as per movie the martian for this setup we'll need five things we'll need a small language model and we recommend the salesforce xlamb2 because it's open source it's small you can run it today we used a particular quantization of this model that actually fits 16 gigabytes of ram it's the 4-bit quantization version we'll need a inference server or service for local inference we recommend Lama CPP. There are reasons why we do this. I'll explain it later. For remote inference, once you're ready to move your workload to maybe a cloud production, there are services like Together AI, Hugging Face. We like Together AI and Hugging Face. You might also want to consider a runtime service to run your Python code. There are a couple of examples. I work on Tower, but you can also use Model or Fly.io. And I recommend using a framework for a Gentic a genetic framework long chain is a good one i personally prefer that one there's also llama index and others why do we recommend llama cpp this is more a little bit for folks who use apple silicon and like simplicity there are several good inference servers local inference servers but llama actually is able to use local gpus on apple whereas vllm for example does not And second good reason to use Lama CPP is it uses very simple Hugging Face model naming conventions, which Lama doesn't follow. So how does a Python runtime help you in your work? Well, it helps you take any Python that you write on your laptop and then package it up into an application and test it locally, but then ship and run in production remotely, maybe in your cloud or in their cloud. That's what Tower does. Some of these runtimes have a self-hosted version, such as Tower does have one, and they usually offer you an orchestrator for control flows. What else can you do with Tower? Well, you can do boring but necessary things like feature transformations or ETL. You can run ingestion frameworks such as DLT on Tower. You can move data into analytical storage based on Apache Iceberg. You can do transformations, transformation jobs using DBT core or pollers or other libraries. And you can even run your UI on Tower as well, including Marimo notebooks or long chain agents. A little bit about Iceberg. Who of you has heard of Iceberg, Apache Iceberg? Okay, fairly popular technology here. The reason why Iceberg becomes important with agents is because two years ago, some lawyers in New York State made a terrible mistake of using the normal GPT to completely fake a legal brief. They actually submitted this brief to the court and got burned because many of the case numbers mentioned in this brief were hallucinated. They were hallucinated because GPT didn't have access to real legal cases. So since that time, people realized we have to give agents access to real business data. And Iceberg has very good properties on scaling and performance and accessibility that allows both inference engines as well as data warehouses access a single data set stored in public storage. So how do we do experimentation with the setup? You'll first install Lama CPP. You'll start running a model, the XLAM2 model. And you'll write your agent using long chain and maybe perhaps use Tower to debug and run it in production. For my demo, I took an example of an agent that retrieves stock information. But I made it a little bit more complex, and I said, look, there are external stock APIs, for example, Yahoo Finance, where the agent can take this data. However, I don't want to consistently or constantly go to this external API. I want to be able to cache my data in a database. So this agent will make decisions. If the data is already in a database, it will take it from the database. Lakehouse, Apache, Iceberg Lakehouse. if the data is not there, it will go to the external API. The interesting piece about agents is how we define what they do, the business rules. So we don't do our typical airflow DAGs anymore. We write business rules. We write things like you can take stock data from an external source, but you should probably take it from a cached source, from a database, if you can, if it exists. And this is the preferred way of doing this. So we can now define our flows not in graphs, but perhaps in business rules. Some of you might ask, well, how does agents decide which tools to call? It's very interesting because if you use long chain, it uses information in your code, including names of functions, including the arguments you have, and also including even your doc strings to feed it into the decisioning process and determine which tool is the best one to accomplish a task. All right, so I have a quick recording here. I have about two minutes to go through. Of course, I use Claude to develop my pipeline. And this pipeline looks like this. I basically ask Claude to develop a tower application that is a data agent that answers stock prices, and the stock prices should be for a particular set of stock tickers, and there's a time range, and I'm just looking for the biggest volume of stock trades. So what Cloud will do, it will first learn, based on examples that I provided to it, code examples, how to write tower applications. And once it knows what the structure of Tower Applications is, it will start generating my new application, which is a data agent. It uses the Tower MCP server. The Tower MCP server provides basic commands like create an app, create a configuration file, add parameters, remove parameters. Here you're looking at the business rules that I previously defined in the prompt that I gave to Cloud. These were the business rules I wanted to follow. So Cloud inserted it as a string into my agent. And what will happen is I want to show you kind of the process to deploy this agent into cloud production. I will probably take another minute. Sorry, guys, I know I'm running a bit over time. Tower accepts any Python code. You can literally give us your existing Python code and we'll run it. The only ask we have is we need a config file to know what the name of the app is and whether you have any parameters. So this is what we call a Tower file. But everything else from your Python project you can throw over to us as is. I'm just showing you how it looks in Cursor. Here's the long chain agent executor that will be started. And we're getting to the end of the reveal, as they say in Magic and the magicians. All right, so there's some validation happening. The app is almost done. And now it's being deployed to our production environment. using the MCP server, and I will share something with you that you will see later in the demo. The first version of the app will kind of work, but not really, and this is something that you will see a lot in your development of agents using other agents, AI assistants. The first versions will probably fail. In this particular reason, it will be because a dependency is not installed or it's the wrong dependency but runtime services the python runtime services like tower are able to feed production logs back into cloud so that cloud can learn and adjust and this is what is happening right now the logs from tower are being routed into cloud the dependency will be modified in pyproject.oml and the second version of the app the agent will actually be successful So you will see how it all succeeds. I'm going to speed up a little bit. You'll have to trust me on the successful execution of the second app. So hopefully I'm at the end of my talk. Happy to take questions maybe later or even at the table before the next speaker comes. Hopefully you were able to see that experimentation is possible. this is the recommended way of learning about agentic workflows. Small can be powerful. Use local setups. Use open source models. You don't always have to use Airflow DAGs to define your workflows. You can actually start writing your jobs using guardrails and freeform text. Would like to connect with me a bit later. I will be at the booth, tower booth, downstairs. As well, you can take a qr code pic and connect with me on linkedin thank you for attending this talk appreciate it

Speaker 1 [28:39]

We'll take maybe two questions from the chat. So let's say we have a scenario for a multi-agentic workflow. Does each agent need to have its own SLM? In the case that each agent does a specified task, would that be more efficient compared to using a single LLM?

Speaker 2 [29:01]

Well, you wouldn't use a custom SLM per task. It's totally fine to use existing SLMs, such as the XLM2 from Salesforce, and just modify your prompt and your business rules. Works totally well. You might want to maybe invest later on in fine-tuning a model and actually create your custom copy of a SLM for your task based on synthetic data that you will generate and run through a model generation process. But you can start with existing versions of SLMs.

Speaker 1 [29:40]

Okay. For selecting SLM, what would you say a user should prefer? High parameter with low quantization or higher quantization with lower number of parameters? And why?

Speaker 2 [29:55]

Right, so this is kind of a trade-off you'll have to make. Remember, I recommended a 32 billion parameter XLAM2 model with 4-bit quantization. I think the 8 billion parameters didn't really work for us. They are still stupid and dumb. But kind of starting with 30 billion parameters, things get interesting. For agentic workflows, they start getting really good at reasoning loops. They terminate correctly. They don't run infinite loops. One of the interesting things you'll encounter, these agentic loops can never terminate. They will continue running because the SLMs are incentivized to call as many tools as they can. This is an interesting fact that you will learn. I would say starting with 30 billion parameters, 4-bit quantization is great. It will fit your consumer device. If you can use a 70 billion parameter model, you'll get slightly more accuracy.

Speaker 1 [30:57]

Do we have anyone in the audience that would like to ask a question? We can pass a mic.

Speaker 2 [31:08]

A quick question to the leaderboard you showed, right? So 70% was like the highest number, 70-ish. Yep. That does not mean that... 70% of questions are correctly answered. It does, yeah. Sorry, I'm leading.

Speaker 1 [31:26]

No, no, you, you, you, you guessed my question, right?

Speaker 2 [31:26]

No, no, you... Because if 7 out of 10 requests were like garbage, agentic workflows would be like garbage too, right?

Speaker 1 [31:28]

Because if.

Speaker 2 [31:33]

It's a combined score of four other scores. You can actually, if you go to this leaderboard, you'll see individual scores. The numbers here, the 77.47, doesn't mean that 23% of your questions will be garbage, answered wrongly. It's a combined score of four other scores. Think of it as an index. it's just the index and accuracy is the wrong name for it it should be called overall index yep all right I'm happy to take maybe questions at this table to give the next speaker a chance to set up I will also be available downstairs close to the more plenary for more questions