Building Serverless Python AI skills as WASM components

Speaker 1 [00:08]

Thanks very much. So hi everyone. A lot of people are building like cool gen AI demos with frameworks like Lungchain or Lama Index, but then getting these demos into production is where a lot of people struggle. And about a year ago, my team made two strategic bets to help teams bring AI into production faster. And the first bet was that we believe serverless principles can abstract away some of the complexity that people have in operating I use cases and the second bet we made was that we believe WebAssembly might be a good fit and especially the WebAssembly component model to build this serverless platform and today I want to share the story why we made these bets and I want to share some progress on how these bets turned out and I hope to also convince you that WebAssembly is like a very cool technology and maybe to spark your interest so you might try it out in your own patent projects my name is moritz i work as a software engineer at alfalfa and what we do at alfalfa is we're building fairy eye which is a software stack for enterprise and governments which they can operate in their own data centers and they can develop and operate their own generative ai use cases with the help of Feria. The plan for today is that I want to start by looking at some of the challenges which AI engineers are facing in bringing AI into production. Then we'll take a closer look at WebAssembly and specifically the WebAssembly interface type language, short WIT. Based on this, I'll introduce Feria skill, which is the Python SDK my team is developing. And in the end, I want to spend some time to look at next steps, which would be more agentic-like workflows, and we'll also see how this connects to the MCP protocol from a topic. So what are the challenges that AI engineers have in bringing AI into production? So, the user persona we are considering the most in my team is the AI engineer, and this is not someone writing CUDA kernels, but basically a person that uses AI and generative AI to solve actual business problems. And suppose they want to enable some knowledge worker with a RAC use case, and they've gone ahead and implemented some methodology. They sort of iterate a bit on the retrieval strategy and on the prompting. They evaluated it on some data set, and now they want to take it into production. And now they need to think about questions like how does my use case scale, how safe is it, how can it be monitored, about authentication, about the web framework to use, the OS to use, about the error handling you want to do when GPUs are busy. So a lot of things that you then need to think about. And as companies want to take like tens or even hundreds of these AI use cases into production, these questions really become blockers for adoption of generative AI. And we consider a lot of these challenges to be accidental complexity. So these are not the questions the AI engineer would want to think about, but rather the questions they are forced to think about because we don't live in an ideal world. So we live in a world where compute is limited and where things can go wrong. So we need to spend a lot of time thinking about this, where actually what we would much rather want to do is to help the AI engineers to focus on just the essential complexity of their problem so they can spend most of their time to really find the best methodology to choose the right model, to choose the right prompting strategy, the right indexing strategy in their vector database. So how can we move the scope for the stuff that the AI engineer needs to worry about from basically all of this to just this? And how can we move operational complexity from the user code to basically the host. And I think to be fair, everyone who's an engineer would really want to focus on just the essential complexity of their problem, and technologies and abstractions like containers and mature web frameworks, like fast API, are well-working abstractions. So what is different for AI use cases compared to normal software development? And just by looking at many AI use cases, What we see is actually many similarities so like all of them need to think about authentication all of them need to connect to inference most of them if they do like some retrieval need to connect to some vector database they all have like similar concurrency and error patterns and this is where we believe some responsibility can be moved to a dedicated service or runtime. And now this runtime, which we would want to develop, would need to be extensible in some way, so an AI engineer can configure or extend it to their concrete problems. And there's different levels on how you can think about this extensibility. And maybe the most basic level for extending a system would be to say, well, I have a service running. Maybe this is just exposing some chat interface of some assistant. and then you could configure it like with your own system prompt so you'd like sort of bring your own system prompt or you could configure it with one of multiple retrieval strategies and this is like very easy to build maybe but it's also very constrained in in what you can do is like the person who wants to configure it and then so this would be configuration and then like lang graph takes us a bit further right by introducing a bit of a declarative very domain specific language where you can build certain artifacts which you can then deploy and depending on your use case this is an abstraction which can work really well because it takes you to like 80% really fast but it's also this kind of abstractions where users tell us that it might work very well to start but then you spend like days or reverse engineering the framework because there's just this one thing which you cannot do and then you need to like sort of work around this and we believe that ultimately AI applications and agents are just software so like why would you want to write them like very like different you could just write them like in software in like the way you used to writing stuff which could be Python so we don't really want to invent this like a new domain specific language so what we're aiming for is like for some system where the engineer would write Python to basically solve their business logic and they would then we would have this one runtime which basically takes the Python code from the user and runs it for them and takes care of things like monitoring and scaling and this brings us to Web Assembly how many of you have heard of Web Assembly before okay so the majority which is good so I mean it's like an instruction format and then like a compilation target from different languages and WebAssembly programs are organized into modules which is like this core unit of development and compilation and then modules contain certain functions and types and memories but modules are a bit limited in the kind of way how they export and import functionality, they are limited to basically only functions which take and return very basic primitive WebAssembly core types which is essentially just numbers. So if you want to compose two WebAssembly modules which are written in different languages, they might have a very different understanding of how strings or more complicated data types are represented. And this is where the WebAssembly component model comes into play, because WebAssembly components basically agree on the format how they exchange information and how they share information across their boundaries. And the way they do this is by specifying their imports and exports in a language called web, so WebAssembly interface type. And you can sort of think of WIT as like a universal language contract between these components where they define their binary format on how they exchange information. And this is like a very scoped-on version of the wit file we are using in our deployment. So what you can see is that this file is composed of different things, of worlds and interfaces. And then there is some definitions in these interfaces of types and functions. And a world is like a very high-level concept. And a world basically explains what a component imports and what it exports. So the imports are basically the component telling someone, hey, this is what I expect to be available to be able to run, and then the export is the functionality I myself am exporting. And you can see for our world skills, so skill is the term we use in our team to basically refer to this AI methodology. We include this other world, the CSI world, which we'll get to in a second. And then in the CSI world, we have the four different imports listed. is chunking, then document index, which is the vector database we maintain at AlphaAlpha. We import an inference module, which allows us to talk to large language models, and also like a language module to detect languages. And then what we export in the skill word is just a single interface called a skill handler, which exposes a single run function. This function takes as input a list of bytes, and then also returns a list of bytes, and And then we agree on with the host that this is like a JSON representation. And this run function is then how the host who runs this component would be able to use the functionality that the component defines. And CSI stands for cognitive system interface. So this is the terminology we have come up in our team to basically describe the interface that you would want to use if you're writing AI methodology and the stuff you would want to have available to interact with large language models. Yeah, so this is just having a closer look at the inference interface. This is again a bit of a scope down version and what we see here is we define this message record and then a message in a conversation with an LLM has a role and a content, we define a finish reason enum, why the model stopped generating. And then what we see here is this chat function, where basically the guest component could specify the model and a list of messages, and then it would get back a chat response. And one thing to notice here is in this interface, when executing the chat function, there is no concept of authentication. And the reason for this is because this authentication token never actually like leaks into the code that an AI engineer would write. Because what we can do is in our service, which we call the Feria kernel, which we have running, which is the host site implementation. Once we have an incoming request, we can authenticate this in our service. And then we call the guest component, which never needs to know about this whole authentication. And then whenever we do an outgoing request on behalf of this component, for example, if we resolve a call to the chat function, we can actually take the user token, the incoming token, and use it to authenticate against, for example, inference or the vector database. And this decoupling of interface and implementation also means that even for existing components, which someone wrote maybe two or three months ago, we can still improve them because we can iterate on the host side. And one example for this is a feature which we've been working on in the last weeks, which is a more in-depth tracing from the host side, because we know that all the important interaction with it, which an AI engineer would be interested to see, they all go through our cognitive system interface and through the WIT world. So we can just do host side tracing, and the AI engineer, again, does not actually need to know about it. So I'm in this Python conference. SO MAYBE IT'S A GOOD TIME TO ALSO HAVE A LOOK AT SOME PYTHON CODE. SO LET'S TRY TO UNDERSTAND HOW YOU WOULD BUILD THESE COMPONENTS OR SKILLS IN PYTHON AND HOW OUR SDK SKILL WORKS. THERE'S A VERY COOL NONPROFIT ORGANIZATION CALLED THE BYTE CODE ALLIANCE AND THEY DO A LOT OF COOL PROJECTS AROUND WEB ASSEMBLY AND WEB ASSEMBLY SYSTEM INTERFACE AND ONE OF THEIR PROJECTS IS A TOOL is a tool called componentized Py, which can basically package a single Python app as a WebAssembly component. And what it essentially does, and this is a very slimmed-down version, because it does a lot of things under the hood, is it takes in a Witworld on the left-hand side, which we also have available in our runtime and in our SDK, and it takes in some Python code, which is written against this Witworld, and then it'll produce a WebAssembly component. And you actually have, like, in this WebAssembly component, you'll find the entire Python interpreter built against VasiLibc. So this is how we enable the build step from our SDK. But, like, what interface, like, what's in app.py? Like, what interface do you write your Python code against? So you can also use componentized py to create bindings for a particular world. and then you can write your code against these bindings for you so for just like the inference interface which we just had a look at this would be the auto generated bindings and then in our team like we always have this example if we just want to do a quick demo or something of a very simple skill which doesn't do much it just takes an incoming topic and then it asks an LM to generate a haiku about this topic so how would like this methodology would like look like written against these bindings so what we do here is directly from the generated bindings we actually import exports and imports and then what we build is what component has pi expect which is like the skill handler interface which has the run method which we need to implement and by doing this this component as pi, if you point it to this module, can actually pick up on this and then build the WebAssembly component. And there's not too much happening here, so we're just getting some input bytes, and then because we know, because we agree upon with the host that this is JSON, we load it into a pidentic model, we're generating a prompt, and then we're actually constructing a chat request against the LAMA model, and then we're executing this chat request and dumping the the result back to JSON bytes again. And this chat function will then actually get resolved by ComponentSpy to use the function which we then expect from the host site once we run this in our runtime. But there's one problem with this, and it is that it's pretty hard to test, because you can build this, and then you can push it somewhere, and our runtime could run it, But you cannot really run it locally on your machine, because you would just raise the non-implemented error, which we just saw in the bindings before. So there is some decoupling needed here. And then what we did is, in our SDK, in order to decouple things, we put all these functionality from the bindings just behind a protocol, which is just the CSI protocol here, which does not contain any implementation. And then we have one concrete instance of this CSI protocol, which is with CSI, which then actually resolves to, again, invoking the bindings. And now this means that AI engineers in Python, they can just write their logic against this protocol. And on the next slide, we'll have a look how, in our SDK, the same methodology which we just saw would look like if written in our SDK. So this is the same logic as before, and there's a couple of things which have changed. So now we're importing stuff from our SDK, which is actually on GitHub, and we're defining the input and output as pydentic models, and then we're decorating our function with the skill decorator, and we're taking an instance of the CSI as the first argument. And then the inner part actually stays the same. We're constructing a user message, so the prompt, we're asking the model to generate a haiku, we're executing the chat request, and then we are returning the output. And we also see there is some kind of dependency injection going on, so similar maybe to how fast API does it. We don't actually get to see the bytes in input and output, but we just interact with the Pydantic model. And then we never go against any bindings here. But what is happening under the hood? How does this work with the skill decorator? So this is the decorator itself, which just takes in a single function and then returns another function again. And the main thing to notice here is that we don't actually modify the original function which the user provides in any way. We just return it. We don't do anything with it. What we do is, and this is like some stuff is a bit simplified, we inspect the signature of the function and then we construct this skill handler in here at interpretation time and insert it into the globals of the function so componentized pi can pick up on it. And then you see in line 10, like we're doing this dependency injection where we're loading the incoming bytes into the pidentic model. And then just in here in line 11, we're actually like inserting the width CSI, which then results to calls against the original bindings, which componentized pi will then resolve into the function and calling it with it. And then we're dumping the results to bytes again. But we don't do anything with the original function. So like this whole WebAssembly concept is sort of encapsulated in this decorator here. And what this means is that testing actually becomes kind of nice in Python because, like, we just need to provide a different version of the CSI, which kind of works in Python, and you can actually test your code. And this is how you would do this for this example. So this is something you could run with PyTest where we're just constructing some input. We're asking the model to generate a Heiko about alt milk, and then we're just asserting that creamy is in the result. and what we're using here is in our SDK we have this like testing helper which we call the dev CSI and this is a pure Python implementation of the CSI protocol and what it does is it actually goes against a running instance of our runtime against the Ferrier kernel because we also offer this functionality via HTTP so basically whenever I do, like when I run this what's happening is I'm just like executing all the Python code and then like when I'm doing the chat request The dev CSI is doing an HTTP request to the kernel, and then the kernel is, like, our runtime is actually doing the inference request and returning the result to our code here. So the nice thing about this is I can just, like, write my methodology just in pure Python. I can use all the tooling I like in Python, my debugger or whatever, and I never leave this ecosystem. And also what we noticed, what people like about this is, like, this decoupling of this This whole interface also allows to write nicer tests, because maybe sometimes you just want to test some control flow in your AI logic or business logic, and you don't even care if it goes against a real model. Maybe you just want to mock it out. So we also provide just a stub CSI, which is just a doobly, which you can load up with your own expectations, and then you can basically adapt it to just however you want to test it. So whoever likes to do stuff test driven, this is kind of like a nice abstraction. So now that we know how we can build stuff in Python and how we can test this, how do we publish it? How do we get it from our machine actually to the runtime? So containers are the standard way for packaging and distributing applications. And the Open Container Initiative defines the standard for container formats. And there is actually also a standard on how you can publish just WebAssembly as OCI-compliant artifacts and publish it to OCI registries. And this is what we do in our SDK. So basically what we expose there is for the build step, we just rep componentize Py. So in our SDK, you'll just do uvrun faryaski build and point it to your Python module, and it will build the WebAssembly component for you. And then once you've done this, you can actually just run FeriaScale publish haiku.wasm, and then you configure a registry in your environment. And this will publish the OCI artifacts to there. And then our runtime can be configured to sort of listen to certain registries and pick up the component from there. And this means you can actually develop, test, build, and publish your artifacts very, very quickly yeah maybe it's time to like take a step back and look what we've done here because like like at the abstraction we've created because what we've done is we've created a very opinionated way of how you like write your methodology in in Python you can do it in pure Python and you can express your logic but you're also very very constrained in what you can do because basically all your interactions with the outside world are limited to this CSI interface which the host provides to you you cannot do more you are not even allowed to do up on HTTP we could enable this but we have not enabled this so it's a very constrained world but like operating in this very constrained world actually allows us to provide value because we can like build nice abstractions on top of it and there's multiple examples which we can can have a look at we've talked about the tracing before because we know everything is going through this interface, we can sort of do stuff with it. And another example is error handling. Because going against inference, there's a lot of stuff which can go wrong. GPUs can be busy. Models cannot be available. And there's errors which are recoverable and unrecoverable. And what we can do is we can actually, in the runtime set, decide if we believe an error is recoverable and retry stuff, like any HTTP request. But we can also, like, whenever we are encountering something while, like, resolving, like, a CSI request and we believe this error is not recoverable, we can also stop skill execution and give a nice error to the person which was doing the initial HTTP request against our runtime. And another good example for this is actually streaming, which is a feature we also introduced recently. and as a skill author we provide you some way in the CSI how you can return intermediate results in these chunks but you don't get to decide how these events look like in how these service end events look like in outgoing HTTP, you can just define the payload, so we're taking some assumptions here for you, but this also means for every skill which is deployed to a runtime which does streaming, any UI developer going against this will know in advance what the format is for the messages they will get. They will know what the error format is for, like error messages they could get. So you can actually start building your eyes and you know they will be compatible with certain types of skills. And if we look to more agentic-like patterns where, for example, in stuff like deep research and chat GPT, you want to know what your agent is doing. Is it searching the internet and this kind of stuff? And what we can do, because all of these things actually go through in the CSI interface, we can also tell the UI about what's going on without the AI engineer actually needing to think about this in the skill code itself. So this brings us where we want to go next with this. And the topics we are very excited to work on next, which is agents and MCP. So LMs become really useful if they can talk to the outside world in some way and do stuff. And one example for this is searching the internet via Brave Search, for example. And currently, if we would want to add this functionality, we would need to do an update to the Witworld. So it's kind of an expensive operation. So we cannot do this for all the tools we'd like to support. But there's actually one difference between all the stuff we have currently in our CSI and then tools. Because the stuff we offer in our CSI, we believe this is stuff that should be application controlled, that you would want to use in your code. Whereas tools, we sort of think more of it of something which an LLM would like decide to want to use at some point. So they are model controlled. And this means they could also just be more expressed in a more generic way in natural language. We don't necessarily need this whole type safety, but we could just imagine this kind of like tool interface where we define what a tool is. It's basically just a name and a description, some parameters. And then the runtime, like a guest component could ask the runtime, hey, what tools are available? What can you give me? And it could, like, if the model wants to do it, if the skill wants to do a tool call, and originally the model, it could then offer a function to actually do this tool call. And for these tools in particular, we believe this idea of a runtime can provide a lot of value. Because if you imagine, like, a world with, like, many AI applications which just run standalone and many tools, like, how do you apply, like, governance and observability here? Like, how do you track which application is invoking which tool? Maybe for some tools, because they're, like, kind of, like, operations which cannot be undone, some delete, you actually want some, like, human in the loop interaction. So you sort of want to make sure that the tool only gets called after a human has approved it, maybe like some application, even if it's like acting on behalf of a certain user, which only have access to like some subsets of tools. So like actually like by having this runtime as a layer of like governance and observability where like all these tool calls go through it, we can sort of, we believe we can like solve a lot of these problems. Because we know for sure that the runtime sort of observes what's going on. But, like, how would the runtime know, like, where all of these tools live? Like, we could start hard coding them in our, like, code base, right? We could just go ahead in our, like, Rust runtime and implement Brave Search, for example. But, like, we're developing an on-premise product, which customers would operate on-premise. And then they, like, probably have tools which they want to use, which we don't know about. And it just doesn't scale if we just implement it for them and hard code it. So we kind of want it to be extensible in some way. And this is where we're actually very excited about the MCP protocol by Entropiq. Because it provides a more generic way to think about tools. Because we have this definition of MCP servers, which expose certain functionalities, which can tell someone, hey, this is the tools I have available. And then they can actually execute tool calls. And we also have MCP clients which can basically interact with these servers and the way we're thinking about this is that our runtime will become like an MCP client which you can which a customer could configure and connect to certain MCP servers and then like skills which are written in Python and executed as WebAssembly could like go through through wit and through the cognitive system interface to like then actually ask runtime to resolve these tool calls. This is what we're very excited about. So, yeah, I hope this has given you, like, some inspiration on, like, what you can do in WebAssembly, maybe some of the challenges. We've been using it in production for, like, a year now. We're, like, very happy. Thank you so much for listening. If you want to chat, come around to our booth. And we are also hiring in my team, but also in other teams that I'll have offered. Thank you. Any questions? So the questions were supposed to be posted in Slido, but I can't see any questions right now. So you can raise your hands if anyone has questions, and we can pass the microphone. I saw two hands. Thanks for your talk, but how do you deploy the MCP? How do you manage all the servers and the old connections during production, especially in real life? Because most of the time, I've tried a lot of MCP and a lot of things. Like we have agent to agent now from Microsoft or from Google. But then how do you really deploy these things? Because they work, because they have several servers communicating together, and you need to manage that everything is working. Sometimes in entropy, you have to start the entropy server before you can use it. So how do you deploy things? So the question is, how do you deploy the MCP servers? Not the runtime itself, but the MCP servers? Exactly. This is like a question we have not, like this is not decided, we have not figured out fully. It's an interesting question because a lot of these MCP demos which people are using at the moment it's just like running locally and then it just it's opening a sub process or something so people are still trying to figure out authentication. So we basically think of like in two directions either like you actually add like some authentication on top or you like look deeper what is emerging there in the next months regarding authentication. authentication or you basically say this is running like because our whole stack is running in kubernetes anyway like in a cluster and then you could just like spin up the customer could just spin up the servers in there and not expose them to the outside world but just like within the prop like within the cluster and then this would maybe like make the authentication problem easier but it's definitely like a very interesting question to like understand and we haven't like fully figured it out. Okay. Then maybe my question, how do you, like you said, you have this dev CSI component, but you always need to make HTTP requests to your API, even if dev mode. So, if I'm running my tests locally on a server, I always need to contact your servers in order to run the LLMs, or is there any mode for also local LLMs running that? Because it's a big problem if I have like a CI pipeline and my Internet connection is gone, then my tests fail all the time because I cannot get new servers and all this stuff. So right now, the Dev CSI implementation, you always need a running version of the kernel, so you need internet connection. But I guess you could also think about having an implementation where you do local inference or something, but then you would need to have a way of ensuring that these models are available locally. like the stuff in the CSI is not only limited to like doing inference calls against model but it could also be like some lookup in like our vector like database and then this like becomes like very difficult locally which is like and like we spend a lot of time thinking about exactly this like how can we do testing nice and we found actually like this middle ground where we ensure that the like logic is running locally but then these like calls to inference and something get resolved over HTTP that this is actually like fine for most people but yeah correct you need internet for this. I think that would be the last question because coffee break time is, you can ask the question and then we will do the break. Thanks. I'll make it short. Thanks for the nice talk. I think the solutions are very elegant and clearly the solution is very attractive features. I think in some cases I wasn't entirely sure how, for example, all of this interface stuff, which I think is very elegant and seems convincing, how that points to WebAssembly. Like, if I had seen this talk and I didn't know it was about WebAssembly, it would have been hard for me to come up with, like, what's the actual link? Is it just because WebAssembly comes with these interfaces built in, or the infrastructure for these interfaces? Why couldn't I just have written this in Go, C++, Rust, like, why do I, why WebAssembly? So the reason, like, where WebAssembly comes in is basically just for the deployment part. So for all the development, for everything you do locally for testing it, this, like, just works without WebAssembly. So WebAssembly is just our way of saying how can we take something the user writes and then deploy it to runtime. But without actually, like, we don't want to ship, like, something where the user also provides, like, their own web server or their own OS. but we want to have something which we can talk to on a binary level where we have this very granular host-guest interaction. I don't know if this answers your question, but we can also chat at our booth. Cool. Thanks very much.