State of In-Browser ML: WebAssembly, WebGPU, and the Modern Stack
In-browser machine learning leverages WebAssembly (Wasm) and WebGPU to execute code and models on the client side, reducing server costs and improving user privacy. WebAssembly provides a binary instruction format that allows near-native execution speeds in all major browsers. Because Wasm lacks a standard library for system-level tasks, toolchains like Emscripten are used to compile C/C++ code into Wasm, providing necessary runtime layers and virtual file systems.
Python integration in the browser is primarily achieved through Pyodide, a port of CPython to WebAssembly. Pyodide allows the installation of pure Python packages via micropip and provides pre-compiled builds for libraries with native extensions, such as NumPy and Pandas, through the Pyodide package index. For developers seeking higher-level abstractions, PyScript enables Python logic to be embedded directly in HTML. Alternatively, MicroPython can be used for faster startup times and smaller bundle sizes (under 300 KB), though it supports fewer features and packages than CPython.
Model inference is handled separately from the Python interpreter to avoid overhead and enable GPU acceleration. WebGPU allows for general-purpose compute, moving beyond the graphical limitations of WebGL. The ONNX Runtime Web serves as a universal adapter, executing models converted to the ONNX format on both CPUs and GPUs. For large language models (LLMs), WebLLM provides GPU-accelerated inference, while vLlama enables the execution of GGUF-format models on the CPU via Llama.cpp. While Wasm is currently limited to 32-bit addressing (capping RAM at 4 GB), these tools collectively enable the deployment of models up to 3 billion parameters directly in the browser.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track PyData & Scientific Libraries Stack and was classified suitable for intermediate domain / intermediate python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
Over the last few years, the tooling has matured enough to make "ML in a tab" worth taking seriously. Today, you can execute Python code in a sandboxed environment, ship interactive demos as a single URL, and even run LLM inference entirely on-device, without installations, servers, or sending data anywhere. In this talk, we will give a practical overview of the current in-browser ML stack, focusing on what is realistically possible today and the practical limits you still have to design around.
We will start with interactive environments such as JupyterLite and explain how they work under the hood via Pyodide: what it means to run CPython compiled to WebAssembly, how the filesystem and networking model differ from "normal" Python, and what that implies for performance, I/O, and package support.
We will then move from notebooks to applications with PyScript, showing how the same building blocks can be used to create shareable browser-based tools. We will also briefly cover the lower-level approach: using Pyodide directly and orchestrating it with JavaScript for granular control over loading, packaging, and data interchange.
Finally, we will cover in-browser inference workflows for both traditional and deep learning models (via ONNX), and LLMs (via wllama and WebLLM), and discuss how WebGPU can accelerate these pipelines.
By the end of the talk, attendees will have a clear overview of the in-browser ML ecosystem and the practical intuition to decide whether it's the right choice for your next project.
Target Audience: This talk can be relevant for a broad audience. However, at least intermediate knowledge of ML / familiarity with Python ML ecosystem is required.
Outline:
- Introduction + Motivating examples [4 min]
- Running Python in WebAssembly [6 min]
- Overview of Pyodide [2 min]
- Package management [3 min]
- Runtime and memory constraints [1 min]
- Overview of interactive dev environments / JupyterLite [4 min]
- Building applications with PyScript and direct Pyodide bindings [7 min]
- On-device ML inference using ONNX, WebGPU, WebLLM, and wllama [5 min]
- Q&A [4 min]
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:34]
Hello, everyone. Welcome to Platinum Room. We are shortly going to start the talk from Oleg and Irina. Please have your seats. We'll just wait probably a minute more so that we are just punctual. By the way, if you haven't already signed up for Lightning Talks, I really encourage you to do that. The Lightning Talk booth is somewhere on, I think, on the second floor. and we'll have lighting talks at the end of the day, of course. Cool. Please give a huge round of applause for Oleg and Irina who are going to talk about WebAssembly and WebGPU today. Handing over the mic to them.
Speaker 2 [01:26]
Yes, good morning, everyone. Thanks for coming. For the next 25 minutes or so, we're going to talk about how to run Python and ML models directly in the browser. If you'd like to follow the slides on your own device, please feel free to scan this QR code. And meanwhile, I'll do a quick survey. I would ask you to raise your hand if you have ever tried to train or run an ML model directly on the client side yeah I guess maybe 10% at most if this topic is rather new for you and I assume it is for most of you, you are in the right place because by the end of the talk you will know what technologies are available where and how to use them, and what are their limitations. But before we proceed, let us introduce ourselves. I'm Oleg, this is Irina, and we are co-founders of Dataforce Solutions. At Dataforce, among other things, we develop LUMEL, an open-source MLOps LMOps platform that helps you build, deploy, and monitor your models. The project is available on GitHub under Apache 2 license, and we would really appreciate if you check it out, provide your feedback, or maybe just support it with a star. However, this slide is here not only for the advertisement purposes, but actually LUMAL is the reason why we got into the topic of in-browser ML. A few years back, we started to explore the initial concepts of the product, and while our initial ideas were quite different from what we have right now, there was one idea that persisted. that idea was really simple a lot of products don't allow you to do anything until you sign up and to us this didn't feel like a right thing to do because it creates a lot of friction for the users before showing any value at all therefore we decided that we wanted to build a module that might not be as important for the platform in a greater picture but that would allow users to experiment with some of the functionality at least a bit without creating an account. The module we made in the end is called Express Tasks, which basically allows you to build models that are compatible with other modules of the platform out of the box. For example, we have no-code interface to train tabular models in the AutoML fashion. We have JupyterLab instances. We have a no-code builder that allows you to define an LM chain and automatically optimize the prompts, and so on. But, as you might have guessed, this all works 100% on the client side using the tech stack we are going to talk about today. But obviously, there are even more use cases. You can have an interactive documentation of your Python library. You can create a shareable data tool without any backend infrastructure, and you can even run a large sandwich model directly in the browser on the client side. So that was a rather long introduction so let's finally proceed to the next part focused on running Python inside of the WebAssembly. First let's figure out what WebAssembly even is. As per WebAssembly.org, WebAssembly is a binary instruction format for a static-based virtual machine. Wasm is designed as portable compilation target for programming languages enabling deployment on the web for client and server applications. So it's an an instruction for a virtual machine, kind of like what Java promised 20 years ago with its compile once, run everywhere. Except it operates in a much lower level than the Java bytecode. And because it's so low level, run times can cheat compile it really efficiently. In practice, you can achieve near native speeds. Obviously, there is some overhead, but it It can be much faster than running JavaScript for computer stuff. You can also run WebAssembly almost everywhere. There is currently huge traction around running WebAssembly on the server side as a possible alternative for Docker containers, but it's not really our focus today. We are interested in the browser compatibility. And here everything looks great. WebAssembly is supported by every major browser, even the mobile ones, so compatibility is really not an issue anymore. Now the question is how to use this on practice. After all, we wouldn't want to write the WebAssembly manually. You might guess that there are compilers that can target WebAssembly, and you would be right, but this part is a bit less intuitive than it seems. The thing is that WebAssembly can only do computations on its own, therefore only mathematics on integers and floats. It cannot open a file, it cannot do a network request, it doesn't have anything like a standard library, and so on, which means that A, WebAssembly model should rely on the functions provided by the host environment for many of its capabilities, and B, it's really beneficial if the communication between the vast module and the hosting platform is standardized. After all, if every runtime exposed its own set of functions, portability would quickly fall apart. Therefore, for WebAssembly ecosystem, the big part is not only compiling the code, but also defining common ways for Wasm to interact with the outside environment. There are several projects that try to solve this. For example, Wasi and Wasix are efforts to standardize system, style APIs for WebAssembly. However, they are also more focused on the server side. Therefore, for us, Emscripting is much more interesting. It's a rather mature toolchain for compiling C and C++, especially for the web. It ships as a large runtime layer, includes a standard library, a virtual file system, it also bridges to browser and JavaScript functionality, and overall allows to bring a lot of native code to web with relatively small changes. This sounds great, but we're at PyCon, therefore it means we must be more interested in writing Python, not C or C++, but luckily the reference implementation of Python that we're using every day happens to be written in C, which means that there should be a way to compile the interpreter itself into a WebAssembly module and use it inside of the browser. And Pyodide is exactly that, a port of C Python to WebAssembly that, among other things, allows to run Python inside of the browser, install Python packages, call JavaScript from Python, and vice versa. I will show how to use Pyodide a bit later, but for now, I would like to focus a bit more on the packages. Basically, if the package doesn't have any non-Python extensions, it can be installed from PyPI right away using a tool called micro PIP, which you would use more or less the same way like you use a normal PIP. However, if a package is not pure Python, it gets a bit more tricky as the extensions have to be built for WebAssembly as well. For popular libraries like NumPy, Pandas, and so on, there are already pre-compiled builds that can be installed from PyDate index. This index is separate from PyPI. Currently, it has around 250 packages and is constantly extended. Therefore, there is a high chance that the package you need is already there. However, if not, you would have to build it manually using a utility called pydate build. Usually it's not a big deal to build it, basically just a single terminal command, but sometimes it can be a big headache and also require a lot of code changes in your library. Therefore, I would say if the package that you require is not pure Python and there is no precompiled builds, then it might be a good reason to consider not using Pyodide and WebAssembly at all. There are also some other limitations you should consider before using or not using Pyodide. First of all, the functionality that would require sub-processors or native circuits wouldn't work. For example, this means that you cannot easily connect to a SQL database and retrieve the data. WebAssembly is 32 bits, meaning that the maximum amount of RAM is kept at 4 gigs, and on top of that, some browsers limit it at half that size, therefore, if you need more than 2 gigabytes of memory, it's not always possible. There is a WASM64 spec that is already implemented by some, probably many of the browsers at this point. However, it's not supported by Pyodide yet. Finally, the persistence is also a bit tricky. You get the virtual file system out of the box, so there is some persistence layer. However, at the same time, it's sent boxed to the browser tab in the domain this step is running. Yeah, therefore, it's a bit limited. At least in Google Chrome, there is an API for native file system access. However, I think it is still considered experimental, and anyway, it's not integrated into Pyodide out-of-the-box. There are still other limitations, for example, Pyodide runs slower than native CPython and so on, however those limitations are not as important as the ones outlined on the slide. So now let's talk about how one would practically use Pyodide. In a nutshell, two main types of use cases are in-browser dev environments and Python-driven front-end applications. So let's start with the first one. The best example that would fall into this category is JupyterLite. It's basically a JupyterLab running in the browser without the backend server. It has the same frontend, the same UI, even supports some of the extensions, but it uses PyDate instead of the regular Jupyter kernel. From the user's perspective, if everything stays mostly the same and, in fact, if you were given a JupyterLite instance instead of the normal JupyterLab, there is a high chance you wouldn't immediately notice that something is different. Running a JupyterLite instance is also extremely easy. You can even host it for free on GitHub pages because it's just a static web server. And there is a dedicated page in the documentation that shows how to do it. And yeah, that's not much else to say about JupyterLite, but I would encourage you to try it because it only takes a couple of minutes to set up. So now let's go back to building frontend applications with Python. We already talked a lot about Pyodide, but we didn't really have a single example that shows how to use it. And in fact, it's really, really simple. You simply initialise the PyDiet instance and then pass the Python code snippets as strings into the runPython method. Specifically in this example, we are calculating the average of an array using NumPy. It's also possible to access the JavaScript scope using a built-in JS module, which for example allows us to manipulate the DOM. So all in all, we can run Python, and we can access JavaScript scope from Python, which should allow us to build pretty much anything, however, at the same time, it feels a little bit low level. Which brings us to the last project in this section called PyScript. PyScript is developed by Anaconda and largely builds on top of Pyodide, but adds some convenient abstractions so that the whole experience of writing Python for the browser feels a bit more native. First of all, it allows to supply a PyScript config that defines the packages that need to be installed, the files that need to be placed into the virtual file system, and so on. Then Python itself becomes just another script tag in your HTML code, and similarly to JavaScript, you can either write it inline or provide a reference to to a pre-existing Python file. Finally, there is also a bunch of helpers provided by PyScript that simplify the bridging of your Python and JavaScript logic. For example, you can easily create event listeners and assign the handlers using a simple decorator. And one more thing that is really cool about PyScript is that it's possible to use MicroPython as an alternative interpreter instead of Pyodide. For those unfamiliar with it, MicroPython is another implementation of Python interpreter, which was originally meant for micro devices, but currently fits into the whole WebAssembly ecosystem really naturally. The pros and cons of using MicroPython instead of Pyodide can be summarized in just a few sentences. First of all, MicroPython itself is less than 300 kilobytes, which means that your final bundle size is approximately 25 times smaller than when you're using PyDyte and by extension this usually results in much faster code starts. However, at the same time, MicroPython differs from standard CPython in feature support. Obviously, it largely depends on the version you're using, but your available Python code is not guaranteed to work with MicroPython right away and on top of that you are losing the rich system of packages because they need to be rebuilt for micropython separately okay and with that we conclude the part about python ecosystem in the browser and heading over to Irina who will talk about the inference stack.
Speaker 3 [15:46]
Yes, so the inference stack in the browser is rather different from what was discussed in the first part of the presentation, mainly because it's not Python-based. There are a few key reasons for that. First, remember that we cannot compile Python to WebAssembly, but only the interpreter. While Python itself is not nearly as fast as the compiled languages, here we're at even more additional overhead. Therefore, it would be really non-optimal for larger models and especially language models. Secondly, to the best of my knowledge, there are no libraries that would allow you to use GPU for model inference for Python. But let's first talk about the possibility of using the GPU in the browser. There is a common misconception that running the ML inference on the GPU in the browser is impossible because the only way to access the GPU is via WebGL, and WebGL is designed mostly for graphic processing and not the general compute. While the second statement is mostly true, WebGL is not the only way anymore, as there is also WebGPU, which is much more modern and was designed with support in general compute in mind from the beginning. It's also widely supported by the desktop browsers, and the support on mobile is also steadily growing. So now let's see how we can use it for ML inference. The easiest way is if you have your model converted to ONIX. Onyx is a universal format in which you can represent your models as a directed computational graph of standardized compute nodes. In other words, it acts as a universal adapter where you can convert the model from almost any framework into Onyx and then it can be consumed by a large number of platforms. Though Onyx is just a specification of the format, in order to do the actual inference we need a runtime that supports ONNX models. And here we don't have that many choices as far, by far the most complete and accurate one is ONNX Runtime maintained by Microsoft. As you can see, it's available on all major platforms and has bindings to many languages, but obviously the most interesting for us is ONNX Runtime Web. It can run on both WebGPU and CPU via WebAssembly. It's really simple to use basically just several lines of code to load the model and generate the predictions. There are also some libraries that build on top of ONIX. For example, Hugging Face has an official transformers GS library which allows to run pre-trained models. It's very actively developed and, by the way, got a major V4 release just at the end of the last month. There is a rather big collection of spaces that currently support around 200 model architectures including large language models like QAN or DeepSeq. You can try it out without doing any setup. However, this is not the only way to run LLMs in the browser. You might have heard about the MLC, which is the project that compiles and runs the models on the MLC engine. Similarly to Onyx, it has a web-based runtime called WebLLM, and if you go to chat.weblm.ai, you can try it directly as it provides a small library of models and the chat interface. Overall we've been using WebLLM on and off for about one and a half years, mostly for exploratory purposes. It behaved quite stably with only some occasional failures to generate the predictions, and while I don't have hard numbers in mind, the speed of the generation always felt very impressive. However, one thing that I would consider a huge drawback is that it does not have a CPU backend. Obviously, GPU is much more performant, but not yet universally supported, so having a fallback option would be great. Speaking about running LLMs on CPU, if you ever try to run one, you must be familiar with the project called Llama CPP for running models in the GGUF format. The project has a huge adoption and ecosystem. For example, Hugging Face Hub has more than 160,000 GGUF models. Under the hood, Llama CPP is powering for example Ollama, another very popular project with more than 50 million monthly downloads and many other tools. And the reason why I mention this is because Lama CPP also has a web version called VLama. Similarly to WebLLM, there is a web app you can try out, so feel free to do so. And this concludes the inference stack part of this presentation, so let's do now a quick recap of what was discussed today. WebAssembly provides a way to run a Python interpreter directly in the browser. This can be used to either have a Python dev environment directly in the browser using JupyterLite or make frontend Python-based application using Pyodide and PyScript. Pyodide supports most of the pure Python packages out of the box. Many popular packages with native extensions are precompiled and can be used right away. For performing ML inference in the browser, the most flexible option is ONNX, and by extension Transformers.js, that can run on both CPU and GPU. Specifically for LLMs, the alternatives are WebLLM, that has its own format and supports GPU only, and VLAMA, that allows to run GGF models on CPU. In conclusion, the ML ecosystem in the browser is still relatively new, but it's steadily majoring and allowing to implement more and more use cases every year, so we definitely encourage you to experiment with it. Before ending the talk, a couple of quick announcements. First of all, we have brought around 200 sheets of our new 2026 edition sticker packs, so please pick one up after the talk. We definitely don't want to carry them back home. In addition to that, we are looking for volunteers to do a 15-minute questionnaire-style interview about your usage of LLMs and agents for data science work. So if you're a data scientist and would be so kind to donate 15 minutes of your time, please pin one of us on LinkedIn. We would be very thankful for that. And if you don't want to do the interview, it's fine. Please also free at us, and we're always happy to chat and answer your questions. And that's it. Thank you for your attention, and I hand it over back to Oleg.
Speaker 1 [23:05]
Thank you for the great talk, Oleg and Irina. It was really interesting and fascinating to learn about the state of affairs of how good browsers and on-edge devices have become to be running these models. Thank you. I'll just take some questions at this point. We have quite a few. The first one that I see is, how does the bridging between the inference in JavaScript and any code that you write in Python work. Can you show any demo by chance?
Speaker 2 [23:42]
I'm not sure that I got the question correctly, but basically the question is how to use the inference stack from Python, I guess. It's really simple because from PyDy to PyScript, you can access anything that is in the JavaScript scope. So you can access the JavaScript objects, you can call the JavaScript functions, and so on. And the inference stack has bindings for JavaScript, so by extension you can also call them from Python. So this shouldn't be an issue. I don't have the examples on the slides, but you can pin me on LinkedIn and I probably can provide it.
Speaker 1 [24:20]
Okay, thank you. I'll take the next question as well. What models can you run with the 32-bit limit of WebAssembly?
Speaker 2 [24:29]
Actually quite a lot. So you can run even 3 billion models and at this point 3 billion models are quite capable. I would say that even 450 million models don't feel as dumb as they were a couple of years ago. I wouldn't really recommend you to use anything below probably 1.7 billion for any serious work. But yes, you can do a lot more than you think.
Speaker 1 [24:58]
Okay, yeah, so running, I don't know, 100 billion models. No, no, no, definitely not.
Speaker 2 [25:03]
No, definitely not possible.
Speaker 1 [25:05]
Still not there. Can you recommend something to do robust voice activation for a custom word in browser? Something in a fashion of, OK, Google.
Speaker 2 [25:19]
No, but as you mentioned, there is a large collection of spaces on the Hagen phase, and I'm pretty sure you would find something for that. We unfortunately don't really work with audio, so I don't know.
Speaker 1 [25:36]
Yeah, I think in that case, some sort of speech recognition on edge system. Yeah. And what's the primary motivation of these frameworks when they get started? Is it privacy-centric, that your data is not being sent to...
Speaker 2 [25:54]
I would say there are two main drivers. First of all, privacy, because, yes, you don't have to send any data to the servers, but also the scalability, because everyone has a compute device. And, yeah, for example, I'm not really sure about the name, but there is an educational platform that teaches Python, and I guess they're running their whole stack on GitHub pages for free, and it has, like, thousands of users, so it can be really, really cheap to do so.
Speaker 1 [26:23]
I think Code Academy, right?
Speaker 2 [26:26]
Code Academy is different. There is a specific platform that teaches Python to kids.
Speaker 1 [26:32]
Bison, OK.
Speaker 2 [26:33]
Python, okay. So basically, if you have a really small instance to host your static web server, you can scale it to probably tens of thousands of users at almost no cost.
Speaker 1 [26:47]
And probably the final question that I see, which common machine learning packages are available pre-compiled via Pyodide?
Speaker 2 [26:59]
A lot of them, so scikit-learn, pandas, numpy, polars, so PyTorch is not available, which is a big problem, but everything that is not too big, I think, is available at this point. And you can go to PyDiet, I don't remember the link exactly, but just Google PyDiet package index, and there is a list of the packages and their versions.
Speaker 1 [27:25]
Yeah, I see one last question as well. This one's probably interesting. If I start to work with Python machine learning in the browser for the first time, what will be my biggest pain points?
Speaker 2 [27:39]
Yeah, it's an interesting question, depending on what's your background and what you want to achieve with that. Maybe, it's really difficult to answer, but I would say it's really annoying when a transitive dependency for a package is not available and there is no easy way around it. Yes, at least for us it was a big problem. but as we said it's maturing really rapidly and right now it's much better than three years ago and for a lot of use cases you wouldn't have any problems at all
Speaker 1 [28:20]
I guess in the beginning you would have to do some sort of assessment whether the packages that you want to run are available or whether you have to build them.
Speaker 2 [28:31]
And building, as I said, because we had a case when we needed to build a package and it was a huge pain. As I said, if the package is not already available, maybe it's a good idea to stick to a normal CPython and not go into this direction.
Speaker 1 [28:51]
That's all for the questions. Thank you so much for the talk and answering these questions. Can we please give a huge round of applause for their talk, Ken?
Speaker 2 [29:01]
Thank you. Please grab the speakers.
