From Scratch to Scale: Turning LLM Code into Architecture Insights Keynote

Speaker 1 [00:00]

Thank you. Awesome. All right. So I unplug my notebook.

Speaker 2 [00:04]

I don't know.

Speaker 1 [00:05]

We have to prepare

Speaker 2 [00:07]

Um, so, I don't know.

Speaker 1 [00:12]

That here so actually I think the story actually the story started when Sebastian the setting saying out like this computer The story actually started in the psychic talk in Austin, Texas. It's sci-pi three years ago, it was brazenly hot and Sebastian asked a question I said hey This is this great guy who has all these amazing posts and is very willing spending so much time and is really good Educating people in the field of large language models. I said, okay, this is the guy sitting over there The other thing I mentioned I wasn't aware of I mentioned a German accent

Speaker 2 [00:53]

Ha ha ha.

Speaker 1 [00:54]

And then I googled, oh, yes, Sebastian actually originated from Germany. So actually at the social event we talked, I said, hey, it would be really great for you to come by with all the great work you do at the conference. I think two books came in between, A Planned Holiday as well. But actually I'm not sad about the timing because I think you and your expertise and also your upcoming book about reasoning models, It's it's just better timing even like to have you around here If you know those Sebastian he has like I don't know when you sleep with all your courses video blog posts Yeah, I I think you can I can wake you at 4 o'clock in the morning Say name me the latest 20 models and the benchmarks and stuff I think it's such an amazing contribution for everyone to understand the technology better. So please be the one welcome to Sebastian

Speaker 2 [01:54]

Yeah, thank you so much for the invitation and as Alexander mentioned I have a strong German accent So I actually I was half prepared to do this talk in German. So I'm from Germany and I'm actually quite excited I have never been to a German conference and so I Actually quite impressed how much fun it sounds like this conference is gonna be because when I go to other conferences is very technical But it sounds like there's a lot of fun stuff happening from yoga to parties to everything here So, yeah, I will be a bit technical, but, yeah, as Alexander mentioned, there's also a lot of fun stuff at the conference after the talk. So, yeah, so today I want to talk a bit about LLMs, and there are a bunch of things I wanted to cover. As usual, I'm, I think, always a bit too optimistic in terms of how much time we have and how much we can cover. I would like to maybe do a four-hour workshop to cover everything, but it's not possible. so I'm trying to get something interesting down in 45 minutes here, and I thought it might make sense to start with the LLM ecosystem, just like a brief overview, so everyone knows what I'm talking about, but I think, I mean, it's 2026, most of you have probably already used large language models, and you are probably using large language models every day now, so I think I don't need to introduce how large language models work, but I thought it might be really interesting to understand the latest LLM architectures because there are so many out there and there are new architectures basically every day. And yeah, I wanted to cover something that is not available somewhere on the internet, like something that is a bit more, I guess, unique that I wanted to put together for this talk. And then, yeah, I wanted to also talk a bit about the trends, like what's next for LLMs, AI agents, and so forth. And at the end, I also have like a short getting started section for those who, let's say, tinkered with LLMs, but haven't like, let's say, coded one, like just giving some advice and some tools and projects to check out. So yeah, so I wanted to start with just like a simple demo. And so this is an LLM I coded locally. It's a very simple, very small model, but it gets the job done. It can do simple things. It's like a very simple reasoning model. I ran it actually on this little laptop here. And just to motivate this talk, this is coded 100% in Python. I mean, Python in quotation marks because I'm using the PyTorch library, and the PyTorch library may use CUDA or C++ under the hood. So it's not pure Python, but since it's a Python library, let's call it 100% Python. And the interface here, that's based on Chainlit, which is, yeah, Python plus some TypeScript, a JavaScript library to render this in the web browser. But it's essentially like a Python project just to motivate how much you can actually do in Python. And so most of the LLM ecosystem at some point right now still goes through Python as the common language to introduce ideas, to run experiments, and share the models eventually. So for the training, most people still use Python. Like I mentioned before, there's the PyTorch library. People at Google, they also like to use JAX, but PyTorch is really the most, I would say, most popular one. And under the hood, of course, PyTorch calls CUDA to make everything fast and efficient. And what's quite interesting, though, I mean, still in 2026, it goes all through the same bottleneck, and that is the Hugging Face model hub, which is, you can think of it as GitHub for model weights, for the open source, open weight models. So everything is kind of stored on the model hub. And then there are different tools that are not Python-based. So for example, if you're running a small server, serving LLMs locally or on a small cluster, usually SGLang or VLM are popular tools. They're mostly C++, CUDA, and so forth. And then for local LLMs, there is, for example, Lama CPP and O-Lama, which is based on Lama CPP, like a nicer interface, and then also MLX, where MLX is specific to Mac computers. O-Lama also has MLX support, I think, as of last week. So mostly the training is still very Python-heavy, and then the model hub, there's a Transformers library by Hugging Face, which can load all the models there. and so this is essentially how the ecosystem looks like in 2026 and originally for this talk I thought I might dive into these things how how these tools work but I thought it might be a bit boring because to be honest with you there is a lot of documentation on the internet so if you want to know how I guess sglang works I think the best place would be going to the sglang documentation so for this talk I thought I would do something maybe hopefully more interesting something that is not so readily available on the internet, and talk a bit about, let's say, my workflow, how I, let's say, learn myself about new LLM architectures, how I compare them, how I get useful information out of them, and that usually leads to also implementing those architectures and using them in, let's say, production contexts. So the whole dilemma, though, with this approach, and I hope this is maybe useful to you, outlining these different steps is nowadays papers are not as detailed as they used to be, so there is still archive where people upload papers, like technical reports, but compared to like five years ago, they are more like technical summaries, but they don't really have that much information about the training and the architecture itself of the LLM. There are exceptions like Nimotron, Olmo, and so forth that have still a lot of details, But most of the architectures nowadays, the open weights, the LLMs, they're just posted on the model hub. There's maybe a short blog post, but there's not that much detail anymore in these papers. But what we can do, and I think most of you are coders, I mean, it's a Python conference, so I think most of you are programmers and coders, so you might like tinkering. And that comes in handy because code really doesn't lie. What I mean with that is if we can re-implement an architecture that is out there, match the results, and we have readable working code, that's kind of like a source of truth. From that we can really learn how the architecture works. It's basically reverse engineering the architectures. So I wanted to briefly, in the next 10 to 15 minutes, walk you through my process. When the new architecture comes out, how I learn, let's say, about that architecture and what are the useful informations we can extract from LLMs. And that helps, in general, to understand how different LLMs compare, what the strengths and weaknesses are of the LLMs, and much more, basically, like getting really a solid understanding of how LLMs work. And so, yeah, I wanted to walk you through the process of going from configuration files and code to actually like architecture drawings and concepts we can learn from looking at these architectures and comparing them. So yeah, the first step is not very exciting or interesting and a very obvious one. It's following the news basically in that sense. I mean, there are a lot of newsletters, social media and stuff out there. And yeah, if you want to stay up to date, I guess It's necessary to, at some point, check what's out there. I'm picking here an example that's actually a bit older, Gemma 3 270M, which is an open-source or open-weight model by Google that came out last year. There's now also Gemma 4 that came out last week, and I did the same process for Gemma 4 as well. But it's a larger model, and it doesn't run on the laptop, so I tried to keep it small in case you want to replicate this later after this talk and run it on your own computer, for example. So let's say I learn that there's a new architecture out there. The next step for me would be to check if there's a technical report or a blog post. And I would, of course, if it's available, also read that and go through the information, keep notes or take notes and so forth. There's nothing really exciting about these steps. But then the more interesting stuff comes after because there's only so much information available in these technical reports, that at that point, I would check out the model hub where companies upload their open weight models. So you can, most of them are freely available with, depends on the license. Some of them are free to use. Some have a few restrictions. I think Gemma 3 has a few restrictions for commercial use. Gemma 4 though, I think it's like a standard MIT or Apache version 2 license. But that aside, from the model hub, there's usually like a repository, like a GitHub repository, there are the config files that already contain a lot of information. So if I go then to the config file, I can already see, okay, there is something like sliding window attention. And I can see the size of the model. And I can also see, for example, there's full attention and sliding window attention alternating. And I can see there's a hidden activation function called gelu. And so there's already a lot of information in the config file that gives you an understanding of what this architecture looks like. And then specifically, if you already know a reference architecture, you can already get a feeling for the trade-offs. So sliding window attention, we will get to the details later on, is, for example, a trick to make it run faster in inference or also mainly to reduce the KV cache size to run it on, let's say, smaller hardware. But we will get to that later. But there's already a lot of stuff we can learn. And usually at that point, I would take one architecture I already implemented in the past or where you can find some information out there. And to be honest with you, still in 2026, most architectures are very similar to GPT and GPT-2, for example. So you may then look at the original auto-regressive large language model. Maybe there's a more recent one that you are familiar with. So for example, QUIN 3. and from there I would then try to read through the config file and see how similar are these architectures to each other to kind of like get a get a rough feeling of what they're trying to do so you can already I mean it's a bit small but you can see there are tiny differences they have different norm layers at different positions but overall it's a very similar architecture still like in 2026. I would then from there like once I have a good understanding I would kind of like dive into the code, take an existing architecture. This is Quen3. I would just have my code open and then step by step, going through the config file, I would make these changes. For example, Quen3 has a silu activation function and Gemma3 has a gelu activation function, which is more like similar to the GPT architecture. It's not super important detail, but it's something you can already see by looking at these config files. And then doing this top-to-bottom, you can code up an architecture and run it. So here, I don't want to show you the 200 lines of code because it might be a bit crowded on the screen, but assuming I implemented the whole Gemma 3 architecture, I can then run a simple prompt. And so in this case, I picked something simple like, give me a short introduction to large language models. And so if I run this code, what happens, this is a real example. It outputs something, I honestly don't know what it outputs. Something that is definitely not correct. So this is definitely not the answer to my prompt. It was even switching the language. And Gemma 3 is a multilingual model. So I can learn from here, okay, this is a multilingual model, but something is not right. I made a mistake here. So something is definitely messed up here. And so the next step what I would do is I would go through the architecture step by step and try to find the mistake. It can be a bit tedious, but I think this is probably the best way to learn really about the details, like starting from an existing architecture, making the changes, and then kind of like failing, but then also going through things one step at a time. Nowadays I'm a bit more, I would say, organized. I would like to implement some unit test style check for each layer. And by the way, I should have links to all the code materials if you want to check it out later. I think the slides are linked in the PyCon conference page, so you could technically later also run everything yourself if you're interested and modify it. But yeah, so the point here is I would go layer by layer. and at the top I would already notice there is a difference so there is a difference at the very beginning something I messed up and so when I'm saying I'm going through by layer by layer I'm meaning here that I'm running my own code and side by side I'm running the hugging face transformer reference implementation and so what I'm doing is kind of like re-implementing things and basically reverse engineering things and I see okay there is something not right so at some point I made a mistake. There is something about this RMS norm where I messed up. And so that's the point where I would try to find out what it is. So I would go into the source code that is the transformers library, the Hugging Face transformers library. And as I mentioned before, the model hub where the open weights are, they can be loaded into this transformer library. What's the point of implementing things from scratch? It's purely like a learning exercise because the transformer library is quite complicated there's a lot of stuff in there it's a great library but they support so many LLMs and so many advanced features that the code becomes kind of like hard to read it's not sequential there are a lot of files and so implementing this from scratch is a great learning exercise it is a bit like a Easter egg hunt so you have to go from file to file and then eventually you can find the implementation here for the RMS norm and in this case if you do this exercise you will see okay there is something particular about this RMS storm. So they have this term one plus a self weight which is a bit unusual. So I think I have a comparison here. So I would basically look at this and compare it to my previous implementation and then see okay there is something they did differently and this is like the point where you kind of learn like where you learn okay there's something they did. Why did they do that? Well sometimes if you're lucky they mention that in the paper. I think in this case they didn't, so you would have to do some thinking, maybe look online to find some suggestions or ask maybe people who worked on the Gemma architecture. But what they did essentially here, it's not the main point of the talk, but what we can learn is that they just slightly changed the formula to stabilize the training. So here on the right side, they have this one plus scale instead of just timescale. So that helps them to initialize the parameter at zero. and in classic deep learning if you have a parameter you start it so with a zero weight centered at zero it's easier for something like stochastic gradient to optimize that than something that is centered at one so just like a minor detail but that's something we can learn from the code that is not available in papers or in other resources and this is just a very small simple example because it's very short but the same applies to the different attention mechanism to the activation functions and the transformer block in general. So there's a lot of stuff we can learn by just doing this kind of like ablation one by one. So I would, continuing with the story, once I figured out all the mistakes, I would then run my checking tool again and see, okay, this time it passes. Everything matches the reference implementation. And this is actually a great way to learn also because you can self-check your solution. So you have a way to know whether you're correct or wrong. It's kind of almost like reinforcement learning with verifiable rewards in training reasoning models, where you have a reference solution, and you can self-check yourself. So it's actually a great way to learn. Now, step 12. So everything here is done on a smaller scale with random weights. So here I'm just initializing the model with random weights and compare to the hugging phase random weights because it's just faster to load the model because sometimes if you're working with large models, it takes minutes to load the weights and run it. So this is just all done on a small unit test type of scale. But then once everything is implemented, I would load the pre-trained weights from the model hub, generate my response again, and now it's the same query. Give me a short introduction to large language models. I can see now it's outputting coherent text. So it's outputting something like large language models are sophisticated artificial intelligence systems and so forth. I would do the same thing for the Hugging Face Transformers library. And ideally, I want to see those two match. And so that tells me, okay, now I re-engineered, I re-implemented the Gemma 3 architecture. And it works exactly the same way, which is great. So now I know, okay, everything is correct. And I have my minimal code implementation in Python that I can learn from because it's easier to read. It's shorter, and it doesn't have all the features that Transformers has, but it's more self-contained, essentially. And yeah, so this is essentially my little workflow for learning about architectures, and I've done this for a lot of architectures. It takes a lot of time, so I wouldn't recommend doing that for every architecture because it's very time-consuming. So I have more than 50 architectures up there, but I have not done the exact process for all 50 because for some of them they are so similar you can actually just modify it based on the config file to get an idea of what the architecture looks like so yeah I hope this this workflow was just you know useful like if you want to learn how architectures work that's that's how I would do it but the main topic of the talk is also what is like interesting about today's architectures in 2026 how do architectures nowadays look like compared to a few years ago So these are just recent architectures from, I think, starting up in the upper left corner is GPT-2. But other than that, they are all like architectures from 2026. And I wanted to talk you a little bit through what's interesting about these architectures. So I put together a few weeks ago this LLM gallery. So this is just like a short link to my website where I have more details about each architecture, like from papers the code and everything all the information I extracted and if you're interested you can browse through this it's just like a plain website and also a little comparison tool where you can have two architectures side by side to learn about their similarities and differences so I hope that is useful it's just something I put together recently based on all my drawings over the years and yeah so the next part of this talk i wanted to briefly talk about what's new in in terms of llm architectures what are the interesting things we can learn so the main focus if we are looking at all these 50 architectures and what they have in common and what they try to achieve is all of them kind of like focus on minimizing the kv cache like reducing the size of the KV cache. And I don't know how many of you have implemented LLMs or are using LLMs on a regular basis, so I wanted to briefly mention or explain what a KV cache is. And so before I get that, the motivation here is to essentially also run LLMs with longer responses like reasoning models, but also agents, and we'll get to that later. But yeah, the main commonality is reducing the KV cache, and here's like a slice through the attention mechanism. It's very technical and I don't expect you to, if you have never seen an attention mechanism before, to understand all the numbers and letters here. But the main point is there are things called a key and a value. And so these are like intermediate values that are computed when running an attention mechanism. And usually they are computed each time. So LLM generates one word at a time and here we have maybe a better illustration and every time you run the LLM you feed back the input from the previous round. So generating text with an LLM is an iterative process but each time if you are not careful I mean by default you would re-compute keys and values that you have already computed before. If you have a KV cache what you do is you store values that you have computed before and you just reuse them. So it's kind of much faster So now instead of feeding back the whole input and then computing the keys and values again, you take the keys and values from the previous round and feed them together with the new text to the LLM. So again, this is very technical. I have a whole article about this. This would maybe be a two-hour talk also just to go through the details. But the main message is we are saving intermediate values in this KV cache to make things faster. But then when we are saving them, it uses up memory, like storage. And then if we have reasoning models that create hundreds of thousands of tokens, and if we have agents like OpenClaw that run reasoning models in a loop, that is a very expensive memory footprint now, because we are saving these keys and values for all the iterative generation steps. So just to give you the reference, this is the regular attention mechanism in GPT-2 before all the fancy optimizations. This is like the original. And in the original attention mechanism, we have for each query, we have one key and one value. So it's like a one-to-one, basically, ratio. Now, in 2022, 2023, people started moving to some concept called grouped query attention. This, I think, was mainly popularized by Lama 2 end of, was it 2023, for the larger Lama 2 models. So it's nothing new, anything, but it's still very popular. And the idea here is that two queries, like inputs, they share the same key and value in this drawing. You can also go larger. This is just to fit it onto this slide. But the idea is that multiple queries now share keys and values. And in this particular case, now if we think about storing the keys and values in memory, we cut the memory in half. Now we need only half as much memory as with a regular attention mechanism. It's like this little trick here. And the downside is that you might reduce the modeling performance by a little bit because now, while the model has fewer parameters, in a sense, you are just compressing things. But you can get it to a state where it's almost as good as the full attention mechanism, but you save a lot of memory, basically. A lot of architectures, I mean, I mentioned this is from 2023 when it was introduced, used, but today, still a lot of architectures use it. For example, Sarvam is a recent model from India, or Gemma 4 that just came out last week. Google Gemma 4, they also still use group query attention, so it's a mechanism that is still used. And here's just like a drawing of kind of like visualizing how much you save. So on the y-axis is the KV cache size across all the layers in the LLM, and the largest, Up there, the highest consumption of memory is with a regular attention mechanism. And now if I have this group query attention with different group sizes, you can see you bring down the memory cost by a lot. And on the x-axis is the context length. So the longer your output is, the LLM output, the more you save. And like I mentioned before, reasoning models and agents like in OpenClan use a lot of context, a lot of tokens. So the longer your output, the more you actually save also. It's like a very important concept to use the KV cache and then also to reduce the KV cache to be able to actually run today's models. But this is only one trick to reduce the KV cache. Other models try to solve it differently. For example, this is DeepSeq version 3 that came out December 2024 and R1 that came out in January 25. so here what they do is at the top is the regular attention mechanism what they do is they store a compressed state so instead of saving or storing the keys and values in their full let's say representation in the full tensors they have like an additional step a down projection step so it's a matrix multiplication to compress it to a smaller state and then save that compressed state in the kv cache and then during inference there is an up projection to projected up into the original dimension but the idea is that the here in blue the blue box it's like a compressed state and it's the same compressed state for key and value so you reduce first the size by half by sharing this and then it's a smaller representation so it's even smaller than just cutting it by half and so it's again a lot of savings if I would plot that you can see similar to grouped query attention you save a lot of memory and so you might ask okay why are there two different ways now, which one should I use? We'll get to that in a second. So both are valid and legit ways to save KV cache in practice. In the DeepSeq version 2 paper, they had like an ablation study where they compared GQA, group query attention, with multi-head latent attention, and they found at roughly same compression rate, multi-head latent attention preserves a bit more of the modeling performance. It's not quite as I would say in quotation marks, harmful to the modeling performance to use multi-head latent attention compared to group query attention. But there are architectures, families, where they use both. So a recent example was Sarvam. It's a large language model that was very popular from India because it is the first language model to really support Indic language as well. And they have the smaller 30 billion model with group query attention and the larger model, 105 billion parameters with multi-head latent attention. So back in the day, I would have said that people still use group query attention because it's easier to implement. Multi-head latent attention is more complicated to implement. Here, they have both. So we can't say, okay, they didn't implement MLA because it was too complicated. The reason is that for smaller models, group query attention may still perform a bit better. Like I was talking to some colleagues and usually multi-latent attention is more worthwhile if your model has a certain size, like 100 billion and larger. But yeah, both are popular mechanisms. Another mechanism for completeness is sliding window attention. That is the one used by Gemma. So Gemma uses both group query attention in addition to that, sliding window attention. So sliding window attention is yet another trick to save KV cache memory. And so how it works is in the regular attention mechanism, At each token position, you look back at every other token position in the past. It's like this auto-regressive attention mechanism. In sliding window attention, you restrict the size of the context, so you only look back a certain number of tokens. You make the field of view smaller. So that's essentially another trick to reduce the context, and with that, make the KV cache smaller, because then you don't have to store everything back to the first token, but only up to a certain number of tokens. And yeah, like I said, in Gemma 4, they used both group query attention and sliding window attention. And again, so with that, you can save even more. It's maybe a lot of information on this slide, but again, at the top, it's the regular multi-head attention mechanism. And at the bottom is group query attention plus sliding window attention. Again, a lot of savings here. And that is one, because I mean, Gemma 4 just came out last week. it's still one of the most commonly used combinations to to reduce kv cache size for completeness maybe one more so this is deep seek version 3.2 i'm mentioning it because it's kind of right now the template for a lot of other architectures like the cutting edge architectures this architecture came out in december so it's already five months old very old by llm standards But there is, for example, right now GLM 5.1, which is right now the best model out there, the best open-weight model. And it uses exactly the same architecture in terms of multi-head latent attention plus deep-sig sparse attention. So deep-sig sparse attention, so on the right, multi-head latent attention we have just seen already. And on the left-hand side, deep-sig sparse attention is very similar to sliding window attention. But in sliding window attention, we have this fixed window where we look back. And in DeepSig Sparse Attention, it's not a fixed window, it's a learned window. So there's like an indexer, a lightning module, and it's essentially, you can think of it as a parameterized version of sliding window attention, which is a bit smarter. It selects positions in the past that are, let's say, contain useful information, but it's also the same idea that we're not looking at all the past information. So it's, in a sense, also making the context window smaller to reduce KV cache size. Yeah, and like I said, GLM 5.1 is right now the best-performing open-weight model out there. If you are, let's say, interested in the most state-of-the-art open-weight model, that's GLM 5.1. And they use exactly this multi-latent attention and deep-seq sparse attention from deep-seq. And this is actually quite interesting. It's only something you would learn, let's say, by looking at the architecture. Because, I mean, there's a lot of information. all their benchmarks, but they don't really tell you what's behind there. But looking at the code, looking at these config files, you can actually see it's actually the same model. It's a bit bigger. They have a few more layers, but it's essentially the same mechanism. Maybe one more. So that's method five to reduce the KV cache because this was very popular in the news like two weeks ago. Apparently, it crashed some of the memory chip stocks. So there was this method by Google. The original paper came out in 2025. And then this year it became really popular. I think because there was like a new blog post by Google But yeah, turbo quant is a technique to compress The keys and values in the KV cache without sacrificing modeling performance So sorry, so it's and yet another way to reduce the KV cache size and all the methods I showed you before they can you can be used in addition to each other so you don't have to decide Oh, I use this or that you can use this in addition to everything else And so yeah, I wanted to for completeness also mention this because it's right now a really hot topic Trouble command so you probably read about this in the news. I think it was even like in Wall Street Journal because Chip stocks went down 10% on the stock market because now people think we need less memory like this less requirement for memory when we run LLMs and there are many more methods out there so I don't want to bore you to death here by just talking about these very technical details but there's also kind of almost linear attention mechanisms in Quent 3.5 and NVIDIA Nimotron Gemma 4 had another trick using keys and values as the same representation and so forth so a lot of stuff nowadays is about making LLMs more efficient really, like reducing the KV cache size. And yeah, so this was like a little bit of a whirlwind tour, technical details. I hope I could motivate you a bit in terms of looking actually at the code instead of just benchmarks, because there are a lot of interesting things we can learn. And I think this is also a useful kind of tour to learn when you are implementing things on a laptop or a smaller device, what are some architectures, what are the trade-offs, what should I look out for, what does it mean if it has sliding window attention, should I enable it or disable it, and so forth. And now you know, okay, it does actually affect my KV cache size, which affects memory. But this is also something that is really important nowadays because we went from regular LLMs last year to reasoning LLMs. So reasoning models are essentially models that are, they are training methods on top of the regular training methods that turn a regular LLM into a reasoning model. And reasoning models are essentially just beefed up LLMs. It's the same architecture, just with a bit more training to have them output more tokens and use more tokens for intermediate reasoning or so-called, quotation marks, thinking steps. Another really popular topic these days is something like OpenClaw, an agent, agentic loops. And so this is something I would say this year is going to really matter because in the previous years, people optimized pre-training for LLMs to have better foundation models. Last year, people optimized the training of the reasoning models on top of the pre-trained models. This year, a lot of people work on agents. And the agent is essentially just like a code wrapper around a reasoning model or regular LLM that gives it a bit more capabilities to basically get more performance out of the LLM. So if you're interested, I mean, so first, coding agents are very popular nowadays, there is Cloud Code. And there's also OpenAI Codex. I also, for education purposes, built my own coding agent, mini coding agent. Here, my focus was kind of like to explain the six concepts or components of a coding agent. So a coding agent in general is a wrapper around a reasoning model and there's an agentic loop where the model would receive a query or inspect the repository choose some action for example editing a file deleting a file moving a file running a web search or something like that and then dealing with the outputs feeding it back and it's like a loop where the model has access to the for example repository if you have a local github repository or different tools shell tools Unix Linux shell tools like grep and find and so forth and it's essentially running this in a loop and this gives the LLM more capabilities also you know back in the day people were complaining about hallucinations in LLMs it's still a problem but if you have a web search for example you can already reduce a lot of unnecessary hallucinations. So for example if you ask the LLM who won the soccer World Cup in let's say 1998. So instead of having the LLM trying to remember that from memory and maybe hallucinating you could technically just use a web search and you can also have the LLM do the web search and give you a result. And okay this doesn't mean the LLM is always going to be right because it can do a web search and it can do it badly and get some wrong information from the internet but you can at least reduce if it's done well you can reduce these hallucinations and agentic coding harnesses they are usually so you have additional training steps you are training the LLM to actually use or operate within that harness and really that gives you a lot more performance than just using an LLM in the web interface for example so it's an exciting topic so there are these different components like the live repo context the prompt cache and here again that's where the KV cache matters because you don't want to recompute everything every turn. So there's like some history, like the conversation history, the repo context. You want to keep that cached and to make it fast and to not waste compute because using LLMs is very expensive these days. And so that's again where it matters which architecture you use and what the optimizations are. So it's just like yet another interesting topic to look out for. And I do think this is like right now the focus where a lot of people build really cool things at the moment, like in this agentic context. Lastly, I honestly don't know how I'm doing on time. I hope I have a few minutes left to talk about getting started tips. It's a short section, but this was probably very technical and I hope it was not too technical and there was something useful in there that you can't find, let's say, readily on the internet. So I tried to make it interesting, but I know it's a lot of information. So I wanted to talk a bit about like some, you know, like steps or resources that you might find useful. if you are just getting started. So I think, personally, I like bottom-up approaches, like really understanding one thing before moving on to the next thing. And I do think it really makes sense to understand how LLMs work before starting on agents also, because then you kind of learn to appreciate all these nuances, like what is a KB cache, why do I need that, how long should I set my context size, and so forth. And then, at that point, I would also implement a few LLMs and training algorithms from scratch just to get a feeling for it. And I do have plenty of, let's say, resources online that might be useful, like GitHub repositories with open source code. There are also books associated with these repositories that give you, let's say, more like a narrative and more detail. But all the code is on GitHub. So there's LLMs from scratch, which is essentially just the base LLM, the pre-training, the fine-tuning and so forth, and the architecture. And on top of that, the reasoning from scratch is the reinforcement learning with verifiable rewards and inference scaling to get more out of the LLM. And then also the mini-coding agent I just briefly mentioned, which is a wrapper around base and reasoning models. After these basic steps, once you have a feeling of how LLMs work, honestly, I would, in production, use existing libraries. So one I mentioned before is the Hugging Face Transformers library. And by the way, I'm not affiliated with any company or so, so I'm independent, but I'm mentioning that because I do believe that it's a useful tool and it doesn't really always make sense to reinvent the wheel. It is a useful exercise to implement LLMs from scratch but it's also mainly for education purposes to kind of understand the trade-offs and then in production I would personally use an existing tool. For training, for example, there is the Transformers library as a wrapper library. There is also Torch Titan, a pre-training library, and TRL is also by Hugging Face. It's for reinforcement learning and also other additional fine-tuning steps. And these give you access to more advanced features that would be very tedious to code. I mean, you can code them, and it's possible within a few weeks and months. But to be honest, if someone already did the work and spent many years optimizing that, I would actually use that. but it's also open source, and they optimized it to run on multiple servers, clusters, and so forth. So it's a pretty robust library. Oh, sorry. So I put this here as intermediate because wrapper libraries are cool, but they try to do a lot of things. So they are also still general purpose libraries, and they are like for, I guess, small projects. For real projects, people still build their own libraries based on PyTorch. For real projects, there are project-specific libraries. With that, I guess professional usage here, I mean people who actually trained a cutting-edge state-of-the-art LLM. Unfortunately, many companies don't share these details. for example, or Lama 4, GPT-OSS, you can't find the training source code. You can only find the architecture, but they don't tell you how they trained it because there's a lot of proprietary information in there, and they also don't want to disclose what data sets they used for legal purposes. But there are a few projects out there where they did train cutting-edge LLMs. For example, there is OMO by LNAI. It's not the, let's say, latest state-of-the-art model, but it's a respectable 32 billion model size. I mean, it was pretty good when it came out. And they do have their source code on GitHub. And then there is also the Hugging Face Small LM, Prime Intellect, and Nemotron 3, which are all really big models where these are models that have been trained in production, and they all have their different projects. And so the source code is all on GitHub. I think this slide is linked if you want to check out these projects later. But this is like for professional use and this is like how let's say companies trained LLMs. Yeah, like Alexander mentioned, I just finished last week my Reasoning from Scratch book, Build a Reasoning Model from Scratch. It took like one and a half years to write. A lot of work, lots of experiments in there. So if you are interested in that, it's right now up for pre-order and all the chapters are already available on the Manning publisher's website, and there is also a book signing that Alexander organized at four. So if you want to stop by, this is build a large language model from scratch. It's like the foundation model. This is the sequel with more reinforcement learning. It can be read as an independent book, so it's not dependent on the other one because it starts with the pre-trained models, but they go well together, I would say. So yeah, with that, I think, I hope I didn't go.

Speaker 1 [45:03]

No, thank you so much. Actually, it was part of my fault. Big applause to Sebastian. Thank you so much. I think we should do a Sebastian Russia conference.

Speaker 2 [45:18]

No, no.

Speaker 1 [45:19]

No, so we're running late. So all the sessions are scheduled at 11.45. Stay on time. We're cutting the coffee break short. Your Q&As are safe because we have a fireside chat in this very room after lunch. They won't be lost. We will address them there. Thank you very much. Thank you very much, Sebastian. Thank you, everyone.

Speaker 2 [45:40]

everyone. Thank you.

Speaker 1 [45:41]

Thank you.