Build an AI Document Inquiry Chat with Offline LLMs

Speaker 1 [00:05]

Hello, everyone. If you're late or you're not sure where you are, so just to make sure, you're at the Build an AI Document Inc. or you chat with offline LLMs. Anyone not in the right place? Cool. Most of you were here before, but if you have just arrived, there is a few instructions to get started. There is a little bit of infrastructure limit, and we're way over that infrastructure. If you do not have access, please make sure to sit down by side by a colleague and work together this is a workshop for us to learn so sitting pairs do not just stare but be proactive on that and with you Pavitra and Philip

Speaker 2 [00:49]

Thank you.

Speaker 1 [00:59]

All right. So I think we're at limit, right? We have 50 spun up. So I still don't see folks pairing up, but please do and follow along. Okay. So yeah, I'm Pavitra. I work as a developer advocate at Quonsight.

Speaker 2 [01:14]

I'm Philip, I'm a senior software engineer here and the lead developer of a tool that we're going to use later on in this tutorial.

Speaker 1 [01:24]

Okay. Just a quick side note, because, I mean, just so we know and we can speak the same language, so the platform that we are running this tutorial on is called Nibari. It is a customized Jupyter hub. And the only thing special it has is you all have access to GPUs, folks who are using this right now. And the environment that has all the, you know, all the packages and libraries we need, it should be auto-selected. Otherwise, it is the global global PyCon DE. So you should be able to select your kernel in a notebook, And you can select this one in case it's not at any point, right? So that's the only thing you need to know. And yeah, we can get started with the tutorial. You want to go? Just some goals, I guess. We'll follow this rough outline. So we'll start by running a local LLM first. I'll let you show them which one it is. Okay. Yeah. Then we'll introduce RAG. Folks who are not familiar, we'll go to that. It's a way to improve the accuracy of LLMs. Then we'll use the local LLM with Reg and Regna. And finally, we will put this all together in a UI. And you should be able to just explore and experiment with it. So that's the current flow. So hand it over to Philip for the first. PHILIP TANZIER- Yeah.

Speaker 2 [02:39]

Just real quick, since we are two speakers, if you have questions in the middle of the talk, just feel free to just raise your hand. Like one of us is always running around and try to help you. Of course, we're still just a single person. So if like five people at the same time do that, you just have to wait a little. OK, so the notebook you're currently seeing is the one marked as 00. And then we can switch to the, oops. That was not intended. Oh, it worked. OK, so I know LLMs have been the craze for the past year and a half, I guess. But just for the off chance that someone doesn't know what an LLM is or a language model is in itself, I'm just going to give a rough outline. So a language model in itself is just a model that, given an input, predicts the next output. So you can imagine it basically like as the autofill on your phone. If you have typed a certain amount of characters, it guesses what the next characters or the next words will be, and a language model is doing exactly that. If you put another L in front of it, so a large language model, these are the models based on this transformer architecture. This is basically where all the tools that you may or may not have heard about in the past one and a half years, like ChatGPT or the underlying LLM, it's called GPT-4 currently, there are so many more. This is the actual, yeah, language model that is driving these tasks. But yeah, so in general, the language model in itself is just a very, very fancy way to autocomplete text for you. So just when you see these models online and if you actually look into one, so I'm going But we're going to use one here in a second. There's one thing that you usually also want to have that's so-called, it's called Instruct Tuning. So if you ever look into LLMs, running them locally, you will always, or you often find ones that are marked with Instruct. This is basically just these LLMs trained, after they have been trained to auto-generate this text, they have been trained to actually answer questions for you. So all the LLMs that you will see in the wild that are able to answer questions, they are all Instruct tuned. Basically, just a fine tune after the initial train on, yeah, let's call them questions. Like you give a question and LLM gives you an answer. Okay. So, the ones that you or like most people are comfortable using are the ones hosted by a large company, be it OpenAI, be it Anthropic, be it, there are so many more providers out there. But this has some implications for you. You basically, if you develop your application just with OpenAI, then you're locked in into this vendor. You have the whole problem of data security or privacy in general, like if you want to use an LLM for your task and you need to send all the data to OpenAI, you better make sure that you're even allowed to do it. And you're not losing any kind of company secrets or whatever on the way out. And so these LLMs, in general, they are, of course, these companies have their own variants from it, but in general, they are not like magic. They're just language models that you can run yourself as well. And this is when we call it local or offline LLM. We're just going to take openly available weights, load them, and run an LLM on a GPU instance for you guys. And the one that we are going to use for this tutorial is Llama 3. So Llama 3 was released last week, so it was kind of nice for us to include it in the talk. We could have used a different one, like the talk was originally planned for Mistral, if that says something for you guys. But we're going for Llama 3, because so far it has been breaking a lot of benchmarks. And it's available on a permissive license. So if you want to use this commercially, I think you can unless you have more than like 700,000 users a year or something like that. So do read the license in detail if you want to use this commercially. But for this talk, we are certainly covered. If you go into running these models locally, you will very soon run into something that's called quantization. And the whole thing that is behind that is usually when these LLMs are trained, they are trained with floating point numbers, either float 32, which is the most common, but sometimes also float 16. And the number that I've just mentioned, float 32 or float 16, is the number of bits that you have per single value. And a problem with that is that these models are really, really large. So if you look into, like, for example, how Lama 3 was trained, it was trained on a data center of 25,000 GPUs, which each GPU having like 80 gigabytes of VRAM, like, unless you're meta, you're not going to have this kind of infrastructure. So inference is, of course, much, much cheaper. But you still need to load the weights inside your GPU to still to even be able to run it. And so just to have like a rule of thumb, if you see like a model that says, hey, I have 8 billion parameters, or any number of billion parameters, in this original representation, you need roughly like 4 gigabytes of VRAM per billion parameters. So for example, if you use the smallest version of this Llama 3 series, the 8 billion parameter model, in its original float32 representation, we would need 32 gigabytes of VRAM. And it is already exceeding most of the consumer GPUs available. So yeah, you can chain it, but even then, 32 limit is already pushing it. And we didn't even have the input itself, like if you want to process a prompt and so on, this also takes VRAM. So this is usually, yeah, it's done for the actual training, but after the inference, you do a technique, it's called quantization. What quantization does is basically it takes the float 32 representation, so floating point number, and breaks it down into a fixed number of levels. So you can imagine it just going from floating point to integer. And with that, we can drastically reduce the amount of memory that we actually need to load. And so just given the other rule of thumb here is we have roughly N times 8 gigabytes of VRAM for each billion of parameters, where N is the number of bits that we used for quantization. And so depending on how much VRAM you actually have available, you can select a number. Just to give some rough estimates, like eight bits per weight is almost equal with the Float32. So that is fine if you actually have 256 levels for each floating point number. Six seems to be currently a sweet spot. So this is something like between six and four is where all of the models perform really well for the amount of gain that you get from quantization. And so for this tutorial, we are going to use 6 bits per weight, and for that, for our 8 billion parameter model, we only need 6 gigabytes to load it in. That should fit in most consumer GPUs, or let's call it reasonable current consumer GPUs, and it's going to fit in the one that we currently also have provided for you. OK, so if you have quantization, there are a number of different techniques and a number of different libraries to load all of these. They are, let's call it, roughly equal in terms of usability. We don't have time in this tutorial to go in actual comparison between these different quantization schemes in different libraries, so we just picked one. The one that we use is xLama2 or xLamaV2. Currently, from the benchmarks that we have seen online, this is the one that gives you the best inference speed, best, I'll just let it stand in that way. Oh look, we made a typo. It's not Mistral, it's Llama 3. So you see, I'm not lying. We actually planned this for Mistral 7 billion. OK, yeah, and from that on, if you have a GPU available, and you should be able to have one right now, you can actually start running these cells. So if we import all of this stuff, don't worry, the import actually takes quite some time. The important part, or the expensive part here is Torch. If you know Torch, this is like a 2.5 gigabyte dependency. So if you import it, that takes some time to load all of this stuff. And from there on, we have very few things to do. So the first thing that we do is we need to provide this xLama2 library with a model that we're going to use. And we have selected one for you. You see in this name, you have the Lama3 8 billion. Then we have this Instruct tag that I talked about. This is an Instruct tuned version that we can use to answer questions. And then you have this EXL2, which is the quantization format that xLama2 uses. There are more. I've listed them here. You often will see something like GTPQ, GGUF, ABQ. They're like AWQ. There's a lot of formats. But if you want to try this with different models later on, the one that you are looking for is XL2. So we have these models downloaded in this chat directory for you all. Like I said, this is roughly six gigabytes, so you don't want to download them now on the connection of the conference, because we're going to overload it. Yeah, and then this year is a lot of boilerplate, where you set stuff up. We have to do it. But it's just basically setting up, OK, there's this model, prepare to load it into the GPU. How are we going to cache? How are we going to tokenize? Oh, do people know what tokenization is? OK, I see a lot of nots. Someone not familiar with what tokenization is? Yeah? OK, at least one. OK, now let me just give a quick explanation. So if you want to use a language model, be it a regular language model or a large language model, at some point you will have to translate text into math. Because these models only do math, they don't do text. And so tokenization is a way. No. What am I talking about? I'm sorry. I was at embeddings. Sorry. Yeah. So if you take a sentence and you need to tell, like the model internally works on tokens, not on words. So token is basically, let's call it like an index for a specific sequence of characters that is used internally to even present the sequence of characters. And tokenization is basically the step going from text that you have into these integer numbers. So every single LLM out there basically has their own dictionary to do that, or uses a different tokenizer. This is why you load the tokenizer together with the config for your model to have the right vocabulary that your model understands. OK. So the next step is we can load a generator. This is the actual thing that is going to do the generation for us. And then we have a lot of settings here to do that. So the important ones are, sorry, if you guys, because who had the question, if you're running into problems with the VRAM, that the VRAM is overflowing, you will need to have to restart your kernel. Because the problem is, if you load this and you leave your kernel running, the kernel that is running is hogging six gigabytes of VRAM. If you do this three times for three notebooks, then our 16 gigabyte GPU will no longer be sufficient to just load the weights. So at the very end, there is a node in here. Just stop your kernel if you're done with a notebook to basically kill the memory that this kernel is hogging for the model that you just started. OK, I was talking about the configuration settings. You have a lot of settings here. Internally, what happens is instead of just generating one next token or predicting one next token, the LLM actually predicts multiple ones and gives them a probability that these tokens should be the next. And you can select how it actually gives you back tokens. And so the temperature, the top K, top P, and this token repetition penalty, they are ways to influence how the LLM samples these. So if you, for example, set the temperature to 0.0, this means it will always take the one with the highest chance. This is good if you want to have determinism. But if you want to have creative writing, you probably want to have it generate a few more, let's not call it random, but a little bit more variety. And so these are the parameters that you can tune to do that. We're just going to show them here for a question answering context that we're going to move to fairly soon. We probably want to have determinism. So you will see in the later configurations that we use, we're just going to set the temperature to 0.0 to get determinism and just completely ignore the other parameters. So this basically means we have no sampling there. OK. And so now that we have that, we can actually start and generate something. So this is just taken from the inference script of the ExLama2 library. And so you just give it a prompt, and in our case, our story begins at PyCon DE, where once. And we just ask it to generate a maximum of 150 new tokens. And so if we do that now, we are going to wait a little. And it gives us an output. I'm not going to read it. Everyone that has a GPU instance available can generate their own stuff. But you see, for creative writing, this stuff is actually doing a decent job on generating a response. And then one thing that's also very common and is a lot better for the usability is you saw we just waited around four seconds for any text to appear, although the actual process is inherently sequential. So you always generate one token at a time. And so there's no need for us to wait for such a long time to actually generate an answer. We can actually stream it. Streaming is a little bit more involved. It takes more boilerplate to do it, but it is not really more hard than it was before. So we can try exactly the same prompt. And this time we're just using a streaming generator rather than the base generator that we saw before. We're starting our stream. We are taking one token at a time. You see we're looping basically here over these max tokens. And we're going to break early if we hit the end of it. So if you do that, you will now see we get output directly.

Speaker 1 [19:13]

Um, yeah.

Speaker 2 [19:13]

Yeah, when we hit when we execute the cell and since we had selected a few variables on top to actually do like Some random sampling you will see in general the the idea is the is the same like it conveys the same message But like since we have some sampling it will not be word by word and will not be Yeah, it will not be exactly the same Cool Yes, so there was a question like how do you select the tokenizer? So the tokenizer is very much bound to the model that you are using. So each model, maybe we can do like a quick tour here. So if you want to go into the actual weights that you saw, we are in this directory here. We are in the Lama 3 8 billion instruct XL2. And if you go in there, you will see at some point there's a tokenizer to JSON and there's a tokenizer config to JSON. If you go into them, you will see the parameters and the vocabulary that this specific LLM is using. like there are multiple LMs, if you have like a family, for example, whatever, like if you have like the Lama 3 family, they will all use the same tokenizer or the same vocabulary. But in general, you should not expect that two models should have the same tokenizer. Usually they just come with a tokenizer for each model and this is why you configure them in a way, like you just take this file and create a tokenizer from it. Like there's no real, let's call it standardization yet. There's no fixed format, but this is kind of the format that the community has agreed on. And so all the common libraries should be able to support it. Oh, yeah, sure. That's the wrong. So can you give me a thumbs up when it's large enough? I see one thumbs up. Some thumbs down. There is a thumbs down at the end, so let me just increase more, better now, okay. So I hope I can get everything on screen, cool, yeah, so this was it for the local LLM, so you already know now how to run a local LLM. Do we have for this one here a little more, yeah, yeah, let me just show you real quick where you can download these weights, actually. Like, we did it for you, because this is like a six gigabyte download, and we don't want everyone to download this now. . OK. If you go up, there should be a site how to do this yourself. Yeah. OK. . Yeah, exactly. Sorry. Sorry, we have the. I think it's the tab, the click, actually. click so I wanted to go there I'm sorry so I can Sorry not my laptop Yes, please I'm sorry Okay, so so hugging face. This is like if you are in the if you are familiar at least somewhat familiar in this LLM space hugging face is the central hub to share weights and On there you can find basically basically, just go to Hugging Face, like there is a .co, probably there should be .com. If you are on the Hugging Face hub, you can basically search for weights, in our case it would be Lama 3. And the Internet is slow. Yeah, okay. I'm just saying, like, there is an M missing, probably. Oh, Hugging Face, perfect. timing for this tutorial that they are done. Okay, okay, so at least it's not our fault. Yeah, okay, we can try again at the end, but basically what you're... It's up, it's working. It is working? Okay, well, we have like differing opinions. Okay, so there's a search bar on Hugging Face Hub. You could basically put in the model name that you want, for example, Llama3, and then you will see like a bunch of people that have taken the original weights, so the original weights published by Meta, and have quantized them. And so some people have a fine tuning on them, and sometimes you see an extra text called biomedical, or there's different spins on these original weights. And the way you do it is basically just clone the Git repository, which has these large filets in it stored in Git LFS. And then you can just point your library, this xlama2 here, to this folder you just cloned, and you should be good to go. Again, if you use a different quantization scheme, you will need to use a different library. Like each quantization scheme has some ups and downs, has a different library to use. If you want to stick to the slides that we have, just go with the XO2 weights. Okay. Any questions so far before we move on into like making... No, so the tokenization is bound to the actual model architecture. So, like, if you have Lava 3, it will always have a Lava 3 8 billion. Let's make it concrete. You will always have the same tokenizer. Quantization just changes, like, it's called the math inside library, but it doesn't change how the model understands the language in a sense. Like, it doesn't change how the tokens are used. I don't see any other questions. There is one. Yeah. Does Meta publish how deep the neural network is that you're using? Because basically you're using a fixed system, right? Yeah. And so they have decided originally how big and wide that neural network is. Yes, exactly. And do you know how, do they, is it part of the sort of attribute? Yes, like if we talk about these weights, let's go there again. At some point you have this large file which is called output.safetensors. This is actually holding like six gigabytes of weights. and you need to have, like, an exact plan how these weights are arranged to get the exact behavior back. So the model architecture, of course, for these open models is always known. For models like GPT-4 or Claude from Anthropic or something like that, it's guessed. Like, there's a lot of chatter and a lot of people have opinions on how it looks like, but this is more company secret, so we don't know exactly. I mean, most providers, they publish, like, we have used so and so many parameters, or we have used, like, multiple experts, it gets pooled, stuff like that, but you don't really get, like, an answer on how this stuff works exactly. Well, it's their company secret, so you shouldn't expect to. But like, for the open ones, it's always 100% known how this works, otherwise you wouldn't be able to run it. More questions? Is LAMA one of the models where you know exactly how the tension structure looks like? Yes. So like I said, everything that you are able to run and download from Hugging Phase, this is exactly known. Yes. Sorry, another quantization question. Yeah. So do you lose information by doing that? Absolutely. And do you have to, when choosing a method of quantization, do you need to take into account how you want to use that model? How is it done? JAN-FELIX SCHWARTZMANN- Well, to answer the first question, yes, you absolutely lose information. The question is then, how much information do you actually have encoded in the float 32 representation in the first place? Like I said, with 8 bits per weight, you can almost match the float 32 performance, meaning you don't have more than eight bits of information encoded in the number in the first place. Of course, you will never match the exact, but it's It's just as close as you get to the actual number, the better you will be. Like I said, eight, it's already like a factor down like of four. Like just in the bits, of course, like internally it's a little bit different, but like, yeah, you lose information, but like I said, between four and eight, there is a sweet spot where it's... Yeah. You... Exactly. Like the way it's usually is presented is you have like a graph where it says, okay, okay, if I use whatever, three bits per weight, which is really, really low, like, using a metric and I compare the original weights against my quantized weights, like, how much performance drop do I have? And you would see that, like, for six bits per weight and above, these two lines are basically a match. They will not be the same, but, yeah, you can, of course, check for the use case that you have. Like, the way that I've seen it most is just use as many bits per weight as you can use that your GPU will hold. So we do like a rough calculation. Like for our case, we have like 16 gigabytes of VRAM available in GPUs that we provided. And then you can say, OK, I can fit like with like 8 bits per weight, I can fit like 8 gigabytes in there. I can use this and still have like 8 gigabytes left for the prompt and the actual work that the GPU is doing. So you should be. But if you use larger models, this is a lot more an interesting question, where it's like, OK, I have to have this trade-off between how much context can I actually put into my GPU, and how many bits per weight can I use? And this is where you need to trade-off. For these small models, it's usually just use as much as you can. Yeah? If I want to trade on specific type of data, like biochemistry, right, can I use the tokenizer to tweak the way that it doesn't tokenize certain specific words in biochemistry, but it does tokenize other words. In other words, can you basically adjust the data that is being used to effectively? Well, the tokenizer is just used basically to translate your string into IDs that the LLM uses. If you change the tokenizer in the sense to drop words, then you're just basically leaving out information for the LLM. The question is, like, there's probably no need to do this on a tokenization level. You could probably do it before if you say, okay, I don't want to use this. The way it usually works, if you want to have, like, a more biomedical context, for example, what happens is if you have the original weights, they're trained on, like, an absolutely massive corpus of data. And it just is, like, a very generic data. It's just all kinds of language. If you want to have this more, give it more like a biomedical spin, you then take like a biomedical corpus and just fine-tune these weights on this. So basically, to change the probability, if I say this word in a biomedical context, the next word will most likely be this rather than some other word that it is in a generic context. So this is what you will see if you have this extra tag on a hugging face that's called biomedical or whatever you have on there. So this is usually then fine-tuned on a specific corpus for a specific use case. It's not going to say that it's not going to work on other use cases, but it's just then going to, well, I'm calling prefer completing this biomedical context. Yes? So you want to talk about drag, right? Yes. You mentioned about these tags for domains such as, for example, biomedical. Yeah. So how do you compare going for a RAC, for a known knowledge base, versus these tags? So what are the pros and cons? I mean, of course, the RAC comes with only the known sources, and tagging maybe is fine-tuned with the larger or unknown sources. So how do we choose when it comes to should I go for a fine-tuned for biopsy or should I go for a RAC? So we're going to talk about Rack in a second, just to give you the generic answer for how LLMs work. You have the actual weights, which encode something like, let's call it, long-term memory. This is something that the LLM has learned, but it's something that it very easily tries to hallucinate on. Imagine yourself getting an answer for something that you have learned like 30 years ago and haven't used since. You probably will find the right things, but it's very easy to elucidate stuff around that. So fine tuning is usually there to make it easier for the model to get the language right. So to get like, OK, if I have this term, it's probably this term next, but it's not really a good strategy to encode actual knowledge. You don't want to train it on your corpus and then just ask it about questions about this context. What you do then is actually, instead of just asking a question, you give the model more information to reason about, saying like, OK, here are x documents. Please give me an answer on these documents. And I think this is a very good bridge to our next notebook. If no one has a question, I would just move over there and then explain what RAG is exactly for this use case you just mentioned.

Speaker 1 [33:51]

Okay. And I will remind you to just stop the kernel on this notebook and close it before we move on to the next one. Let's go to... Okay. So just to get like a read of the room, How many of you have heard of RAG or used it before? Okay. So this section is going to be pretty much useless. But I guess for folks who are not familiar, RAG is a different way, like I think Philip just explained how fine-tuning is also a way to improve the accuracy of LLMs, but RAG is a way where you can provide documents and context to the LLM and then, you know, ask questions and get answers about those. So just, I guess, to illustrate why this is needed, this is just chat GPT, and we asked what is PyCon DE, which it was, I think, trained on because PyCon DE has been around for ten years, as we heard today. However, it did not know when was PyCon DE 2024, but it's clearly not September 23rd today. So what we can do is is use rag, right? And we'll see if we can do better. So, an illustration of how and like how rag works, like the technique. So the user, we have two things to do. First, provide context, relevant documents. And two, the actual prompt, right, to the LLM. So what happens is both the documents, first the documents are tokenized and vector embedding is created and put it in a source storage. So in Ragna, actually, we use the source storage vocabulary. It's a vector database where, you know, that's a term you'll be more familiar with. And the same thing happens with the chat, right? And so with the chat, with the query that you have about these documents. However, once the query is also tokenized and an embedding is created, the relevant chunks of your document that are related to the query are pulled out and provided to the LLM. So the relevant chunks are taken, they're passed to the LLM, and the LLM also, of course, receives your question as well. So you, ideally, if this works correctly, you have a more accurate answer that was based on the documents that you just shared, right? So is that clear for folks who are not familiar with RAG? We can move on, right? So that is RAG. Next. What is RAG now? So this is the tool that Philip created. So Ragna is a RAG orchestrator where you can use different LLMs and source storages so you're not bound by them, right? And Ragna comes with a REST API, Python API, and a web UI. So it lets you use all of these. And I guess within Ragna, what you have is a scalable task queue. So you get to play around with these through for today. And for today's tutorial, we'll be using Python API, so using that with any LLM out there. So we'll use both OpenAPI's GPT as well as Lama, Lama 3, to, yeah, to build things. And today we'll be building a chat interface, chat prompt. Okay. So any questions about orchestration? So the second half of this is just the other things you can do. So the UI, as we said, is not in Ragna, right? So Ragna has the Python API and the REST API. The web UI is something you can build with the REST API. Similarly with the IDEs, today we'll be using Jupyter Notebook as the IDE to connect to the LLMs and the vector storage right so any questions here how does it determine which documents are relevant I think that's where the example here was similarity search but do you have a specific one

Speaker 2 [38:17]

It depends how deep the question wants to go. The question you're asking is about the vector databases. Or you put in a question, you want to know which are the relevant chunks, which are the relevant parts of the documents that I have stored before to pull for this. I'm just going to give you the answer for vector databases. There are millions of techniques out there to do this, but the ones that you will find the most in the context of RAG is called vector databases. Vector database, like the name implies, it stores vectors. And instead of doing an exact search, like searching for I have this vector, give me this exact vector, it's like I have this vector, please give me 10 that are most closely related to this vector. And so if you have, this is what I wanted to say earlier, but I mistreated for a tokenization. Embedding is the way that you go from text to a vector. So you have a text, and you want to represent this as numbers. Because in numbers, you can fairly easily resolve something like, how closely is this vector related to a different vector? We have tons of metrics out there. If you use L2, if you use cosine similarity, there are tons of metrics to do that. But you need to have an embedding model first. You basically take the text, you embed it into a vector, you store all your documents and chunks into this vector database, and at runtime, you take your prompt, embed this as well, go to the vector database, and say, hey, give me the 10 closely related vectors to this. And depending on how good your embedding model is, and depending on how good your question's phrased, basically, this will turn up more like good or bad sources for this process. Does it answer?

Speaker 1 [40:23]

Okay. Then I guess we can get started with just first steps of Ragnar. So we'll be using GPT-4 for this notebook, right? And you need API keys from OpenAI to use GPT-4. So we have shared, like, folks who are on Nibari will have access to a key. But for your, this notebook actually can be run locally if you want, right? So what you'll need to do is get a key from OpenAI, so there's a link in the material, and export it on your terminal before you start Ragna. So, and yeah, you should have access to GPT for them. Does that make sense? Okay. So for now, we'll just load the key from .env. Okay. And, okay, so for documents, because the initial question was about PyCon DE, we decided to share both the FAQ page and the PyLadies pages. This is just the website pages actually downloaded as PDFs and shared, okay? But if you are curious, there are more files available in...

Speaker 2 [41:43]

Uh...

Speaker 1 [41:45]

There are more files available that you can add to this if you want to ask questions. Yeah. So now we are switching to the GPT-4 API. Yes. But before, the notebook before we were using... Yes. Preparing for the local. Yeah. And we're using that now. Yes. And we cannot, obviously we cannot use GPT-4 locally because we don't have the way. Yeah. That is true. Yes. So we did want to share both. To be clear, this notebook is only going to use GPT-4. Yes. notebook, we used a local LLM. In the next notebook, we'll use a local LLM with Ragna. Got it? So this is just to keep the Ragna syntax simple. That's why we are not using the local LLM in this notebook and introducing Ragna first. Okay? All right. So we will use those documents and select assistants. So in Ragna's vocabulary, we use assistants for LLMs and source storages for better databases. So because we shared the OpenAI API key, We will have GPT-4 and GPT-3.5. Just importing those. These are shipped with Ragna, so Ragna has support for these. And several other assistants, which you can actually take a look at in the docs. You just need keys from all of these. These are vendors, right? So you need API keys, and then you should be able to access them. So going back. And for source storages, currently Ragna has support for Chroma and Lansdb. Okay. So the first step is we create a rag instance. And we do rag.chat where we provide documents, the source storage, and assistant we want to use. So for this particular chat, we are using Chroma and GPT-4. And then Ragna is asynced by design. So there's always a prepare step before you ask questions so that things can start running. And sometimes this takes a while when we have documents and, like, large documents. So you create the embedding and things that will start automatically once you start executing this. So the current documents are fairly small, but if you have something like 50 gigs, it's going to take a while. Okay. So that should be it. And now we can ask the same question. Let's see what answer we get. OK, yeah, that's correct, I think. And next is?

Speaker 2 [44:36]

What the, like, Pavitra already has showed it, like, you have the ability to put in any kind of source storage or vector database and any kind of processing pipeline in there, so this is up to you to choose, but yeah, like, from all the built-in stuff, we don't have anything over the other vendors as well. What we've seen in projects that we did so far is everyone had their own way of doing stuff and parsing their own documents. And every document or every document type has their own special requirements. So it's really, really hard to find something that works well in general. So we are providing the basics to pass basic PDFs. And like I said, this is not special for Ragna over any of the other tools. But what is special in a sense is you can strip in or you put in your own pipelines that you want to use, and you still use the same API, use the same UI, and we will see in a second. I think that's the hard engineering part of knowing how to process things. You can see that, for example, with Lama Index, that's how they monetize their product now by providing a PDF of your idea. Yeah, and I mean, at that point I can probably do 30 seconds of promotion. Like, we come from a company, Quonsight, and we offer consulting services around that. So if you have the need for a RAC pipeline and you want to have the knowledge and want to deploy it yourself, then this is something that you can speak to us about and we can hook you up with that. But like I said, the whole project in itself is fully open source. You can do it yourself. If you don't have the knowledge and you want to have help with that, we are here to help. OK, that was a different question? Yeah.

Speaker 1 [46:40]

plain text

Speaker 2 [46:42]

Yes, like if you try to parse PDFs and you have actually looked into what you get out, it's actually really hard to get, let's call it relevant information out of a PDF. So yes, you will of course get all the text but you will always get a lot of extra tokens or like extra signs that were put in there. So I would say like if you can create a tool where I give you like a PDF, like a randomly formatted PDF, whatever it looks like, and you can actually just extract the text, you can be the next big thing in this whole AI. For real. You will get a lot of money if you can invent such a tool, but this also shows you how complicated this actually is. For all the projects that we've done, it's usually very much driven for this specific use case on how you actually extract the data. So you try to find some kind of similarity between the documents, for example. If you always have, let's call it, medical documents, or all your documents are formatted in the same way, then you can say, OK, I often see this phrase. For example, if you have a lot of letters, you can just try to remove all the letter heads from it. Because that is stuff that also gets pulled out if you just pull all of the text out of a PDF. But I can't just build a remove letter head PDF extract into Ragnar, because each letter head is going to be different. And so, yeah, like the previous commenter already said, this is the really hard part when working with data. Yes, over there's a question. Sorry, I didn't want to take this over. So could you, but how's your experience Are you working with tabular data, like CSV files? SIMON VERSTRAETEKER- Similar. It's hard. It really is hard. I mean, tabular data in itself, if you really have a lot of tabular data, LLM slash RAG is probably not the tool that you're looking for. Tabular data means structured data. So if you want to ask questions about that, you're probably better off than in just reading, for example, in the CSV file, reading the header and giving the LLM a specification saying, okay, this is what my data looks like, and then, for example, ask it, hey, write me some codes that I can analyze this data with, and then just run the code yourself. This is probably the better pattern. So RAG is usually the best pattern in itself if you're working with unstructured data text. GPT-Efficient is pretty good for GIFs. Yes? I honestly haven't tried it yet, but like, yeah, this is also a point. But then you have this whole, the whole point of this talk is that you don't want to get locked in by open AI, so you will probably need to look for an open source model, or look for a model that works on multiple vendors. Yes? So I'm guessing you're using Chrome on RAM, basically. You don't have the data as persistent. Do you have examples later where you basically persist? JAN-FELIX SCHWARTZMANN- So then my first nod was wrong. We're not using it on RAM. We're using it persistent on disk. But it's not like one shared instance for everyone. But for example, I noted for a fact that Chroma has a hosted version where you can just host it in your data center and just connect to it. And so the way that that would work is just you would create your own source storage, call it whatever hosted Chroma, hook it into Ragna, and then just everything works as before when you have done one central, one central. Yes? Are you dealing with metadata from the PDF? Currently, no. Not by default. We are in the process of adding something like that, but currently not. Yes? I know Amazon has a model which can extract text from PDFs. So will it work, for example, in combining ? Sure, yeah. So right now, it is a lot more grouped. We have the whole embedding and chunking pipeline inside the source storage, but we already have PRS up. It's not released yet, but we have the code there to basically split it out. And what you are basically saying is you have this document handler. This is the term that we use inside Ragnar. Give it a document, and give me output of the document as some text. If you have your own custom thingy there, you can just put it in and go from there. That's the whole point of Ragnar, to have this orchestration mechanism where it's like, OK, you can select the parts that you want. If you want to use this model, then go for it. Yes? How do you control the chunk size here? The what? Chunk size. Chunk size, yeah. So Ranga internally currently has the chunking, like I said, the chunking strategy is unfortunately currently fixed. We are building on making it more flexible. Chunking is done on a token level. And so what you can do is here, I mean, we don't have it yet, but it's in. It's in edit number four or something.

Speaker 1 [52:17]

Who has it?

Speaker 2 [52:19]

Shoot, I'm sorry. It's not my notebook. So for Ragnar, this is currently the way to set the chunk size based on tokens. Like I said, we're currently refactoring that. Can we, is it important a question to have it now, or can we let Pavitra finish the notebook and then I'm happy to answer questions at the end?

Speaker 1 [52:49]

I think we were pretty much at the end of this notebook anyway, so it's fine. Yeah, we were just going to look at these sources used for this particular prompt. Just present it in a fancy way, that's all. What we're looking for is answer.sources and answer is what we got here, right? So, yeah, the sections of the PDF used to answer this prompt, that will be displayed to you. If you use that, I'm just going to minimize it. Okay. And the final thing is streaming. Because Dragon SAsync, we can stream answers pretty quickly. And what you do for that is set stream equals to when you do chat.answer. And that actually should just work. So we are printing just for each chunk in your answer. And this is one final prompt, not doing anything, but just because we wanted to add one more, that's all. But I let you, I guess, use the prompts that you want and see how this works. So that is the end of this notebook. So if you did have one more question, we can get to it right now. Okay. Okay, them first, yeah. Here, I think let's just use this.

Speaker 2 [54:27]

Yeah, right right now. We are going by number of tokens for 40 sources So the splitting happens by the number of tokens so for example by default we have like a token limit of 500 per chunk and then at the end we say okay I like how many tokens do you want to put into the into the prompt and say like for example? 4000 then which means like we have like eight eight source

Speaker 1 [54:51]

sources.

Speaker 2 [54:54]

Yes, of course. Yeah, so we will have that in a later tutorial, but so there is something about the advanced configuration where you can basically set these parameters to whatever you like. Yeah, currently the behavior is hard coded. With the next release this will be much more flexible and you can define your own scheme basically. Yes, there's a question back there. Yeah? Do you overlap document chunks? JAN-FELIX SCHWARTZMANN- Yes. If you see here, we have the chunk overlap. This is something that we realized was pretty significant, because if you don't have overlap, you will very easily have chunks that are disjointed, and it's very hard for the LLM to reason. Imagine you have a paragraph about something, and this paragraph is just split into half. It's really hard for the LLM to reason about it if you to have this paragraph actually split up into two parts. And it's also hard for the embedding model to really capture the essence of the paragraph if you split it in half. So this overlap is something that we experimented a lot with in the beginning. And again, I can't give you really close numbers for your specific use case. But what we found is just having roughly 30% to 50% overlap just gave us the best results. In terms of trade-off, of course, more overlap will give you better results in the sense that the DLLM has an easier time answering it. But this also means you have a lot more vectors in your source store, in your vector database. And it also makes it harder to find the relevant one. Imagine if you have 90% overlap, and you ask a question, and you basically will get three times or four times the same paragraph out of the vector database. Because if it's looking for one sentence or whatever, then this sentence appears in a lot of chunks. OK, one more. Yeah, I'm sorry. I can't go by, I tried to go by occurrence. So yeah. So I just have a question because you said that there will be an update for what happened in the release. So very soon where it will have more flexible touches, more really quality when that . So you're asking me to give like a release schedule now. So we have a large PR up. It's still pending review because I needed to prep for this tutorial. But I'm hoping we can get this done in the next few weeks. There is more stuff that I want to change for the next release. Like, I'm happy to give you more of an in-depth answer after this talk to what we're trying to have on the roadmap for the next release, but I don't want to say a date yet. Okay. Next question. Back there. I'm sorry. Let me... Can we give the... The mic? I have one. The mic. One, one. In terms of reg, pump pipeline in Chang Corp Olympic and vector databases, how graph databases fit in? Could you repeat what you said after how, how what? How graph databases fit in? When to use them, how to use them, why to use them? I'm sorry. Could you repeat the question again? I'm still not getting it just. When you're talking about chunking and reg pipelines and data ingestions and creating vector databases, instead of using vector databases, when and why to use graph databases with elements? And that's exactly the reason why we didn't call the abstraction vector database in Ragnar but rather source storage because we don't want to force you to use vector databases. In general, we're going to, well, we're going to just implement the assistant but the only two, let's call it features that any kind of database would need to have to work with Ragnar is we give you a few documents and ask you to store the information about it And later on, we are going to give you a prompt, and you give us back relevant chunks of it. If whatever you have, the graph database, whatever database you have, can implement this kind of interface, you can use it with Rack. We're just used to saying vector databases because that is the part of this whole, let's call it LLM craze. Like, everyone is using vector database. Whether this is the right choice or the wrong choice, that is very much dependent on your use case. Like, vector databases is a very easy choice because you have, if you have embedding models, This is just very, very easy to set up. But in general, you can work with any kind of database, like I said, that implements this interface. Yes, yes, I know. But in your experience, when you're SLAVA TYKHONOVSKYI- Oh, sorry. Oh, yeah. I don't have experience with graph databases. Sorry to answer that. Yes? I have a proposal. Can we hold advanced questions until the end? SLAVA TYKHONOVSKYI- Sure. We have enough time to . SLAVA TYKHONOVSKYI- Happy to. Yeah. Cool. No, there's no feedback loop, but we will see how this works exactly in the next notebook. Yeah, let's do it. Okay, so the next notebook is about we now know how to implement like a local LLM. We now know how to use Ragnar. Let's use the two together. Okay, give me a second. So the abstraction class that we provide over LLMs in Ragnar is called assistant. And unlike the source storages, we, for the assistants, we only have like a single method that we need to implement. It's called the answer method. The answer method gets a prompt, which is a string, and it gets a list of sources. Sources is like an object that we define in Ragnar, but like you can just for this purpose of this tutorial, you can just kind of imagine it like a list of strings that we pulled out of our source storage. And we're just asking this assistance to provide like an answer, to take these two parts of information, combine them, and generate an answer from that. We have done it already here and it's called in a file, it's called local LLM. I'm just going to expand it here to, for you guys to see. So what we have is this very, we are subclassing from this assist, oops, I keep pressing, that's That's not what I want. So we have this, our new assistant subclassing from our base class, our abstract base class. We can define a few things that makes it easier for Ragnar to work with but we don't have to. So we have like a display name, you will see later on when we load this into UI, we actually have like a human readable name for it. We have some requirements like Ragnar's automatically going to check if these requirements are available before it actually starts to load this kind of stuff. In our case, we just have two packages that we want to load. We also want that CUDA is available when we load this, so this is like the back end for or the accelerator back end so we can actually run this on the GPU. Then in the constructor for class, we're just going to load exactly the same code that we had before. One thing that I touched on earlier here is the temperature. So like for rag, usually you want to have determinism, you want the LLM to always answer exactly the same that you, whenever you do like a run a second time. So this is why we usually, or you will see like the temperature is always set to zero so no sampling is going on. And yeah, then I'm going to skip this because we're going to talk about this in a second. Then we have the answer method. So the answer method in Ragnar, you should probably also be already familiar with the code because we had it earlier, it's just this generator starting a stream, looping over the number of tokens that we want to generate and just yielding back the results and yielding because we want to have streaming. The interesting part is here, we have our tokenizer and this tokenizer encodes not our prompt directly but something that we call make prompt. And this is basically where the, let's call it the magic of Rack happens. This looks very complicated, it is complicated to write, it's not hard to understand. What happens here is Rack, the whole pattern in Rack works like, instead of just sending the LLM your prompt, you're taking the sources and put them in the prompt together with the prompt that the user has. So like, let's call it, you have a user prompt, you have sources and you have done the prompt that you're actually going to send to the LLM. It's just basically string concatenation of these two, including some instructions. So what you see here, for example, is answer the question based on only the following context. This is usually what you refer to in the other system prompt. You can put more stuff in there. You often see, you are an honest assistant. Do not answer with anything. Do not make up information, whatever. This is something that you can customize to your need. This is basically trying to tell the LLM how it should behave. And the important word here is trying, because this is just instruction that you gave it. Whether or not it's going to follow it, it's like a totally different story. So this is the whole, let's call it, discipline. I'm not sure if it's a discipline. It's more like an art form. It's called prompt engineering, where it's like, phrase the prompt in a way that the LLM is actually doing what you want it to do. And this is very much dependent on what kind of LLM you have. And, like, for example, if you use ChatGPT, like, whenever they do an update, you would probably have to redo all your prompt engineering or prompt processing because, well, potentially. Which is also, like, a good idea to have this stuff running on your own hardware because then you control updates. Like, we had a project where a client was super happy with what we provided and one day to another day said, like, oh, this whole system doesn't work anymore. And we tried to figure out what happened. And then, yeah, like an OpenAI update happened. And so questions that worked before didn't work afterwards anymore. And there was nothing that we could do other than saying, like, okay, you guys, like, if you want to have it this way, you're locked into OpenAI. You're basically under their thumb. Like, if they change something, that's, there's nothing we can do. You should probably look into solutions to host this stuff yourself. Yeah. So you see a lot of these special tokens here. This is like these HTML tags with these vertical bars in there. Every model has their own template on how you need to format this. This is usually in the, like, I'm not going to try to go to Hugging Face again, but, like, you see in the model card, there's usually an explanation in what form this model expects the prompt to be. And you just follow it, you program it once, and you never touch it again. So this is kind of like, like I said, it's hard to write because you need to be very specific, like the models are usually very specific about this kind of format. And if you have it once, then you don't need to touch it again. And they are very picky about this format because if you've seen this kind of format a billion times already, you expect it to be in exactly this format a billion and one time if you give it a new prompt. So this is why just basically copy and paste is exactly like they have it on the model card and then go from there. Yeah, and like the only thing that you see in here that is interesting for us is we're just going to list the source content here, and we're going to have our prompt in there, and that's it. The rest around it is just the way that this model, Lama 3, expects the prompt to be. And I already also put it down here. Like, apart from this make prompt function, this assistant that we provide is actually like an xLama2 assistant. You can basically plug in any model. It's like this thing is model agnostic. except for the make prompt function. Like the make prompt is what you would need to change on a per model basis because like each model expects something different. So this thing is actually called Lama3 instruct but you could basically use it for other models if you change this one function. Okay, so we're just going to import this one here. We're going to see the display name. Display name is this one here, Turbo Derp is the one who quantized this model for us. There's also the maintainer of the xlama2 library. We can check if this is available and I hope this go through because if not, we're going to have a problem and can't run it but like it should be. Yes, perfect. It takes a long bit to actually check if all the libraries are importable and stuff like that. Okay. And from here on, we can use basically the same pattern that we saw in the last notebook. We are just going to provide a few documents. We are going to start a chat and instead of using like the built-in GPT 3.5 or GPT 4 that we have, we are now using the one that we have just implemented. And now, we can do exactly the thing that we have done before. Let's prepare it. Like Pravitra told earlier, this prepare step is like embedding the documents into our vector a database, so this can take some time. We have, by default, a fairly small embedding model that can run on CPU, but of course, you can use either a large one running on the GPU as well, or you can use OpenAI, for example, also has embedding models. As an API, you can use them. That's totally up to you. Yeah, and so the prompt, I mean, I can't show it to you in full detail, but here's how I'm going to read it. Who is the Python developer in residence? And we're giving it the annual reports of the Python Software Foundation. Let's see if the model can handle this. If someone is familiar with the PSF, the answer is actually correct. So I'm not going to show you the sources. Of course, you can take the message again. You see the sources that were pulled out. You can verify that this information is actually correct. Which is, we haven't really, I'm not going to talk a lot about this, but this is one thing that is really important to make Reg useful for in a production setting. You cannot really trust an LLM to do the right thing. So you will always need to have an option to verify that the information is correct. And Regna makes sure that you always get the sources listed along with your answer for you to have a way to say, OK, Where is it actually pulling these numbers from? Numbers is especially one thing where it really goes off the rails really easily. You will see it in the next notebook. That happens quite easily. Yes, please. How would it deal with information in the documents? That's basically the answer. It really depends on an LLM. I've recently seen videos where you can guess. Someone tried to gaslight GPT or chat GPT into answering 2 plus 2 equals 5. And you just prompt it over and over again. And at some point, it just gives up. And it's just, OK, if you say so, 2 plus 2 is 5. OK, I'm just going to roll with that now. And so in the end, what you really need to remember is this thing doesn't think in any capacity. It's just a way of generating token after token. So at some point, if it thinks that the next best token is just something that contradicts, for example, what it said earlier or whatever, it's just going to output that. This is why I've talked about the sources. It's really important for you to have a way to actually look into that. If you see talks about this, this is one thing that is really important with all these React pipelines, if you built them, you need to have trust in the way that this LLM is actually answering stuff. And giving the user a list of sources that were used where they can actually verify that the information that was sourced is a very good way to do that. But there will never be 100% accuracy or whatever. Yes? As far as I understand, when you call chat.answer, it actually creates a prompt and fits all this relevant information in the prompt. Is there a way to debug it and see the full prompt that was fed in the model using Ravna? Sure. I mean, we have it. Like, if you open your local LLM thingy, instead of just returning here, just print here. Or, like, what level of debugging are you looking for? Yeah, I just want to see the full prompt. Yeah, happy, like, this is, you have these files on, like, on your Nibari instance, just go ahead and hack at them and try to see what's happening. So, the source code for this is actually in this local LLM file. Ten minutes left for this or in general? Oh, shoot. Okay. So, then let's quickly come, like, the questions that are still open, can we move them at the end so we can at least show, like, a little bit of the last notebook? Thank you.

Speaker 1 [73:47]

So just stopping the kernel here, and we can go to the last notebook, which is doing everything we just did, but with the UI. So here, there are a few steps you need to follow, so try to, I guess, do it along with me. First off, in your in the file explorer, you should see a ragnar.toml.template file, right? Yeah. copy over all of this into a new file called ragnar.toml. So we need to do all of this only because we are running Ragna on Nebari right now. Like, on JupyterHub. So we need to do some proxying to make this work correctly. On your local machine, you should just be able to run Ragna UI and it should work as expected. Yeah. All right. So copy it over to a new file. And then if you look at API root path and URL, right? So under API, this is Tomo, so API root path and URL, there is a user field. Just make sure you have your username, like the one you use to log into Nibari. It should also be available, like, in your URL itself, like, your browser URL. Okay? So there are just two fields you need change which is root path and URL and save this file. Is that clear? Also the URL. In a...okay. All right. So just repeating the steps. There is a ragnar.toml.template file that all of you should have, right? Yeah. So copy the contents over into a ragnar.toml This is a template that everyone has that's similar, yeah. In your local one, which is ragnar.toml, just update the user with your Nibari username. So that is the one you got from the spreadsheet, right? And if you forgot what username you had, it should be visible in the Nibari URL at your browser level after user. So mine is this. Does that make sense? Should we also get a name, when, inside of com or just the name? Oh, no, no. That's my username is the entire thing. So this, yeah, yeah, yeah. Okay. So just save this file. And then we should be good to go. All right. So let's open a terminal window. So this is JupyterLab. You should see a terminal at the very bottom of this launcher. I'm just going to move it to a separate side so we have these together. So two things. First is just activating the environment. We've been using the global PyCon DE. So copy that over to activate the conda environment. And shedding this file to Ragna UI. That's what this is doing. This is a little bit of, like, patching script that's running ragnar.ui at the Nibari proxy that we needed to run it on for folks, right? Once that is running, oh, oh, right, we've, okay, this is going to take a little bit longer. This is because we forgot to change from Mistral to Lama in our Toml. So in your ragnar.toml. So, okay, I'll actually take a step back. So Toml is, this file is how we configure Ragnar itself because Ragnar is an orchestrator. There are a lot of things you can configure. And this is, this wasn't really necessary for the Python UI. But when you use the REST API, this, the config file is where all the information is pulled from. So if you see, we have source storages, we have assistance, list of things here. So when you use the REST API, these are the ones that will be available. So here we have Mistral. We will need to change that to Lama. I mean, it didn't work. Yeah. Oh, it's updated for you? Yeah. Okay. Cool. But, yeah, I thought we pulled right before. Okay. I will change it, though. What was it called again? Oh, my God. zoom level is hard to work with. Okay. That's the one. And I will change it. Okay. So now it should work for me as well. Once this starts up, you should see some notes from like notes about just server starting up and such. And then go to this link. Again, like just because this is, all of you have a separate instance, you need to change the user field here to your Nibari username. And, yeah, if you go to this URL, the Ragna UI should be up and running. Did mine work? I need to change that. Oh, I'm seeing an insufficient VRAM. Again. I'm going to restart this one. Yeah.

Speaker 2 [79:14]

What I've seen a few times is if you just click the stop kernel in a notebook, it's not actually shutting down. If you see in the file explorer, next to your notebook, if there is a green dot, it means this kernel is still running. Right-click the notebook and just do, like, there's a shutdown kernel. And if you click that, you have to wait for a few seconds. If the green dot disappears, then the kernel is actually shut down correctly. So you need to have that before you can move on. Oh, I had a question.

Speaker 1 [79:42]

Oh, I had it open as well, OK. Here, let's resize this, right? Third time's the charm. Anyway, so if this is running for y'all, yeah, you should just be able to go to the corresponding URL. Well, we can take questions, actually, while we wait for this to spin off. Oh, there you go. It started. All right. And now, oh. There we go. All right, yeah, so that's what the UI looks like. I think this will be blank for you. You won't have any past chats. This is the things we had. So yeah, the UI, this is something Ragna ships with. And locally, as I said, it's Ragna UI to run it. And we can start a new chat from the UI and select the list of assistants. So we had OpenAI previously, and we had Lama3 that we worked on just now. and pick a storage. So the demo assistant and demo storage you see here, they are not actually doing anything. They're just presenting the same thing in the right format. That's all. It's just for you to make sure, for us to make sure that Ragna is working correctly on a computer. It's not an external issue. So those are not actual LLMs or actual databases. Those are just gonna print the same prompt, if I remember correctly. The Ragna demo assistant, that'll just print the prompt back, right? Yeah, it's not actually. So we'll use something else. So let's use our local one, and we can give it some files. Oh, OK. So there should be files on Nibari, like in the files folder, right? You can probably download them locally and add it here. I think I just downloaded the PSF annual report that I will drop, all right? And start a conversation. We have time? That's time, right? OK, we are at time. So I guess, just to complete this, do you want to ask any question? Should I ask a question? OK. OK. Oh, OK. So you want to do that? So please, Lev, that was an awesome workshop. help me with giving a huge amount of applause to those two. And we are over time for a coffee break, in case someone is hungry. But they will be in the room for more about 10 minutes. And then after that, we have another workshop. So thank you very much. Thank you.