Incorporating GPT-3 into practical NLP workflows

Speaker 1 [00:03]

here and watching remotely. When I submitted this proposal it was called GPT-3, incorporating GPT-3, but since then there have been quite a few other large language models so I took the liberty to rename the talk to make it more general because everything I'm going to be talking about will also hopefully translate to other models and the experiments we did those were done with GPT-3. So yeah, just to get started, I'm Ines, some of you might know me from my work on Spacey. Our company, Explosion, develops the library and we have been since 2015, which is quite a long time ago. Spacey is an open source library for natural language processing in Python. It's been very popular, downloaded a lot, and its mission has always been to provide components and the library for doing industrial strength NLP, so for production workflows, focusing on speed, focusing on ease of use, and also combining cutting-edge machine learning techniques with more classic rule-based approaches and building pipelines for different use cases, and especially focusing on information extraction, which is a thing in pretty much all industries. Everyone has text. We have more text than anyone can possibly read we want to find out more about this text that goes beyond just searching for keywords and that's really what spaCy specializes in and as soon as you start working with NLP and you know you're more serious about what you're doing you pretty much always want data you need data to train your models you need data to evaluate your models and that's how our first commercial product Prodigy was born which is an annotation tool for creating training data and evaluation data for machine learning models and it's really designed as a developer tool it comes with a command line interface with a Python library and with a UI so you can start as a data scientist as a developer spinning up a task creating some data yourself running your experiments and then scale it up with more annotators for all kinds of different tasks from name entity recognition text classification image image annotation classification audio video pretty much anything you can render in a browser you can annotate and really the focus on a developer tool which was kind of new and unusual at the time was because we really saw that you need to be you need to develop your data data is not static just the same way code is not static so it really needs to be a part of your developer workflow instead of just something you outsource and get your data back. So that's what we're doing and we're currently working on a cloud version of Prodigy that lets you work collaboratively on your data without compromising on the data privacy and scriptability. So you can run your own cluster, don't need to send any of your data into the cloud or to us and yeah there'll be an announcement soon we're pretty much almost ready uh to get it out it's been in the works for a long time so that's all really exciting and yeah like um probably many many of you we've um also been very excitedly following the new developments in large language models because especially if Especially if you're doing NLP, one problem has always been how to get more knowledge about the world and the language into your model. Like back in the day, you really had to train your model entirely from scratch, you needed tons of data, and really teach it everything from the thing you're trying to learn and trying to extract to how does English work and what are concepts that are mentioned. And that was very tedious, and then came word vectors, that was kind of nice, then came better embeddings, transformers, all of this really made a difference. And now we're really at a point where we can see that these larger and larger models trained on more and more data, trying to predict the next word or words really work extremely well and not only encode specific information, but really encode a lot about the language and the world, which is what makes them so great. So we thought, hey, there are so many interesting and exciting ways you can incorporate them into an NLP workflow, and that's what I'm going to be talking about today. So first, what are practical NLP workflows? One thing that's always been very popular and still is one of the main ways people work with machine learning is supervised learning and that's not necessarily a disadvantage or a limit, it's actually kind of a feature because if you're doing supervised learning, you can create examples and tell the computer exactly what you want it to do based on examples. That's super powerful because often if you have a problem you want to solve, you kind of know what you want the computer to do and you just need a good way to tell it. So supervised learning is really great in that respect because you can say here's what I want out, try to generalise and learn representations that give me that. So that's very powerful but of course it means you need good data and enough data. Not necessarily big, big data and millions of examples or billions or whatever but you you need very good representative data, you need to anticipate what will go in, and you need to create that, and you need to constantly keep that up to date. So that's one of the parts that's tricky. But that's also something that large language models can potentially help with. And now you might be looking at this and thinking, well, we hear so much about these models, Won't all of this supervised learning or training your own models and creating these things, won't that all be obsolete? Can't we just train bigger and bigger models and run them and then just have chat GPT do everything from start to finish? And the answer is maybe for a lot of use cases, but there's still, especially in practical real-world applications, there are still cases where you really want to use the language model as a tool and not as the entire system. And, yeah, that's especially relevant if you have cases where business logic comes in, which is, you know, your own specific resources, for example, your own logic that only applies to you. So, one of these scenarios are scenarios where specific is just better. The most valuable use cases or the most value is normally in the things that are really, really specific to a use case and not general purpose and don't generalize well across other things. And if you have a large language model, the reason it's so great is that it was trained on so much data that it can give you the most likely best answer and best text and best analysis out at the end, which is great. But if what you're interested in is kind of the opposite, then it can still only be a part of your entire pipeline or entire system, and you always want to inject something that makes it really specific to you. The other scenario is faster is better. While computers are getting faster, it will be less and less of a problem to run larger and larger models, even maybe on a laptop or in the cloud there are still a lot of scenarios where you really need an answer and need the full answer in split seconds and that's very difficult if your model is large and even as computers are getting faster and models are getting larger there will always be use cases where you want something smaller instead of a model that can do maybe a lot of things that you're not even interested in so you want to have your model run on a phone or a tiny embedded device or maybe in the future on a ring I recently bought like one of these smart rings who knows maybe that's the future maybe you can run an LLM on these but for that you want to you know do some custom work to make it faster and smaller and only focus on what you want next data privacy it's something I mentioned earlier it's something that's very important for us for our tools there's so many industries where even nowadays people work on entirely air gap machines nothing can get no internet nothing it can come in or go out and people are discussing GDP are how does that interact with models trained on everything on the internet and I think that's also a very important conversation and already we see that in our customers, our user base. People do not want to send their text to someone else's API and I think in a lot of use cases they also shouldn't. I mean patient records, healthcare, that stuff should not go out into the world and that stuff should also not feed back into language models because potentially there's a lot that these models can leak. So there is definitely work on differential privacy and training on data without actually using the data. And that's all very interesting research. But from a pragmatic standpoint, I think for a lot of companies, it's still much better, much more effective and efficient to just do these things in-house, which means training their own models on their data that run in a specific environment and not sending anything especially not at runtime to a third-party provider so that's also where an LLM can come in for example during the training phase or during the creation phase but not at runtime and finally they also a lot of cases where better is just better and what we if we're looking at chat GPT that does a a lot to raise the floor and that's like super cool but there are many cases where actually what's really important is the ceiling and a model that's like surprisingly crazy good given that it's not trained on something specific is often not good enough than some compared to something that's trained for a specific purpose and to look there's some there's some examples where yeah maybe good enough it's good enough and then there are cases like fraud detection for example, where there's never a ceiling, you can always be better, and you can always go one step further. And again, in these cases, you can incorporate these models, but there's always a lot of custom work you want to be doing to get more out of them. So if you're looking at building something with a large language model in a loop and and maybe use it to do stuff automatically for you instead of creating training data. There are some problems, especially when it comes to information extraction, or if you wanna use the data to maybe train a smaller downstream model that you have to be aware of and work around. And one of them is prompt engineering, which is, yeah, pretty hot topic at the moment, But basically, you have this whole other variable now, which is the prompt that you feed into the model. There's a lot of research around how to best phrase your prompts and what to best do. I think it also it's actually kind of non-trivial. I think it helps a lot to know how these models work, to actually also know a bit about language, to really come up with the best possible way to prompt your model. But it means that's something you have to try out. You have to test. there's no one best answer it's a lot of trial and error and if your prompt changes that also means the output changes so in the output it's easy to get inconsistent results that's maybe also something you've seen if you've played with the models it's not such a big deal if you are the human consuming and reading it but if you want a machine to consume the output and read it that's actually requires a lot more work so you might have the really good response right there but it's unstructured again so now you're kind of back at square one where you have to for example use NLP in order to a create that prompt and then be turn that prompt into something that you can quickly you can compute with and the response might still be better than anything else that you know, you could easily train yourself, but you need to turn it into something that a computer can work with. So these are the main problems. And that's also something we've seen in our experiments and yeah, try to find good solutions for, because if we solve these, then yeah, we can really have a much faster and much more efficient workflow for creating custom models. And so, yeah, one One important aspect is definitely if we are working with large language models as a component, as a tool in order to better create data, for example, the process needs to be iterative. We need to be able to very quickly try things out. Because we have the prompt, change the prompt slightly, the result changes. Need to change the way we're passing the result, try a different prompt, and so on. Try out. How about we let the large language model return JSON? Can that work? Sometimes works well, sometimes doesn't. Maybe we can ask it to give us a list, maybe we can ask it to format things. All of these things really depend on the use case and on the prompt, so we basically needed a workflow that made us able to do this really quickly and efficiently, and also kind of store the intermediate state of like okay here's the code for the prompt and the parsing so we kind of have a record of that and second evaluation is very important if not I'm not sure I would say more important it's always important but it's something that's often ignored but it's one thing to try out things yourself and play around with what the model gives you but you need a really really stable evaluation, you need data, you still need to create examples where you know the answer so you can check it against what you get from that model. And actually at the moment what makes it sort of even more important is if you're consuming it from an API, especially from a third party provider, OpenAI, Cohere, these models can just change and they do change and often maybe in good ways if there's a bug found, if there's an issue, a new version is deployed, and so you can never really rely on getting back the same result. Similarly, you might not even be getting back the exact same answer for the same prompt every time. So you really want to have a stable representative evaluation that you can run every day constantly to make sure that you're really getting what you're expecting and so that's also something we needed to do, we needed to have a fast efficient way to create that evaluation data and we need to be able to run that. And finally we've also seen that in a lot of real world use cases the goal, it doesn't make much sense to set the goal as replacing any task specific models and just having this one LLL magically do everything. There's so many avenues and so many factors where you just want to inject something custom that the most promising path at the moment is you want to use them to improve task specific models and get to something that does exactly what you want much quicker than before rather than trying to get something bigger that happens to be able to do quite decent at what you want on top of a lot of other things. We think in our experiments we've seen that's really where the models are strongest and also that's something you see at the moment in the literature, there have been several papers comparing the result of say a GPT model to a more task specific model on benchmark data sets and we do see that of course if you train something for a specific purpose the results are still significantly better and that's not surprising that's that makes a lot of sense so that's also what I'm going to be focusing on here because I think that's also the most exciting part and it's really the part where you we want to think further than just text generation as the system I think once you really look at it and think past the most obvious. I think there are a lot of really exciting and interesting use cases to solve real world problems. And yeah, so what we ended up building is scriptable workflows. We need a human in the loop. Even if what you're doing is something that can be done end to end by just the language model and you're creating something, you still need evaluation data. You need at some point to come in and say here are examples I know the answer to and that won't go away and you need a way to inject your business logic which could be looking things up in a database or checking things where there is a correct answer and where you don't just want to rely on the model potentially hallucinating a correct fact which is also something we've seen happen. So here's a quick diagram of our experiments and our workflows. In the corner you can see the GitHub repo, these are all, while Prodigy is a commercial tool, the recipes and workflows are all open source. You can look at the code, you can also look at some videos and see how it works in practice and I'll also walk you through some visual examples. But the idea is, we start with a prompt and with a task. For example, here's a text from a recipe, as in, here's how to cook something, we can scrape that from Reddit, and we want to extract ingredients and equipment. And we can have an example of that, of the input text, we can give, as part of the prompt, We can say here's how we want you to do it, list all the ingredients, and list all the equipment, and then we query a large language model. For example, here GPT-35, that's what we use, but now there's also GPT-4 available, there's chat GPT, so there are a lot of different options, and then what we get back is the zero shot or few shot response, hopefully in a format that we can pass and that matches the example in the prompt. And then in Prodigy, we can load that in, view it visually, and curate data that we want for our task-specific model. So we can correct mistakes, correct inconsistencies, and we can skip examples that we don't think are a good fit. And if it's all correct, we can just save it and very quickly end up at a really good data set and also a data set using, yeah, maybe things we hadn't thought of before. So that's the workflow. And to give you an idea roughly of how that looks under the hood, this is kind of more of a pseudo code example because, of course, if you're querying an API, there's like a bunch of boilerplate that's like, you know, that I don't want to put on my slide. But the general idea is this. So Prodigy is fully scriptable. So you write your workflow as part, as a Python function. So the function loads your data, puts together your stream, and then returns the components. So it returns the name of the data set to save the annotations to a view ID, which is the interface to use plain text, named entities, image, audio. All of these have like their own JSON format of how you feed in the data. You give it a stream of examples that you want to view and you want to annotate, and you can give it an optional update callback that's called when you receive answers back. So what we can do here in our case is when you start up the recipe on the command line, we query the large language model and pass the response. So we load in some of our Reddit examples or any text really. We ask the model, add the annotations, pass them out from the text, and send it forward. And if we get examples back, we can also use them to tune the prompt if needed. So if we see problems, if there is something that we kind of ideally want the model to adjust to, we can send it back, include it in the prompt, and then in the future we'll hopefully get better examples, or worse. That's why it's iterative because you're changing the prompt, it might tank the whole thing, so there's a lot of trial and error to use. So here's one example, I hope you can read this. This is how it looks in the UI, so we can see here, because it's an iterative process, you really want to be able to see what the prompt was, what the response was, in case something goes wrong until you've really found something that you can scale up. So we can see here it's actually like really it was really rewarding and exciting to try this out because the model is pretty good especially here we've chosen a topic that we know the model has probably the language model has probably seen a lot of these are all food concepts that's pretty common so it's doing surprisingly well and the approach here is we give it a text and we are asking it to extract comma-separated lists of strings. That works quite well. Sometimes we have tried out different approaches. There are scenarios where it might work to ask it to give you JSON that you can pass, although it might take some iterations to see it is valid JSON, feels a bit hacky. Here one problem you might have that you can't rely on it to always give you every instance so there's still some logic as in you know kind of basic actually basic NLP you then have to apply back in order to really select these entities but it worked so this was very this was very satisfying but of course there are also cases where it gets it wrong it's quite subtle here's the mistake the pans are missing in the in the output so this is just an example of well model just got it wrong it's it's not perfectly consistent just like humans are not perfectly consistent and but what we can do in the UI is we can manually correct it remove the incorrect highlight, add a corrected version that's very quick, just highlight it, and still even if the model gets 10% wrong, that's still 90% less work for you as the human, and at the same time, it also gives you more insights into what's there, how the prompts work, how the responses work, and build up a better mental model, because the truth is we're still at a point where we don't actually know so much about what really happens under the hood and what the best way is to get the best answer. So this is a good way to also try things out. And what we've also built in, which was pretty cool, is you can flag an example that's like a built-in feature in Prodigy and we're using that to add the correct answer to the prompt, tune the prompt. If we are in a scenario where we see the model gets a lot wrong or this specific example is quite important and it can't get it right, then we can add it and hopefully get better responses later on. The same works for a text classification as well, So that's another thing we tried out. We give it a text and a label and we ask the model to basically say whether we should accept this for the category or reject it. And the very nice thing here is we can ask it to generate and display a reason. So that can be part of the prompt. So the prompt asks, is this text a recipe? And if yes, why? If not, why? And for a lot of these things, it can actually do that surprisingly well. So because also a lot of these examples are quite clear, that's really where the model is very strong at that sort of question answering task. And we can take advantage of that. And even if it's wrong, this gives you at least some insight into how that prediction came about, whether it's correct or not. And the same also works for multiple answers, so that's also something, that's just a different interface in Prodigy. We give it a multi-select, pre-select what we get from the large language model. And yeah, here it chooses out of the three, it selects the category feedback because the text is providing an opinion about the use of cream cheese in mashed potatoes. So it does get it and yeah, we're still trying out different workflows. So these are just some of the examples that were very promising, but we can definitely see this working well in a lot of other scenarios too, and I'll also give a quick overview of some other things we're working on at the moment along similar lines. And I think one thing we generally see when working with NLP in a real world scenario is that a lot of the things we're trying to find out, they don't necessarily map neatly to an end-to-end prediction problem. So a lot of the work is in taking a business problem that you have, like populate my database and relate these two things to each other and then do some maths to something, to a pipeline and two components that you can train, write, code, whatever. And yeah, and while a lot of examples of having a model directly predict the output are very cool, it doesn't always map neatly and most effectively to a real-world problem. So here's an example. I've actually used that in several of my talks because it's really actually quite close to what a lot of people are trying to do, even though in different scenarios. So here we want to build a system that takes news announcements of company sales and extracts a nice JSON representation of all the info in it. And this is actually something that nowadays you could try and get a large language model to predict Like verbatim the JSON and I think it could do that in some cases But and that's very and there's a lot of research around this and that's very exciting, but it's also Not very modular and it's quite risky if you're actually Using this in your system because you really have to rely on it to output you exactly this and if something changes you have a problem so the way this is often done in you know a practical setting or the way I would maybe recommend someone solve this problem is first you want to start you start out by deciding whether that system whether that text is about a company sale in the first place because you'll get a lot of noise no matter how good your scraping is you'll always end up with noise so that's the first one. Is it even about that? If not, you discard that. That's probably quite easy to do these days if you have some data, text classification, company sale, yes or no. That probably gets you quite far. Next, you want to get the company names out. So you want to find out which companies are mentioned in the text and the actual text they're mentioned in. And then you actually want relate these companies to an entry in a knowledge base. You want to relate Microsoft to the entity Microsoft, whether it's on Wikidata or some other resource you might have internally, that's typically referred to as entity linking. For company names like this it's quite easy, that's all public info, but often if you're working on something internally in a company you might have your own knowledge base of products and want to link mentions to those products and that always needs custom work, but it's something you can train if you have some data and some examples. And finally, stock tickers, there's still your large language model could probably do that. It might also hallucinate you a stock ticker. So that's a fact. That's something doesn't even need machine learning, you can look that up online and that will be correct and that will be fast and that's just a component you want to add on top. You don't want this to be part of your whole process. And finally price, you can detect the price, you can convert it to and you want to convert it to an amount that you can do maths with. That's also something machine learning can do, but actually I think that Python libraries that can do it for you much more effectively, you can test, there's no need to potentially introduce mistakes this way, parsing a number is kind of solved and I'm not sure any complex model can beat a Python library that you can just download on PyPy. So as you can see here, there are two parts of it that where I think even right now a large language model can absolutely help with and be integrated as a tool which is the text classification step i'm pretty sure you get really good accuracy out of the box and if you could curate this you can very quickly get to a data set that's good comprehensive representative to train that component and even some old school text classification methods If you look at the literature, even a bag of words model in some cases can really outperform your fancy large transformer model. There's a lot of low-hanging fruit there that you can take advantage of. Entity recognition is the same. Extracting company names, that's something you can bootstrap very quickly with a language model, and then you have a base system you can use that you can add your business logic on top of, and you have a robust pipeline that you can test. And yeah, I think I'm definitely very excited to keep experimenting with this because I think this will make a lot of, in a lot of use cases, the first steps especially were the bottleneck. And I think we can, yeah, we can improve this and make it a lot easier to solve these kind of custom problems. And so yeah, in summary, here's the GitHub URL again if you want to check out these workflows. Large language models, they're a great tool for creating better data faster and iteratively. Iteration, incredibly important. You always need task-specific data. There's no way around that. It's a feature, not a bug. It's good. You can tell a system what to do, and you need to evaluate your system. You'll always need to have the answers for at least some of your data. That's just how it is, and evaluation is good and important. and yes, you can end up at a 99% accuracy if your evaluation is shit. That's not what you want. You want a system that's useful, so you want a good evaluation. And yeah, there are many applications in the future that we don't even know about yet, and yeah, I definitely encourage everyone to, yeah, once you think past just, hey, here's an input window and it generates text, there's a whole other world opening up of using a large language model as a tool. And some of the stuff we're working on are data structures for result parsing. For example, in spaCy, we have a lot of that already. So if you take this unstructured response, pass it into something useful that you can compute with, that's very cool. Workflows for robust evaluation, evaluating prompts, comparing prompts, A-B testing prompts, that's all stuff that's very relevant, interactively testing them. And also open source models, there's a lot of cool work happening. And I think it'd be become more and more feasible to run these models yourself locally. And that's something we're looking at. And of course, yeah, shout out to our team at Explosion. This is the work I'm presenting is not just my work. That was definitely a team effort. We've had a lot of people working and experimenting on this. And yeah, if you're interested in more real-world applications of our stack NLP. I also recommend checking out my colleague Victoria's talk tomorrow at two-ish. She'll be talking about building a custom front page for the internet to deal with information overload, which is probably something we can all relate to. So yeah, I recommend checking that out.

Speaker 2 [36:19]

Really, I now cannot stop myself from thinking of having a large language model on the smart ring. Is it really like our future? I don't know, but it's so interesting. And maybe replacing a couple of custom written Python functions by language model is like our next, nearest future.

Speaker 1 [36:40]

Actually, one thing I haven't even mentioned here because it didn't fit in is, of course, writing code will be much easier. There are a lot of models that can help with that. And I was very excited. One thing we tried out is, I think, ChatGPT, we asked it to write spaCy code. And we've really put a lot of work into keeping the library backwards compatible, not breaking shit all the time, having the same syntax. And it can do things like write patterns for spaCy's rule-based matcher, which is kind of like regex taken to the next level, so you can describe tokens, words, and attributes. And it can do that quite well. So there could be use cases where you ask it to generate you a rule that does reliably what you want. So there's a lot of...

Speaker 2 [37:23]

Cool, and then just commit it to your GitHub.

Speaker 1 [37:26]

everything works tested like they're it's i was very excited to see that like

Speaker 2 [37:31]

That sounds just awesome. So let's work for us developers. Sounds very, very nice. Do you guys have any questions? We still have time. So let me pass the mic for you.

Speaker 3 [37:43]

Thank you for the presentation. I was wondering, when you say that when you have a bad example, you add it to the prompt. It means that on top of the prompt, you put that exact example solved. Like for instance, this is an example and then you add your request.

Speaker 1 [37:57]

So here's the text, here's the answer.

Speaker 3 [38:00]

And then I want something like this, right?

Speaker 1 [38:02]

like this, right?

Speaker 3 [38:03]

Exactly, yeah.

Speaker 1 [38:03]

Yeah, exactly.

Speaker 3 [38:04]

Okay, okay. Yeah, and then if there's several, let's say, bad examples, probably it will, you can reach the case where the prompt limit is.

Speaker 1 [38:16]

That's an issue where you can, that's something where the iterative, you have to try things out. I don't, I'd have to also check the code, how exactly we do it, but that's one. It is quite interesting to see the larger context and prompts being available in GPT-4, I think. Yeah. So that's also something you want to play around with. Like imagine you could give it annotation guidelines because that is something you also have to write anyways. you need annotations that are perfect and consistent and for your evaluation at least. And so you might write a page explaining exactly how you want stuff annotated and how you want ambiguities resolved. Stuff like, should doctor be part of the name? There's a lot of... Actually, if you're interested in this, we have a blog post in collaboration with The Guardian. They did some really cool NLP work and they also publish their annotation guidelines because it's also journalists thinking about language. There's a lot of edge cases, even something as simple as quotes. Huge kind of, everything is a kind of worm, so you write these and maybe we can feed those guidelines to the model as the prompt and get good results back. That could be quite exciting, but we'll have to see. that's it's like all vain you

Speaker 2 [39:42]

Any more questions? I believe you were first, right? And while I'm going, I just remind that we are still reading Slido.com, and if you are online, we can ask your questions, so type it down.

Speaker 4 [39:56]

Thank you for the presentation So to my knowledge the terms of use of many LLMs actually prevent you from using their output to build new models Whoever reads those well So the question is does it somehow not apply here, or do you have a special deal at space?

Speaker 1 [40:13]

I mean, this is a legal question that has no precedent. Also, I'm not sure if what they actually mean is the output as in the generated text. I think that's mostly what they probably try to prevent because that really starts this avalanche of text just being used. And here we're just asking it to reply with structure about text we already have. So I'm not sure. This is definitely interesting. I mean, there's also the general copyright issue in general of, like, the model, and can you even use this in the first place? I don't know. I do think if you have open-source models, I think the terms will be different. I think, yeah, with the commercial models, sure, there's, like, with many SaaS tools, there's an incentive of the provider to cut out... Oh, yeah, to make people more dependent on the service and cut that out. I don't know I haven't looked at the terms in detail if that's if it's not allowed then yes you shouldn't be you know I think shouldn't be doing it in a production I mean research I think is fine you can you know you can run anything locally on your machine to try it out but I think I don't think this is something that will hold up

Speaker 2 [41:32]

Cool. We still have time for, I believe, two questions. Who was first? I believe someone from here. And then I come back to you.

Speaker 1 [41:42]

cool i think it was oh well hi i think he was first but it's it's

Speaker 2 [41:49]

We are polite, so if you were first.

Speaker 1 [41:50]

Sorry, he was first.

Speaker 2 [41:50]

Sorry.

Speaker 4 [41:52]

Thank you. I wanted to ask just two questions. The first one, do you do fine tuning for your specific cases to make large models more specific? And the second, do you use some frameworks like Langchain or Haystack?

Speaker 1 [42:14]

but that's also that is an option as well even though it's still you know you still end up with this large model but i think distillation fine tuning there's a lot of potential to cut out all the stuff you don't need and make it more specific and i think in the some of the newer experiments we're running that i haven't shown here we were using long chain and mini chain which I think is a more compact version for the prompt management and also there's all this stuff, this boilerplate around making API calls and that's always great to have someone else solve. And so I think especially for the stuff we're doing with spaCy, spaCy has this config system that makes it very easy to plug in your own functions and that also means that, yeah, you can pretty much sub in any library try it out and use it as part of the workflow and the output will still be

Speaker 2 [43:12]

And our last question. I was going to ask about painting. Oh, cool. So then are you still up to the code?

Speaker 5 [43:22]

Yeah, thanks for the talk. You mentioned the problems in general with these models with inconsistency of the output and the randomness, I think, and also the not clear format. Did you find any good recipes to generalize this in a way to say, like to decorate your request always with a certain surrounding, either with system instruction or like say, independent of the question, always answer, please, that you can always, independent of your request, wrap around, decorate your request with a certain prompt that guarantees you or mitigates this issue?

Speaker 1 [43:53]

this issue? Yeah I think so we've definitely tried out with different so we had these have these prompt templates basically where okay you can play around with different ways of phrasing it then it subs in the text that comes in and then it has like a corresponding function that takes the response and then passes it like if it's a bullet point list so there is some of that but there's still there is there are still cases where we just kind of have to discard what comes in like like there's somewhere okay it just it just happens sometimes it mentions all instances sometimes it doesn't um sometimes what comes out does not map to the original text which can still happen and then we just like may the good thing in this curation workflow you could just skip like if you have enough coming in you can skip but i do think over time there will probably i hope that there will be more kind of best practices around this also as the models change at least it seems reasonable.

Speaker 2 [44:51]

Thank you so much. I believe now we are out of time for the questions. So thank you so much for being so responsive