Haystack 2.0: the story of a rewrite

Speaker 1 [00:06]

I guess you can hear me well. Okay, that sounds good. So I'll be talking about ASTEC, mostly about the rewrite that we had for 2.0. But first, a small introduction. Who am I? I'm Silvano Cerza. I'm a senior software engineer at Deepset. And as you can guess, I work on ASTEC. I used to work also on other companies like Arduino and Pitch before, and I come from this small town in the center of Italy, and now I just moved to the southern of Italy in this beautiful place. So let's talk about the talk. This is a brief agenda. We'll be talking about, I'll give you an introduction to ASTEC, so you know what we're talking about. then we go through the reasons for a rewrite and what to do after you decide whether to rewrite or not i mean in this case we're just touching the uh the yes and then we go a bit more into detail on the actual work that we've done in a stack and then we have just some qa so what is a stack so a stack is just a plain open source python framework that you can use to build your own NLP pipelines. We've been around since a while. You could build, when it started, it was just a framework to build the question answering and document retrieval pipelines, the usual NLP stuff. I mean, before LLMs became huge, then we added that to the framework too. And it obviously evolved over time, and that's one of the reasons why we also took the challenge of our right. How it came to be true? So let's go to a bit of history, of timeline. So it started out with another framework that's actually called FARM, and that stands for Framework for Adapting Representation Models. It was born in... The first comet was actually in July 2019, and it was a framework to train language models. Not the big ones like we have right now, but the smaller ones. but actually Deepset was the first company to train a birth model in German, so we got that. And in November of the same year, ASTAC was born, first commit, and it was depending on farm, so it was tightly coupled with it, and it influenced its API quite a bit. then after a while some months, some years in September 2021 farm was removed as a dependency so we had more freedom there was more possibility to do different things the farm Nomea still stands in the package name of ASTAC it's actually called farm-ASTAC and everything was fine ASTAC was doing its thing document retrieval, QA, users were happy, and at a certain point, as I guess everyone knows, a small little thing called JGPT came out, and yeah, all that broke loose, and people wanted to build new stuff, pipelines were getting more advanced, there were new use cases, so we added the functionality for LLMs to query JGPT and other models, and at a certain point we realized that probably there was some work to do we had some hard times adding this functionality to a stack so the topic of our rewrite came about and then we started talking and there are some pro and cons to consider when deciding on a rewrite there is the uh some of the of the cons is that it takes a part of the team and you split the team on different projects basically you need to decide works on what, and there is some context switching too, it gets hard another thing is that we're engineers, we like to write code but it might not make sense on a business perspective that's just it, in the end the thing is to consider business too I mean, if you're not just an open source project in this case there's also some some pros to it you can completely change the paradigm of your software it's out of right so you can do anything and you cannot even you can change it so much that it gives you the chance to add new features that previously you couldn't do it and you can also tackle technical depth everyone in a easier way you can always do that but out of right makes it easier i guess but in the end it always comes down to two things time and money so you need to decide on that so after a while we made a decision and i guess i guess from the name of the talk you can guess what we decided and yeah we decided to rewrite and we started to do it so i try to here make a nice list of some things to consider when actually starting a rewrite and the first thing is to focus on the pain points if you're rewriting your software probably there's something wrong with it. That could be anything really. It could be that it's hard to create new and new features because there's so much technical depth that makes it hard to do it. Maybe your users want a feature that you can do and that's a problem. And also maybe it's hard to contribute to your project. If it's an open-source project you want ideally to attract as much contributors as possible. If the The code is hard to understand. That's a problem. So you can focus on that too. Another thing is to listen to your community. That could be even clients, whatever. In this case, it's an open source project, so probably you have an open source community to listen to. They might have something that you didn't take into consideration. Interview them. Understand what they want, how they would like to see the project evolve, or even anything. anything, I mean, just listen to your community, really. Or you also risk releasing an RF write that doesn't gather to your user, and at that point, you just wasted your time, really. Another thing is, considering your use cases, that's still connected to the ones before, but what you couldn't do before, what you could do, in which direction can the project go that couldn't before? Could you maybe scrap all the old features that don't make any sense, but you still are around because you couldn't implement new ones for technical reasons, technical depth. That's the thing that you can do when making a rewrite. Another thing is, this is important, set a deadline. Otherwise, you just strive for perfection and you risk releasing never. Or you just fade into nothingness and people just start working on new projects and waiting for your rewrite and somebody took you over and then you're dead in the water when you release your new rewrite. Sorry. Also, don't be scared really to set a deadline. That's important because if you're not meeting a deadline at a certain point, you can also decide, let's scrap this feature or some part of the rewrite. so you can you you can focus on most of the things that are most important that's also a good exercise to do and um and that's it really for the deadline and also another thing is decide your path this could be a life lesson really uh but if you have a project that you're writing And probably people are, hopefully, I guess, people are using your existing project. So you need to decide what to do with that. Do you scrap it completely? Do you want to maintain it for the years to come? Do you want to just say, look, we're not doing anything with it for the future. We'll just maintain it for a couple of months and call it a day. You also need to create some kind of ways for the users to migrate to the new software. Otherwise, it just creates uncertainty. and your user won't like it, I'll tell you. And what else? Oh, yeah, I mean, a rewrite doesn't involve just code. There's a ton of stuff that you need to take into consideration. Probably you have documentation that you need to care about. Releasing just the code doesn't make any sense. The user would be lost, and they would have to scour through the code, hoping that it's well written to understand how it works and what to do. If you have a website, you need to sync with other teams to update that website. Marketing, there's a ton of stuff. Examples, tutorials. There is a DevRel team maybe that need to go around conferences and talk about the new version. This is something that you need to keep in mind, really. It's really important. Never find reaching the release day and realizing that there's nothing other than code to release. so what we've done actually for 2.0 i just talk about the main stuff don't have much time so there's a ton of more stuff that i'm going to talk about and the main thing in my opinion is the pipeline the idea of a pipeline in 1.0 was just like this it was basically a tube you get a query and you get out some some documents or an answer but it was pretty straightforward it basically a tube. You had some decision making in some nodes, components, but it wasn't really that advanced. Every component got the output from the one before and that's how you got the output in the end from your pipeline. And also the components were pretty bulky, you couldn't do much. It didn't give you users the chance to do more advanced stuff. So for 2.0 we completely changed that. Now the pipeline is a graph, a cyclic directed graph. So you can have decisions, you can have multiple inputs, you can have multiple outputs, you can go back in your previous nodes in your graph if you need to correct some errors or whatever. This could be useful for example for agents or agentic-like behaviors. Another thing that you might have noticed is I made the nodes a bit smaller and that's not just for spacing reasons. It's actually because components are smaller. We completely changed the the way that we define the components. Previously they were pretty bulky and they used to receive, because of the way the pipeline was built to start with, they received basically a ton of inputs and even unrelated data that they didn't need. For example, we have a WSPR component that just gets audio input and returns some transcription of that audio, but it still gets labels, list of documents, paths, and in this case it makes sense, but a ton of data that might not not be necessary and they're also and this also push the the devs to create uh bulky components you're receiving uh you you try to do as much as possible with the data you have that makes also harder to test components and also for a user to create their own custom ones for 2.0 we made them extremely small they're they they must do one thing and do it pretty well that's also why they receive the minimum input they need you can you can for example in the whisper component i just talked about it just receives whatever it needs that's a list of files or pipe stream or whatever and this is just some example of the differences between this is the same component from a stack one and a stack two left and right and as you can see in In ASTEC 1, we receive the query, the documents, some meta, I mean, meta might make sense, but not always, and then you just return the output. Also, the output wasn't really the best way to define that. And this is for 2.0 instead. As you can see, we just get the sources. We just need a list of something to convert to. It can be a string that points to a path, a path, a byte stream, whatever. You just get the test's input and you return the documents. And also you can see that we now can define much easier the type of the output of a component. Previously, it wasn't that easy. And this was an extremely great improvement because the components now are much easier to test. The user can create their own custom component like that really in a couple of minutes and you have a component. And this is, in my opinion, the best improvement. Another thing that we improved from ASDAQ 1 was this mess. This is not really messy because I'm not good at making drawings. It's messy because it was messy. The core of ASDAQ had quite some dependencies. The fixed ones, and there were a ton of extra dependencies that you could combine together. were extra, they were depending on other extras, on other extras, and that caused conflicts, that caused the installation time to be extremely long if you depended on multiple extras, because the determination of the version to use could also fail sometimes, and that was a big problem, and also it made it hard to test, we couldn't test this really, the matrix would be huge, it would take days just to test all the combinations. So we went a completely different way and we split them. Basically now we have two different repositories, we have ASTAC Core and ASTAC Integrations. ASTAC Core still needs certain dependencies, but we scrapped the extras. We don't want them at all, because we were a bit scared of jumping into the same issue that we before. We still kept the lazy... a way to install some extra stuff with lazy imports. Basically when you try to import to use or instantiate a component that actually needs an extra dependency, we check if you have that installed and we fail with a nice message telling... I mean, it's still a failure, but at least you get an instruction telling you, look, you need to install this and this, and then it will work this is also beneficial because the a stack integrations now we have and basically it's a completely separate repository each depend it's extra it's basically it's integration it's on its own project you can it's a it has its own dependency basically all dependency all integration depend on a stack core that's why also we have lazy imports for because for, let's say, marketing reasons, we keep some... It's mostly for newcomers to the project. We keep like OpenAI and Hugging Phase components directly into the core because people want to get started with those usually. So the Hugging Phase, for example, depends on Hugging Phase and Transformers, etc. But the quadrant integration that integrates with the quadrant document store doesn't make any sense to have that installed, so we hide that under the lazy imports. And this made the testing time much faster. It was extremely easier to test the core and now also the integrations. We know right away when a certain integration breaks. We saw, I think, the number of integrations went up four times. I mean, we have a ton of integration now, just to name a few, we have Quadrant, Elasticsearch, Weaviate, PyCon, I mean, a ton of stuff. And we're still not testing the old matrix source that takes a ton of time, but at least the user can install just what they want and what they need. That makes it really easy for the user, and also for us, for the contributors and the engineers the work on the project. This made it easier also for our DevRel team. We have a DevRel team that handles the partnership with other companies, etc. And creating an integration now it's much easier to create partnership, etc., because we can just say to a certain company, look, you're supposed to do it like this. If you want to manage your own dependency, your own integration, create your own repos following this template or just add it to our repository and we'll manage that and that was a great improvement and also just to show that adf.write doesn't really improve just the life of your user but can improve the life of your everything basically marketing dev rel documentation after this made it a bit harder to handle documentation because we have to split repositories now but it's still good enough we're in the process of of moving to a nice way of handling dependencies. And that's it really. That's just, I will tell you to just try it out. That's just ppinstallace.ki and you're ready to go. You can just go to our website that I'll be linking right after this slide. and you'll find a ton of tutorials, examples, ton of stuff, the documentation is great, easy to understand, we're really proud of that, so just go and start using it, because that's really fun. I mean, I might be biased, but it's a good framework. And also, if you want to learn more about ASTAC, we'll have another talk about it, it's in Room A1 on Wednesday at 11.40, and Bilge will give it, She's part of our DevRel team, as you can see. And we also meet up coming up on Thursday at 6 p.m. at the Mindspace Krausenstrasse. You need to register for that. So if you want to come, let us know. And that's it, really. It's a bunch of links. The meetup is still in the center. The top one, green one, points to our website, Discord community, where we can find most of the team. I mean, really, all the team. and this is just my mastodon and linkedin I mean you can find also me around and ask things and on github or whatever I mean Silvano Cerza it's me you can find me I have a pretty unique name luckily I have a twitter account but it's just there to prevent squatting don't follow me there and that's it really still have plenty of time so time for questions I was faster than expected

Speaker 2 [19:45]

Yes, thank you, Silvano. And, yeah, we have some questions. You're right. Do you have any insights, any information how many projects using HiStack already migrated to version 2?

Speaker 1 [20:02]

We have some. I mean, we have some metrics in the project, but I mean, if people disable them, we don't really know much about it. I mean, we actually released 2.0 in March, so still recent. There's more and more people using it, we can tell. Even in Discord, we got a ton of questions about it. We see more examples of people building them. And also, actually, The Intel Labs folks actually created a fast-track project that's based on ASTEC 1, and I think they're migrating to ASTEC 2, so that's that too.

Speaker 2 [20:43]

as I continue for previous question what's your plan for how long you think this transition will be so for how long you plan to support version number one

Speaker 1 [20:54]

number one? We plan to release, to support it at least for some months, probably at least by the end of the year it's still going to be supported, but also we have a migration guide for users that's really easy to follow, we have a mapping between the old components and new ones, the previous way to do stuff, etc.

Speaker 2 [21:20]

So guys, we are safe, we have time until the end of the game.

Speaker 1 [21:23]

Yeah, I mean this time the two is much better. I mean I would use that

Speaker 2 [21:29]

Okay, could you tell us about some decisions you made to distinguish high-stack from newer frameworks such as LandChain or Lama Index?

Speaker 1 [21:39]

Okay. I mean, the first that comes to mind is the graph idea. I noticed that Langchain now has Langgraph, but that started out after Haystack 2.0. So I'm not sure who came up with the idea, but it also makes sense. But it differentiates mostly by the fact that we're much more modular than other frameworks, which were more comparable, in my opinion, to Lama index and langchain. I mean, editing a chain is not easy. I guess some people here try to use langchain and it's not that easy as creating a new component in ASTUK, in my opinion. I might be biased again.

Speaker 2 [22:24]

Okay, I see. A few questions about integrations. First one, why did you choose to have all high-stack integration in one monorapper?

Speaker 1 [22:35]

Because most of the integrations really are just Python projects and they follow usually the same structure so it's easier to test them in a unified manner we can have a list right there we can track the issues in an easier way but mostly also to keep it separate from the core because the The ASTAC core has also more workflows in the CI, etc. Oh, no. Sorry. No, it just went dark. But yeah, that's the main reason.

Speaker 2 [23:13]

So, yeah, I also think separating the core from the plugins was a really high and good step forward. But the question was about why all the plugins don't have a separate repository for it.

Speaker 1 [23:29]

Oh, for that, I mean, it's just try to manage

Speaker 2 [23:29]

Oh, okay.

Speaker 1 [23:34]

a ton of the different repositories. It's a mess. Even already having two separate ones, it's quite hard because, I mean, sometimes users open the issue for an integration in the main repository instead of the other one and vice versa. I mean, it's not really easy to manage. Also, I actually had experience doing that and it was a mess, really. When I was at Arduino, we had different cores for different boards and we had a different uh a different repo for each core and for example tracking an issue for that affected all of them was hard and also managing them ci etc

Speaker 2 [24:14]

Okay, and one more question from the user perspective, I would say. High stack has a lot of dependency, even for first version. So here's the question. Is the situation changed now? Yes. And if yes, can you give us some insights?

Speaker 1 [24:38]

Yeah, just trying to install... Just try this. pp-install-farm-astack, that's the first version, and then on a separate virtual env, try to install ASTACK-AI. And you see the speed difference quite easily because it takes a minute or two, I think, to install ASTACK-1 and less than a minute to install ASTACK-2. Also, the lazy import helps and the fact that we're not installing PyTorch transformers. We're still installing some stuff that's quite heavy, like pandas, because we need it, we always need it, but it also made it easier for the user to understand what to install, for example, if they just need to use quadrant, they just use quadrant, if they need to preview it, they install that, previously it was more messy, in my opinion.

Speaker 2 [25:30]

So now it's more lightweight it's more

Speaker 1 [25:32]

it's more lightweight yeah yeah yeah it's faster and lightweight that's that's indeed true i mean that's that's the point why we decided to to split the dependencies to not just to make our life easier but to make it easier for everyone

Speaker 2 [25:45]

Okay, good. And maybe the last question is how you support the users on this migration process? Do you build some tools or it's mostly documentation?

Speaker 1 [25:57]

Right now it's mostly documentation. We have planned for a tool to migrate, but there's also really hard because sometimes it's not a one-on-one mapping. I mean, migrating the ASTAC-1 pipelines were pretty easy, were just a line. There was really almost no decisions in the pipeline. So that's an option that we can make. and we actually want to build it just a matter of time and not, as of now, not our most top priority, really. We have tons of stuff that needs to be done. For example, we're working on evaluation right now because the deadline thing was a real thing that we have to set. And at a certain point, we realized we couldn't ship proper evaluation with 2.0. So we decided, okay, let's move that to 2.1. so we're going to releasing that probably next weeks but that that's one of the reasons

Speaker 2 [26:59]

Okay. Thank you, Luciano, for your presentation.