Beyond Vibe-Coding: A Practitioner's Guide to Spec-Driven Development in AI Engineering

Speaker 1 [00:03]

Welcome to the session. The session will be 30 minutes as well. And five minutes hold to the Q&A. Questions can be asked through the talks.pycon.de. Please ask them. And today's session is Beyond Vibe Coding, a Practitioner's Guide to Spec-Driven Development in AI Engineering. And please, big applause and support for Alina Dalman.

Speaker 2 [00:40]

So you can actually get a few hundred lines of code in just a matter of seconds without making a single precise decision. And in my opinion, that's both like the promise and the problem of building software with AI today. So today I would like to talk about the methodology that kind of tries to fix that. It is called spec-driven development, and I believe it's one angle how to think about coding with AI assistance. So with that, hello, I'm Alina. I'm an AI engineer at CyanEars, a data engineering and data science consulting company. And usually I do spend my days building production-grade virtual mental generation systems these days, mainly on chatbots in areas like customer support or university tutors. So let me directly ground my talk into something concrete. So let's assume we do work on a user facing chatbot and we need to add a new data source. The data source in this case is OS ticket, which is an open source help desk ticket system. And the goal is to synchronise these tickets into our data pipeline so that users can actually search across them using a chat interface and also get answers back with proper citations to the original ticket? Well, that might sound straightforward to some of you, but the moment you actually do start thinking about that closely, questions, of course, arise. Like, how do you actually do this data synchronisation process? How do citation contracts look like? How do you handle all that stuff? And so, actually, why did I now choose this use case? I think it forces, it builds a kind of environment where you need to think about real decisions you would take in a real-world environment when developing actually a feature like that. So here's what may happen when you build a feature like that in a chat session of your favourite AI coding assistant. There may be failures like fragmented design decisions. So you start building the data synchronisation logic as that may be the first point you would need to do to get data into your system and build up on that. So that's fine, you open the chat session for that, let the AI implement all that stuff, and then you say it's done. Go to the next chat session, start building up on that, build some citation data models, And suddenly you realise, wow, there are problems with the things I didn't think of before, and now why did I make this and this and this decision, and it's actually buried in this other chat session from before. So actually decisions are kind of scattered across chat sessions. The second failure mode might be something I would like to call prompt drift. So you start your conversation having a certain mind and goal, like this data synchronisation process, but then you may end up in a conversation just answering the AI questions, and so you are only basically reacting to the AI's output, and the conversation has taken a direction on its own, and you may didn't notice it happening. The third is hidden assumptions. So you had the AI assume, for example, that all tickets have a kind of subject line, but that's not true. And so, like, within your systems, there are reasonable values taken, but you may only find out later after testing it, the implementation, that these assumptions were wrong. So the problem in general is the LLM has no idea at what exactly you are building, it's just good at executing the next step and executing what you are telling us to do. So what can we do about that? The answer is not new, but actually the framing is, so we can write a spec not after the code, like not as a documentation for other humans, but actually some documents before the code, which you then give as a shared context for the LLM and you. And so that's a context an LLM interface may actually never give you. So what do we actually mean about a spec in this context? So a spec is like a kind of can be one structured document or a bundle of documents that actually do live in your repository, and it captures what the system should do and not how, and so it kind of becomes a single source of truth that every session then can reference and you can also build up on it. Additionally, there may be other bits of information which you can actually integrate into that, like decisions, you would otherwise put into other stuff like architectural decision records, you could also document your unknowns, so open questions which you do still have or the AI still have to answer them later in the process, and actually also something like out of scope declarations may be very useful as a line for the AI, so do not solve this and this problem or do not optimise for X by that. So the key insight here is a spec makes implicit decisions explicit before you let the AI generate code. So when we do that, we have to answer some uncomfortable questions of course beforehand. So actually Martin Fowler published a taxonomy about spec-driven development end of last year, and I think it's helpful to think about how specs relate to the development process, and it actually distinguishes three modes. The first one is spec first, so that means you write the complete code before the implementation, but do discard the spec afterwards, so it's like a one-time use. The next thing would be spec anchored, so that means you You actually use the spec across different chat sessions to iterate on it and also keep it in Git for version reference. And so that's basically the thing, the closest I'm describing today or what I have in mind. And the third part would be like spec as source. And that could mean that actually the spec specification is already so detailed and formal that you can directly let the AI generate code from it, almost kind of domain-specific language. But we don't have this kind of domain-specific language yet. Potentially, this code is generated, but not potentially, not even reviewed by a human afterwards. So I hope the methodology got a bit clearer by now. Specify before you generate stuff. Be explicit about your decisions and also document all other stuff which may be important during the actual code generation process. But how do you actually do this in practice? And here I would like to show you one concrete tool that actually does implement this methodology. It's called Speckit and can be used in conjunction with a lot of known coding assistants out there. And I actually chose to show you this kind of tool today, not because it's the only option, but it was just one of the tools I've been using so far. So setting up spec kit is actually very lightweight. So you just install it basically as a set of commands to your AI-assisted environment so that you, again, get structured commands, one for each phase of the development workflow, basically. And so then each phase actually produces some kind of artifacts, documents, which we can later have a look into it. And these artifacts are reviewable, of course, and you can also put them into a version control system. What is a bit, what I placed now not in the process, which you see on the right, which is basically the main spec kit command process is the spec kit constitution command, that's a command to actually set up your project principles, so that's something you would only do before the first time, and of course you can iterate over that, but you won't do that for every feature probably. So that's, like, this constitution contains general project, like, development principles you have in your project. Potentially, you also do have something like that already in stuff like an agent's MD file. So, let's have a further look into the first two commands of this process, the specify and the clarify command. So the specify commands creates and updates, creates a spec file actually, it's a markdown file, and then the clarify command does a kind of structured question answering session and resolves ambiguities together with you and implement this again in this spec file. So what may be in this spec file or what was in the spec file for the given use case I introduced. First of all, we have user stories in there, so that's actually something the LLM generated on its own for our process. We have something like a ticket synchronisation process. As I already mentioned, we need to have data in our system, of course, to do something with that. Then we go a step further, have implement all this logic around interacting with this ticker data and then get some citations, citation-based answers from that. And a third user story here is that we also integrate common threads, so you probably know that from, I don't know, tools like Jira where you have, like, comments and then you can do common threads. We also do have this in this help desk ticketing system and would like to integrate that into our system. So the second thing which is in this spec file are kind of functional requirements, which are, of course, assigned to a certain user story. So there are stuff inside like the synchronisation process by REST API, but also stuff like, okay, I would like to have the citation meter data to include certain kind of fields, and there can be internal nodes in this actual original ticketing systems, And these should be actually excluded from our system. And another part of this file are kind of session clarifications, which mainly occur after this clarification process, where like kind of open questions are answered, or you answer open questions together with the AI, and here came stuff like that we would like to actually include all ticket statuses into our system, so that means tickets could be open or pending or closed or whatever, and we would like to integrate all of that. And but exclude, as I already said before, which was in the function requirements, some internal nodes. So if we were to iterate it now on this process, I think we do have a valid SPAC file, we can move down a bit further down the line to the planning mode. Actually the planning mode produces different files whereby I would like to show you two of them shortly. And it actually in general documents like the key decisions and details of the feature to be implemented. So there's a research.md file, and this one is quite interesting, I would say, regarding the general structure. So we do have a different kind of findings integrated there, like this one here being we have a chunking strategy at common boundaries, means like, okay, how long should the tickets be in our system, the chunks we can get back to preserve some kind of precision and refer to the correct systems. That was the original question, and then there were kind of decisions taken. Okay, we do split all the comments on their own. They may not be that long, and they're semantically data fit together. And then there's also some rational documented for that and in general alternatives considered. So let's quickly jump into another part of a file which was produced using spec kit for the given use case. like from a contract folder, the citations for schemer, there are like five fields should be generated with an explicit value pattern, and there are also additional information given. And this may be actually the kind of contract that may prevent integration bugs, and it's also clear, of course, like what kind of shape the data takes before you're already writing, before you fit in any line of code. So spec kit done, actually if you're also happy with the plan, you reviewed it, you can go further and let spec kit generate some tasks. And actually the tasks are then split down into different phases. I borrowed a part of it which is like the phase one where you do some basic set up. do you need throughout the other kind of phases to implement. So there's a purpose basically documented for it, and then down the line, the single ticket or single task are very concrete, like add this and this enum value to this and this file, and if you give this file actually to another LLM session, it can just already begin to implement that. And one thing I would like to emphasise here that there are like kind of P markers, and the P markers actually indicate that some kind of task can also run in parallel as they don't depend on something else. So for this OS ticket feature, actually, including everything, there are 30 tasks generated in total. And so once you now think all this task file, the tasks sound reasonable, I would like to let them implement, there's also directly a command in spec kit to start the implementation process, which I usually do in like kind of of badges, so you have like one like face per face, because I think like usually these kind of faces are your natural review points and review checkpoints where you can do a review afterwards. But, of course, that always depends on like your workflow, how you like it, and how much stuff you would like to review. So as a short overview, being now through the whole process of Speckit, Speckit produced eight files for the given use case, like this Markdown, like they are all Markdown based, so the files are already named, and then there are additional files like from the planning place a plan file and an actual data model file, and the last point not mentioned, also a kind of requirements checklist which you can also clarify, check again against your system at the very end. So before I would like to talk about the methodology, again, a bit more general, one personal reflection. So writing specs at first kind of felt slow, but when I started then implementing the actual features, I thought I had the feeling it was dramatically faster and needed far fewer corrections than letting the AI generate everything at once or letting just the AI generate stuff I give it one by one. So probably I do have the feeling that the total time is roughly about the same you need to do something. It's just like the overall process where there's shifts from debugging to more conceptualizing and thinking. A second learning is, as always probably with AI, it's maybe useful to form your own opinions before running something, especially here the planning phase because otherwise, like the AI's reasoning, can prime your thinking too heavily and you don't come up with the best solutions for a given case. Yeah, so I actually tend, personally, tend to over-specify things if I have this specification process. Like, for example, in this case, you could over-specify an error handling process for this data synchronization process and then realize at the end, okay, it's already somehow handled on the other side and in the REST API itself. So the last one here for me was also I really need to focus on the behavior my system or this feature should have and not so much on the technical details. So now let me step back from specific spec kit and talk a bit more about the methodology again. again. So I think the fundamental strength is really separating the what from the how. And so, yes, we do have a concrete implementation plan at the end, but the most important parts for the future are the actually required behaviours of the system, so that then at the end, in In the future, you can just refactor, swap out a library, rewrite a module entirely, or anything else you would like to do. So that's what it makes it. That's why, like, specs are kind of durable in a way that, like, I think like chat sessions in your coding assistant never app because you won't find any chat session from a year ago. Yeah, that also is hand-in-hand with, like, the next point, documentation, which you already have as a side effect. So if you do kind of spec-driven development all the time, you almost don't have to write separate documentation again, and the spec files are the documentation. And third, you do have some reproducible context across sessions, and that's really a big advantage to your thing, because when you start a new chat session, you point the LLM to the spec directory you already have, and it already has everything it needs in place. So that avoids this kind of, may avoid this kind of prompt drift. So, what are the negative points? Every tool actually events its own structure. There's no kind of standardisation yet. Of course, you do have some set-up overhead if you first start implementing such a feature. But yeah, if you sorry, I lost the point. But a negative point maybe also here, like just with general documentation, there's always the risk of having stale documentation if you don't update your specs once you change something in the system. Yeah, so in general, the token usage for the LLM may be a bit higher compared to using just chat sessions, just for this planning phase. So I personally do use cloud code for that and let it run on the given use case and use like 170,000 tokens for this whole full spec kit generation planning phase kind of versus 60,000 tokens in the cloud general planning mode. So honestly, you don't need any of these tools or the tools like spec kit to actually practice and implement the methodology. A well-structured Markdown template may be already enough. So I think I would like to give you something there where I think maybe a starting point, like kind of four levels. So for level zero is you don't need any specs. For really small, well-scoped tasks, if you already know what exactly you want to do, do, and can frame it, then it's also maybe fine to just give it to the AI directly. Second phase would be if you want to have, like, some kind of planning already in your process, use built-in planning, the built-in planning modes of your AI coding assistance, do that if you like the format and the contents do not really matter to you. If you really want to have the same structure over and over again over different features, you could develop your own Markdown template, which may be also very nice for solo work. And the third point is the thing I will actually present it, using dedicated spec tools, also stuff like OpenSpec or SpecKit, and there are more, of course, you could use that for complex features, or if you want to like have a post set within a team which is like synchronised and you do use the process all over again and would like to do, for example, some reviews. So I would like to encourage here everyone to try the spec-driven development process at least once. So just take your next feature and try it. But here, again, the most important thing, you don't need to follow any tool or any template rigidly. These, like, either spec kits, open spec, all these other tools out there, these are all just starting points. Just take what resonates with you, how you think, and how your team works, and then just adapt it and iterate on that. So I think the specific format in general, matters far less than the act of actually doing it. So I'm also happy to discuss your approaches to AI-driven development during any break or also tonight in the social event. And I would like to finish with, so the spec is not the documentation alone, it's really the shared context that a chat interface never gives you.

Speaker 1 [26:13]

Thank you, Alina, for an insightful talk. We don't have questions yet on TalksPyCon.de. Does anyone maybe have questions here? A note?

Speaker 2 [26:32]

Just refresh the site, probably.

Speaker 1 [26:35]

Yeah, I do that. Somehow it doesn't work. Sorry. Yeah, we have lots of questions. Sorry. How is your presented approach better than the SPI design-specified plan implement? Do Do you want to read? Yeah.

Speaker 2 [26:59]

Yeah, I don't know this approach, I think, I'm very sorry, but actually this sounds basically like the same approach. Like you design something, then put it in some kind of specifications, put the planning on how you would implement this feature, then implement it.

Speaker 1 [27:17]

Great, thank you. How do you avoid specs becoming excessively long, overly detailed and unreadable for humans?

Speaker 2 [27:31]

Yeah, that's a very good question, I'd say. I don't have the feeling that they become unreadable, but I personally just feel like tools like SpecKid, they kind of have a complicated structure of where they put things, and that's what I feel is, like, as a human, difficult kind of to get used to, like, where is which information written and where do I need to update what and how do all these signals belong together? I think like one thing maybe the same with everything so maybe if you feel the specs Become too long your feature is already too big and you should rather make your feature Smaller that might be one approach to that or it's like just maybe you don't need all the specification stuff AI generated you

Speaker 1 [28:23]

Right, thank you Next question does peck it keep the specification up-to-date if the implementation is later changed How would you deal with it? Keep the old spec to make the deviation visible or update it to reflect code and be helpful as context for further changes

Speaker 2 [28:47]

Yeah, so it always, like, a tool only does what you tell it to do. So, of course, if you change the implementation and also tell it to update the spec or, like, this specification update command skills are found, then it does actually update that. I had a good impression of that, and now I already forgot, like, the second half of the question that was, can you shortly help me? What was the second part? Keep the old spec. Oh, yeah, I think, like, what I really like about stuff like spec kit, there's actually an option to kind of merge, like, the overall spec, so actually just, like, the behavior documentation into a general very long spec where your whole system is implemented, and so that's why, so maybe if you implement another feature, you could then specify it, it's then merged into the general spec. I hope that answers that.

Speaker 1 [29:50]

I think so. Yes, peg-driven coding seems to make sense rationally.