Building MCP at the Speed of Hype: Principles That Outlast the Trends

Speaker 1 [00:00]

And that will tell us about building, sorry, building production systems that do not become immediately obsolete in the data science in the I almost had it. Yeah. Sorry. The floor is yours.

Speaker 2 [00:37]

That was my fear. That's why I have water here. Yeah, we're very happy to be here. We're going to talk about building MCP at the speed of hype. When we prepped our PyCon presentation back in December, we had a focus on MCP technology, and we were actively developing two client projects with it at the time. time. We sent a proposal and what followed were a lot of new changes in technology. And we had new releases and new challenges and our focus shifted slightly. We did not only have to build MCP at the speed of hype, but multiple things around it also shifted. All of this while developing for clients that also faced uncertainties and in between. We We honestly looked back at the presentation a lot and wondered if our core message had also changed, or if it actually could remind us about why we sent in the proposal in the first place. We would like to share some of our learnings along the way, not to tell you everything is perfect, but to give you more confidence and reminders about principles that we believe outlive these trends so far. To start, we would like to present two projects we worked on in the last year, and we will continue with the shifting landscape and what changed to understand the challenges that come along when building MCP applications. We will then present some principles that provided a foundation for us, and we will close our session with our lessons learned. Who are we? My colleague and I, we work with and around data in the public and social sector. We mainly develop dashboard solutions for clients such as DigiAZ and their Negotiate AI tool, or the Job Monitor that highlights labour market trends. We closely work with different clients to build solutions for them by us, and we're going to have a look at two projects that we worked on that were relevant in the MCP space. The first one is a multisource search agent for an international nonprofit. It was developed to combine multiple data sources of the clients in one chat interface to be used for questions internally, but also externally. It has three components. So the first is a custom chat interface built with Vue and Vuetify, and that is compatible with the AGUI protocol, a couple agents in the middle handling different tasks with built-in tools like the web search or also the file search. That is connected to our chat interface, and, yeah, if you don't know, AG UI, so agent user interaction protocol, is an open protocol that standardizes how AI agents stream events and interactions to front-end applications in real time. And last but not least, we had an MCP server that is connected to agents, bringing in context and data from the organization API and database. The second project is the JobMotor MCP. Its main goal was to give our JobMotor team more resources to take into their own talks and explore data that we already have, but just in an easier way. As we developed the JobMotor ourselves, we were quite familiar with all the backend stuff and all the endpoints, just needed another approach of easier data analysis. It connects to the Job Monitor API with data about the German labor market, and it's basically a classic MCP server. The server exposes tools like getRankingData and getTimeSeriesData. We also use the MCP apps protocol to connect a widget with the data visualizations directly in ChatGPT. If you're not familiar, MCP apps is a newer extension of the MCP protocol that defines how interfaces connect to hosts. If that sounds abstract, we have a video to show you. So we are in ChatGPT. We are asking about software developers in Darmstadt, and then we get a tool calling get interactive widgets in a second. Yeah, all in German, but I think you can see you can toggle it, it's interactive, and we can also have a look at the output and the input, so the responses are awesome. With these projects in mind, we will have a look at the shifting landscape next.

Speaker 3 [05:41]

Okay, so that you've seen our two projects now, a lot of what's happening around us, so let me take you through what that looked like. So when we began our project phase in May 2025, GPT-4.1 was the latest model. AI agents, frameworks like Pedantic AI, and protocols like MCP were the big topics, so the things everyone was bowling with and talking about. So, these were the technologies we were familiar with, and we had to plan to use for our both use cases, and then things kept coming up, one wave of new developments, and then another, and another. Now, what I will show you now is not a complete picture of everything that happened in the AI world, but just what we noticed from the providers we were working with in the order we noticed it, and how it felt for us to developing in this shifting landscape. So after we started our projects, AG UI and A2A came along. That's two new protocols, the first one, and already explained, so it's for connecting agents to the front end, and the second, A2A, is a protocol to connect agents to each other. And then in mid-2025, we noticed a major update from Fast MCP and MCP, which gave us new possibilities like better authorization mechanisms and also new ways to structure tools. And then towards the end of the year, OpenAI went into full shipping mode. They dropped a lot of things in a very, very short time. Examples for that is the agent kit, which is a full toolkit for developing agentic systems and ChatGPT apps to add interactivity to your MCP tools. That was also when we submitted our PyCon proposal, by the way, and then another big wave came, and then suddenly there was a lot of momentum around generative UI in January 2026, MCP apps being one example of that, which is a broader take on what ChatGPT apps had started. And throughout all of this, the models also kept moving. We started with TPT 4.1, as I said in the beginning, and then we went all the way up to 5.4, and also TPT 4 was fully retired somewhere in the middle of all of our project development. So that's the landscape we were building in, two different AI projects and a constantly evolving ecosystem around us. This left us with one question. How do you build something reliable when the ground beneath it keeps moving? So let's talk about the strategies that helped us or also our learnings we had during those two projects. And you're probably going to recognize most of them because a lot of these things are just good software engineering practice. And I'm going to start with a really classic one, which is requirements engineering. So the most expensive code is the code that solves the wrong problem. So before you write a single line of code, ask yourself, what are the problems we're actually solving? What are the constraints? What are functional requirements? What are non-functional requirements? What does it look like if your project is finished? And specifically for working with AI, ask yourself, what value are we adding by using it at all? These are questions you're probably going to also ask if you're doing any software project, but here's why this matters even more in the AI context and if we're having this shifting landscape. So let's look at this timeline again. So much happening here, and actually much more than we are showing here, and this can feel very, very overwhelming. But when you have a clear picture of what you're building, most of this just stops being relevant. And I wanted to make here an example with our use case, the Job Monitor MCP. So simply put, our core requirement was to help users to analyze the current German labor market situation in natural language inside ChatGPT. There's obviously more than that, but that's the essence of that project. Now once this is clear, a lot of decisions make themselves. So as I said, we're working inside ChatGPT for this project, so we don't need a custom interface, which means things like a GUI, agentic kit, generative AI, none of these things need to be on our radar. Also ChatGPT is going to handle all of the things like the agentic layer, so the only thing we need to bring in is the MCP server. So A2A is also something which is not relevant for this project. Also with the models, we had a clear definition what the quality should look like for that product so we could evaluate deliberately and update only when it actually mattered for a use case. So a lot of these model updates were also not relevant for us. And suddenly, most of this timeline just drops away, and instead of feeling overwhelmed, you now have a very short list of things that actually need your attention and you can focus on. Good. Good. Now, knowing what you're building is the foundation, but even with clear requirements, you still need to be ready to be adept. And that brings me to the second principle, which is design for flexibility. And I wanted to highlight here three things that specifically kept us flexible, which is separation of concerns, again, a really classic one in software engineering, being prepared to replace and build on standards. Let's start with the first one, separation of concerns. So you already saw the architecture of our search project, my colleagues explained it to you. But here, just as a recap, so we have three components in that project, the front end, the back end, and MCP. And within the back end, everything is also clearly separated. So we have a separate module for our agents, for our API, for the data ingestion pipeline to handle all of the data stuff. And also in the MCP component we have clearly isolated MCP servers for each problem we wanted to solve. Now when AG-UI came along with that project, that separation really saved us. We didn't need to change the agents, we didn't need to change our MCP servers, the only thing we actually touched was the API layer, we replaced it with an AG-UI compatible version, And then we added also the corresponding view components on the front end. So one clean swap, nothing else broke, and that helped us really to adapt to AG UI. Good. Now the second point, be prepared to replace, and this might hurt a bit. So the reality of building in this space is that development moves really, really fast, and sometimes that means letting go of things that you already built. I wanted to give you an example from the search project again. So one of our MCP servers handled website rack. So it was basically an MCP server that was answering questions about an organization content using their website. We built our own rack system for that. We built our own data ingestion pipeline for that MCP server, so the whole thing. It worked. It worked really well, but OpenAI's web search kept getting better at that time, and at some point it was simply the better solution. So we deleted the whole pipeline code, the data ingestion pipeline code, we deleted the whole MCP server, and then we swapped it in with an OpenAI web cert as a built-in tool for our agents. So thousands of lines of code deleted, maybe five new added, that can hurt, but that's part of building well in this space. And by the way, this is where separation of concerns again really paid off, because everything was cleanly separated, we could just remove the unnecessary layers and we didn't need it to start from scratch again. And then the last one for being more flexible, build on standards. Here I won't pretend that you can always know which standard is going to win because there's a lot of shifting things right now, but when the industry is clearly converging on something like MCP, AGUI, it's worth betting on those standards. Why? Standards define what others build against, which means you stay almost compatible automatically. And you can also swap the backends, you can swap the clients. And lastly, they protect you from vendor login. So if you're tied to one provider's proprietary interface and they change the direction, or your customer wants to change the provider, you're stuck. But with open protocols and that problem just goes away So that's the second learning and then I wanted to go to the next principle which is personally my favorite one So all of this MCP AG UI agentic systems It can feel like you need to learn everything from scratch and every week something new drops and you feel like you're falling behind behind. But here's the thing we actually experimented. The technology underneath is often not completely new, so it's more familiar tools in a new context. And I wanted to share two moments with you where that really clicked in for me, and depending on your background, you might recognize the same thing. So let's start with MCP, specifically building an MCP server. So when I first looked at it, it felt new. But the moment I started to build something, for me. So the questions I was asking myself were exactly the same ones I would ask when I'm designing an API. So what does this tool do? What should it return? How granular should it be? How can I design a system that solves my problem? And if you wanted to implement it in Python, there's fastmcp as a package, and if you ever have worked before with fastapi, which is a package for implementing APIs in Python, You will probably quickly see that it's pretty much the same syntax, it's the same patterns, it is the same concepts. There are new things to learn, of course. Working with LLMs changes how you think about what a good tool looks like, and we're also working here with non-deterministic outputs, but you're not starting from zero, and you're building on something you already know. And a second example is MCP apps. recap again. So MCP apps is to build interactive tool outputs and MCP protocol. That might sound new, but let me show you actually what is happening underneath. So there are three parts in MCP. Your MCP server with the tools and resources as usual. And then there's something which I call here UI resource. And that's basically an HTML interface that is linked to one of your tools. And instead of returning plain text, your tool can return a visual component and then in the middle we have the host and that is a chat client for example chat gbt cloud or might can also be your custom client and that takes the ui resource and renders it inside a sandbox iframe and then the host in the view and they communicate through post messages iframes is a technology from 1997 post messages around 2008 so the protocol is new but the building blocks are not. The same for AG UI. AG UI is built on server sent events and JSON events. That is the standard ways for servers to push updates to the front end. So that's why this is my favorite learning, the realization that you're not starting from zero. That took a lot of pressure off me, and it's motivating also to keep up with the new technologies that come in AI. And my colleague, she will show you now how this applies to testing and evaluation.

Speaker 2 [17:57]

Yeah, something that is also not new was our approach to continuous learning and evaluation, basically having a structured test set up. We took a very simple approach and leaned onto our known evaluation processes. To start, we gathered questions with our clients they wanted to be able to answer, so type of gold standard with metadata, like what the tool calls should look like, what tools should be called. We then tested out to help us evaluate the non-deterministic outputs and then calculated measures like precision recall, which everybody knows. We then could take that to our client and get feedback and also did some manual review and could see what worked and what did not work. Using an Excel file with gold standard responses gave us like a qualitative understanding of the data and what cases we needed to cover. Using an LLM judge took away some of the manual testing and standard evaluation measures gave us the basis to talk with the clients with something they could understand and have heard before. And then the cycle continued with adjusting the files again. It also helped with the general communication with the clients and giving them something like concrete examples they could also showcase in their team was a big learning. This brings me to communication with stakeholders. In our project, we had to navigate not only our understanding of AI, but also our clients. So we were asked a lot of questions throughout, like, are we behind? I think our most important learning is to recognize the client's concerns and fears and actually consult them in an empathetic way. They might also ask, why don't we have the news model yet? Well, it's also important to be honest about uncertainties and to have open communication about it. For example, model came out two days ago. It's not that different. We don't really need to switch at this moment was also an answer we could give. Why can't it answer this question? An example here is not to promise unrealistic things and educate on what, yes, AI is capable of doing, but also what it maybe can't do. And knowing, for example, what non-deterministic actually means was a key message to our clients. We need streaming. During communication with clients, we also feel it's important to assess if we actually need novel AI solutions. and see if they are just wanting to implement it to be able to say they use AI and maybe advise them on that. So how good is this? What we also saw, especially in the evaluation part, is to keep explanations to a familiar knowledge and introduce new concepts in a structured and open way. How do I test myself? itself. We have found that we can cover all of these aspects by providing guideline documents that describe use cases about what they can test, what might not work, all in words in a layout that is catered to your clients. With clear guidelines and one point for them to reference, they became more calm, the questions to us became less, and they also learned and became more confident at the same time. And this does not have to be a long and detailed document, just a starting point for them and something they can reference and share in the team. Before we come to the lessons learned, it's still frustrating sometimes. All that seems very easy if you just hear it. We want to take a moment and acknowledge that, yes, we are also frustrated. We have two instances that were especially annoying. Not only we can't keep up, also the documentation might not be updated on time, and we've definitely spent hours searching for information that then was released a few days later. Or we wanted to try out ChatGBT apps, very eager to test it, turns out the feature was not available in Germany yet. So wait and see was actually our best approach in that case. These are only two examples of our frustration in the process. There were many more, but the rest was very fun to develop. So lessons learned. What kept us anchored in a shifting landscape? First one was requirements engineering. So knowing what you're building will help you easily ignore the rest. Designing for flexibility. So getting ready to replace and set up your project to separate and replace easily. And trusting in your foundational knowledge, which can be hard if you're hearing about a lot of seemingly new things. Continuous testing and evaluation, so this will give you more confidence and orientation where you stand and what you might still need. And a non-technical one is honest stakeholder communication and education. To meet your clients the way they are and work together on novel technology. Yeah, challenge, but it was very much worth it to work on. Now if you have any questions, we're happy to take them.

Speaker 1 [23:48]

Yeah, we have a few questions and back to life So How do you ensure the metrics showing the correct values or the correct filtering aggregations etc? Do you have some pattern or arch you can share?

Speaker 2 [24:10]

Do you mean metrics and evaluation part?

Speaker 3 [24:15]

Probably in the evaluation. Possibly, yeah. So I think one important thing in generative AI is that, I mean, the output is non-tumoristic, right? So you cannot really test that. But what helped us was that we had a set of questions, like core questions, that we wanted to make sure that our product can answer. And that doesn't have to be a lot of questions, but it also, like for the job monitor MCP, for example, I used a set of like 10 questions. And that was with iBuildit, the product, and if these 10 questions can be answered, then I would say, okay, this is a good product, this has to be answered, and I tested it also with different models to make it more bulletproof, and that at least helped a lot. And if you're working with things like Rack, I would always recommend to not test the generative output, but to test what the Rack system outputs, because that is often deterministic, and you can test it. I hope that answers the question if not then please come free to later

Speaker 1 [25:19]

Next question. You strive for flexibility, but you're using OpenAI tools heavily. How do you rationalize this?

Speaker 3 [25:28]

I think, do you want to answer it? So we worked a lot with OpenAI products because our client requested that, but as I said, we strive for flexibility by setting on these standards, because if you have these standards, then you can also switch to Cloud, and we use OpenAI for getting the generative outputs, true but that's just client requirements we had but the products can for example the job monitoring speed can be used in other with other providers and you can also like use it with your custom ones

Speaker 1 [26:12]

Okay, we have time for questions How did you collect the gold truth data for the evaluation? Did you consider creating them automatically?

Speaker 2 [26:22]

Yes, we did. It was a long process, I would say, for us, but we made sure that we had very good data. Our team, for example, looked at the website that we wanted to generate test examples a lot, so you actually have to be familiar with the context, and then we also tested out how generating questions would work with AI. We found that didn't really work for us, and we also had the clients that they already had testing or example questions they could give us, so it was a collaborative effort of us having to actually know what we want to answer, them already knowing, and testing out AI help in the process.

Speaker 1 [27:09]

Did you run into limits providing the tools via your MCP servers?

Speaker 3 [27:15]

What kind of limits? Can someone who asked a question if it's here maybe give us a hint or you can come afterwards because I'm not sure.

Speaker 1 [27:27]

We can leave it to...

Speaker 3 [27:28]

Yeah, you can just come afterwards.

Speaker 1 [27:28]

Yeah.

Speaker 3 [27:30]

Sorry, I don't...

Speaker 1 [27:30]

Yeah, we have time for a couple of questions. We can answer these questions later.

Speaker 3 [27:35]

Yeah.

Speaker 1 [27:36]

Yeah. Do you build an MCP server and standard API in an integrated way so that you have always both types or are these totally separate things?

Speaker 2 [27:46]

Can I see it? Yeah, sure.

Speaker 1 [27:49]

Only long questions today.

Speaker 3 [27:52]

Do you want to answer it?

Speaker 2 [27:54]

No, no, I think you're good.

Speaker 3 [27:55]

Usually, we have often the API already done in the first step, because we both usually like products at our company, which can have dashboards and stuff like that, and now since AI came up so much, we are building on top of that, but it's not something we always do together. That's not always the case. So we sometimes also bolt MCP servers that just like directly with the database, for example.

Speaker 1 [28:24]

One last question. Did you consider to build for multiple models like Anthropic Cloud or others?

Speaker 3 [28:33]

Yeah, again, that's just client requirements we have, but we didn't consider, like, building for that model in specific. We would do if, like, our client's requirements come with that, but we didn't consider, like...

Speaker 1 [28:48]

Okay, that was really cool. There are other questions that will be addressed maybe in the Discord. And let's thank our speakers. Thank you. The next talk is in 10 minutes in praise of documentation.

Speaker 2 [29:10]

We'll be right back.