Hype, Hope, or Headache? Making Sense of GenAI, LLMs, and AI Agents with Anecdotal Evidence
The current trend toward Generative AI and Large Language Models (LLMs) creates a risk of neglecting traditional data science. Over-reliance on LLMs for simple problems violates Occam's Razor, as high-parameter models are often inefficient compared to linear regression or scikit-learn models for specific, low-complexity tasks. There is a concern that the field of mathematical and statistical data science is declining as practitioners shift toward "shiny" GenAI tools, potentially sacrificing the robustness of Bayesian networks and other traditional algorithms.
An AI agent is defined specifically as a system where an LLM acts as a central brain, calling tools in a loop to achieve a goal. This differs from traditional automation, such as a dishwasher or a warehouse management system, which relies on "if-this-then-that" logic. The primary value of AI agents lies in automating low-frequency, high-complexity tasks—such as planning a wedding—that were previously too expensive to automate using traditional software engineering because the development cost outweighed the time saved.
To determine the appropriate automation tool, a cost-benefit analysis based on task frequency and time savings is required. High-frequency tasks should be handled by robust, cheap, and testable traditional automation. Low-frequency tasks are better suited for AI agents. The economic disruption of AI agents is compared to the Toyota Model G loom; the loom did not increase weaving speed but introduced a notification system (a bell) that allowed one operator to manage multiple machines. Similarly, the value of AI agents lies in their ability to handle diverse tasks and signal when human intervention is needed, rather than simply increasing raw processing speed.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Autonomous Systems & AI Agents.
Submission
The proposal as submitted by the speaker before the conference.
After nearly 20 years in data science I’ve seen many “revolutions” come and go: neural networks, SVMs, bayesian statistics, random forests, XGBoost and deep learning. Each came with bold promises, and each eventually settled into a realistic place in production systems (read: became boring). Generative AI, however, feels fundamentally different.
In this talk, I’ll share my view why the current GenAI hype stands apart from previous cycles: technically, culturally, and organizationally. Even with some understanding how these things work, I am still blown away by the stream of stunning new capabilities. This is not a “GenAI is bad” rant. Instead, it’s a critical attempt to understand the shift we’re seeing, and the risks that come with it if we don’t adjust our thinking.
Using industrial examples such as supply chains (just because I work in this field), but also personal experience, I’ll show where LLM-based approaches still have serious limitations today, and where GenAI can realistically add value. We’ll disentangle different categories of risk from technical fragility, evaluation problems and mere costs to organizational overconfidence and misuse.
A big part of the talk dives into the rapidly emerging field of AI Agents. We’ll explore what AI agents actually are, where they make sense today, and where the current hype is just snake oil, particularly to senior decision-makers who may underestimate complexity, costs, and failure modes.
The goal of this talk is not to slow innovation, but to enable better decisions. If we want GenAI to be a success in real-world systems, we need to understand both the change it represents and the limits it still has.
An anti-bullshit take on the possibilities ahead, with honesty, anecdotes, and (for those who know me, of course) a bit of humor.
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:00]
You could say, is it really that bad? Is there any problem with it? Maybe some things are stupid. But not. Because there is this strange feeling that somehow there is this new AI and this old, bad AI. And many here are data scientists or machine learning experts and so on. and i guess i guess we we all share this just just the other day we had a discussion and uh somebody asked do you mean this new ai and no no i mean a traditional machine learning algorithm okay i i thought the new ai no no um so yeah also here it's it's our job to somehow tell people no no no if it's a linear problem a linear regression might be the best fit don't try to do it with an LLM. Even maybe you can somehow figure it out. Also, Occam's Razor says the simplest solution is the best solution. So always choose the simplest solution for a job. So you don't need to have five billion parameters for a one parameter problem. So I would say this can be very harmful if somehow there is this opinion that, oh, you're working on this old, old scikit learn model? Shouldn't you use something more modern? And I think that would be very bad. And I guess it will come back and we will use scikit learn more maybe in the coming years. I mean, this is really then anecdotal evidence, right? Because I haven't found something to really to really prove this point. But I mean, for us at the Python conference, this is somehow remarkable that now the number one script, the number one programming language is not Python anymore. It's gone, right? We were we were top ranking in all of the in all of the indices before. And there was one reason it was data science. Everybody who did data science use Python, and then you build your applications also using Python. But the core was, we want to have the machine learning models we do it in python we do jupyter notebooks exploratory data analysis this is all the python world now i mean as i said it's just anecdotal right it's no proof but maybe seeing that other languages like javascript or typescript is number one maybe it shows not so many people are working on actual data science anymore and that's really bad right and then that's that's That's one of my points. I don't see enough mentioned also on LinkedIn and so on. When I started my PhD, I was working on machine learning models, all these things. And I thought, yes, all these things like machine learning, AI. We called it AI back then as well, but more or less this was the future. So, but I thought this will make us superhuman, right? So whatever we do now, it's much better then. And then this happens, right?
Speaker 2 [03:29]
I mean...
Speaker 1 [03:30]
What we actually did is we made computers as dumb as we humans are. Why? I mean, computers could do things like this before, and now they can't. I don't get it. I mean, I'm just waiting for the... I mean, also, when I made this slide two years ago, I was like, yeah, I guess in future I will book my flight, and instead of going to New York, I go to, I don't know, Amsterdam, because just an LLM thought it's maybe a nicer destination or whatever. So I don't know. Then I got this advertisement on my mobile. So here you can search for flights using LLM. I was like, ooh, I don't know if that's good. Let's try it. So, I mean, this is the end of the first part of the talk. because there is yeah i mean summing this all up really there is this uh this problem maybe the world just says oh okay there was the year 2010 until 2020 we did data science we were at a good good trajectory so we built good models that could solve really really sophisticated problems and now if really we decide to just move on and do the new stuff and we don't care about the Bayesian network anymore because now I have my LLM which does all the stuff I don't think that's that's a good a good thing right I mean coming back to my original slide right I am now in the Gen AI team it's not bad it's I yeah thank you it's it's not bad at all right? It's fancy, it's shiny, we do cool things. So it's not about this, but it somehow also proves the point. Maybe it is the case that somehow this traditional deep mathematical statistical data science field is somehow drying out and maybe we should fight for it that it's not happening. And this brings me to, wow, now what happened, right? so maybe maybe you're all sleeping right now so maybe please stand all stand up once all standing and now please sit down if something in your daily work significantly significantly changed over the past six months and now sit down if you think nothing no sit down if you think something will change in the next two years in your work i want to talk to you what you are doing because i cannot think of anything but no yeah so yeah i mean we just saw it so everybody has this feeling that something is happening i mean just i mean i put this here but there are so on this page there are a lot of different signals from from everywhere so why is why are things changed or where are the signals where we see that something is changing this is just a very simple uh plot i mean maybe maybe it's complicated but uh just to give you a very quick uh thing so that's the number of hours a human would have to do the job and um and then you see uh so that that's the equivalent what a human would have to do for an llm to solve it and a 50 50 chance so and you see yes there is something happening right so uh around here it was like okay maybe i can do something a human would take one hour but this exploded somehow and i mean this also can be seen just yesterday i i searched google trends for cloud code and open claw i mean those are the the buzzwords which were popping up around december as well i was kind of surprised that the open claw is declining so steeply and also cloud code so maybe it is just a hype thing and um maybe it is going away but i doubt it but um yeah so but when i started to to have a look at all these things right so my my question was what is an agent why is everybody talking about an agent what is this what is this really because i mean for example i have a dishwasher at home i put my dish in i press start it does something and in the end I have a cleaned dish. So that's like an agent, right? It's automating things that I had to do as a human before. So why are now people talking about agents and what's the difference? So first of all, then, there is just a very strict definition. AI agent is a very strict definition. I think I have the definition from you. So it's more or less you just have a brain which is the llm the llm is calling tools in a loop to achieve a goal so you tell it okay do this then the llm calls tools in a loop until it finally solves the problem or or reaches the goal so that's the very definition of an ai agent and that's what we are talking about if if something else is happening we probably would just not call it an ai agent right so that's that's the because because i mean of course if this then that brings you a long way the dishwasher is just if this then that if uh clean then do something else um it it brings you a long way only if we have this llm this brain in the middle and it calls tools in the loop then we call it an AI agent. And still I was thinking, so when do I have to use this AI agent thing? And maybe if there is one thing you should take home from my talk, then it's this. I hope most of you know this because everybody should have this on his and her desk. So for example take so it shows how often you do a task and how much you can save by automating it and this then is the number you are allowed to invest into building the automation for it so if we take the dishwasher it's maybe something like around five times a day if you have a big family and maybe dishwashing takes 30 minutes so you you in your household you would have six months to build a dishwasher and you would still then in five years be on on a positive return right so you can quickly check okay what's happening yearly five seconds one second okay five seconds i mean there there is no there is no uh deeper thing behind it it's it's school math right but but still it's it's good to yeah thank you it's good to just see it once and now let's have a look at this everybody talking about ai agents yes they are awesome this sector over here before we couldn't do because what we were doing as software engineers we were building these things we build automating automation systems which are used heavily let's say warehouse management system thousands of operations per day and maybe automating away a second still you can work a long time on it right or here it's absolutely clear these systems we already have because this is the dishwasher and we already have dishwashers at home so this is done already but with the new thing we can explore this field over there which is remarkable because i guess there is a lot of money to be made um and it was just not accessible because you can't just afford to put that tiny number of work into into something to automate it but now you have this brain in the middle so you can tell the brain please solve this for me even if it's you're planning your wedding you will only plan your wedding hopefully once once in your lifetime but now still you can pass it over to an ai agent to automate something of it because it has a brain right so that's the for me that's the clue and always when people are talking about it and and they say oh we've built this ai agent i'm like you're in the wrong corner you're automating something which is i don't know used 50 million times a day just build if this then that solution that's probably more robust more more cheap better testable all these things right so yeah show this to your to your ceo it might help to to figure out stuff and there is we just saw most of except three people of this conference everybody uh has uh has a feeling something is changing so there will be a deeper dive we saw it in the opening session this morning so there will be a panel i'm the moderator so i have to find some questions if you if you have some questions just send it to me over over discord then there is an open workshop tomorrow So it's really everybody is welcome to join, and we can find ways to discuss all the things in the final panel on Thursday. And I want to send you home with this machine. No, I don't. Yeah. I mean, maybe some of my colleagues already saw this, but is there anyone who knows what this is? you don't know the english word i didn't know the english word for it either it's a loom but it's a special one it's the toyota model g loom i think i looked it up and it's really the same toyota from the cars today so somehow there is a connection so question is is this the the first automated loom so loom is where where you make your clothes with um so you have all those strings, these threads here, and then you weave some things in between. Was it the first automated loom? No, it wasn't. That's a good one. Yeah. No, so a quick explanation. So there were looms before, and they were not faster. So this one is not faster and not much better in any dimension, but it was the first one which was an economic success and it disrupted the whole industry. Now I explain what's happening. So there were automated looms before, but if one of those threads breaks, everything which is produced afterwards is trash. So what happened, next to every loom, there has to be one human operator checking if one of the threads is breaking to stop the machine, repair it, and then go on. So, one loom, one operator. This one, I don't find it somewhere, there is a small bell. So they have some needles here, and they check if the thread is still ongoing. If one of those breaks, there is a bell which makes bing. All of a sudden, one operator can operate a whole, a whole area of looms because it knows, Oh, there is something broken over there. I go over there and I fix it. So this was a factor of X economic improvement. And now you can go home and think about what was the improvement, right? I think there was a talk about evals. So maybe you should check the talk about agent evals and so on. So with this, beware of snake oil. We see it all day on LinkedIn. Not everything is gold. So thank you very much. Thank you.
Speaker 2 [16:30]
Yeah, thank you, Sebastian, for this talk and your sharing your thoughts with us. We have two questions in the talk's chat. The first one, when open data is the major source, is the service of this data no longer important after the data was added to a proprietary model?
Speaker 1 [16:51]
Is it the right room?
Speaker 2 [16:54]
It is, yeah, that's what someone was wondering.
Speaker 1 [17:01]
Can you ask me? I mean, probably I can find an answer, but it does not reveal.
Speaker 2 [17:04]
Yeah, maybe whoever posed the question might want to rephrase it and post it again in the talk.
Speaker 1 [17:09]
again in the talk is there is somebody here in the audience who posted this question or
Speaker 2 [17:14]
online either way I will refresh and see if you can clarify yeah we have another question as just more general do you think that usage increases exponential
Speaker 1 [17:32]
Good, thank you.
Speaker 2 [17:39]
All right, yeah, I guess, or whoever posed the other question.
Speaker 1 [17:43]
I mean, is there a question here in the audience? We have a microphone, I think.
Speaker 2 [17:47]
We prefer to have it on the talk so that it's recorded.
Speaker 1 [17:51]
but it's recorded. I'm the speaker. Is there a question?
Speaker 2 [17:55]
You might want to, if there's no more questions, I guess you can always just like. Yeah, you can approach.
Speaker 1 [18:02]
You can approach me.
Speaker 2 [18:02]
reach out to Sebastian as well and again thank you for your talk sharing your insights and then in 10 minutes we will resume again in this room with another talk about the next talk wait a second I want to read it out properly that we have in this one is... Where is that? Yeah. Well, it's in the program. It's in the room. Thank you. Just another applause, and we'll see each other all over again.
