Autonomous Browsing using Large Action Models

Speaker 1 [00:08]

Yes, hello, everyone. My name is Nico Kreining. I'm a team lead at Data Science. And Anne is part of my team. We are a software consultancy doing all kinds of data-driven projects, AI, classical machine learning, data engineering. Feel free to talk to us later. Let's dive into the talk. Who came here because of large action models? OK. I want to make one small regard before. So when we contributed to the talk, basically back in September or October last year, we weren't really sure what would be the right term to phrase our project we had planned for back then. And we thought large action models might be the right thing. So what are large action models, actually? The term was introduced by the team of the Rabbit R1, the device you can see there, as a system that understands and responds to human intentions on computers. So, what they actually wanted to achieve is that the system understands the user intent, also by voice, use tools to fulfill the desired tasks, and make complicated decisions on the way to that. And what they imagined back then was basically to create, like, a really large data set with, like, intense user formulated, and then actions the model should do. That was their vision. And as we wanted to do something kind of similar, we back then thought that action model might be the right term for that. But then there came also untrophic and release computer use. So a tool which is basically able to control your complete computer by basically looking on your screen and clicking on different buttons on there. And all of that was basically solved using an LLM. And when you look what is it actually doing, it's not using the voice, okay, but it understands the user intents, it uses tools to fulfill the desired task, and it's making really complicated decisions on the way, like thinking which button to click, what's in the menu, deciding to browse and to use Google and stuff. So what we actually would say today is that LLM agents do basically what large action models were originally promised to do. So that's why we don't really talk about large action models anymore. We talk about LLM agents because they basically do all the things large action models should have done or were promised to do in the beginning. Just as a small remark. And we talk about autonomous browsing. If you look about this, there are basically two different parts. The one is basically, as I mentioned, computer use by controlling the full OS. So looking on the screen and acting similar to you in a way by using the mouse and the keyboard and clicking on things. And traffic was the first open AI operator, it's basically maybe more popular these days. And there's also an open source implementation which is called open interface, which is not that well known and probably a bit laid back. But there's also another way to make LLMs use the browser. And that's basically by directly using the browser. This has a couple of benefits because now you don't have just the screen. You have also the DOM available and can understand and read the HTML structure underneath. Google is aiming the same by building a project called Marina, which is basically a Chrome that does a computer use style controlling of your browser. And there are a couple of open source tools. There's computer use and lavage, and we are going to talk about them soon. And I want to motivate our use case. So we are a software house, and we have built projects for customers, and our customer wanted to have something that an agent is able to fulfill the complete application process on an arbitrary website, completely autonomous, and perfectly matching human quality. So the idea is basically we pass in, for example, my CV, and I say I want to become a data scientist at Sioneers, we hire them by the way, and it should basically take that CV, take all the information which are in there, and then go to the website, fill in all the forms, and do all the stuff. And that was basically our starting point where everything began.

Speaker 2 [04:34]

All right, so let's jump into the technical details here. So the first question is, how do we actually give the agent access to the browser? Or maybe the first question is, why do we need to give the agent access to the browser? Because normally if you communicate with external services, you would use an API, right, which offers a structured format which has clear input, expectation, and gives you a clear output. But as Nico already mentioned, we want to be able to perform these application processes on an arbitrary set of websites. So we don't know beforehand which websites we encounter, we don't have any partnerships with them, and we cannot assume that they expose an RP. And it's also not sufficient to simply scrape the HTML code from the website, because modern websites, they render the JavaScript code on the client side, so the real structure of the website only becomes apparent once you render them within a browser. So, to give the agent access to the browser, we use so-called browser automation frameworks, which serves as a proxy between these low-level browser protocols, which are kind of comparable to machine language, which nobody really wants to program in, and they give them kind of a high-level abstract interface to communicate with the browser. There are three established tools to use here. There's Selenium, Puppeteer, and Playwright, and these tools were actually originally not designed for this agentic use case that an agent is operating within a browser, but rather they were designed for web application testing. So you can test your web application within the browser and do automated tests. And they are now kind of abused for this setup. So Selenium is the oldest one among these, and it has a very wide ecosystem with a lot of plugins. It supports many programming languages. But the architecture is a bit outdated, and the execution speed is slower. Puppeteer is released by Google and it's very well integrated into the Chrome dev protocols and it's very efficient to use there, but it's also only available in JavaScript and for the Chrome-based browsers. And most recently among these three established one is Playwright by Microsoft, which has a very modern and efficient architecture compared to the others, but maybe as a disadvantage has a less major ecosystem. However, among these established, it's probably the way to go if you start a project now. There is one new addition to these three, and that is StageHand, which was released by BrowserBase in 2024, and which was especially designed with these agentic workflows and this integration with AI in mind, the disadvantage here as well as the less major ecosystem. All right, so if you now want to build this autonomous browsing agent, you don't want to start at scratch. So you want to choose kind of a starting framework, and it's always a good point to start at an open-source framework. And as Nico already mentioned, there are two main open source frameworks here, which are lavage and browser use. And I want to take just a bit of time to compare them. But just maybe to say up front, if you start this use case right now, you would probably opt for browser use. Because as you see, compared to lavage, browser use is still actively maintained. There is an active developer community which pushes this framework forward. It also uses Playwright, which is the more modern alternative compared to Selenium. However, as Nico already mentioned, at the start of our project, browser use was not yet available, so we built our project upon Lavage. But I want to take a short second to compare the architecture, because I think it's quite interesting. So Lavage uses a multi-agent setup. So what they basically do is, at each step, there are multiple agent calls, if you want. So there's like one supervision engine, which then decides on which task to do next and which calls specialized agent. Whereas browser use uses a single agent set up where at each step only one agent is called and it does all the tasks at once. So in particular, in browser use the web understanding and the UI understanding are combined. So the agent sees at the same time the screenshot of the website and the HTML code, whereas in lavash this is separated. One agent sees only the screenshot of the website and one agent sees the, there's a specialized agent which only sees the HTML code. So the advantage here is that you don't need to make any assumptions about the HTML code in lavage, but it's more complex and it's a bit more expensive because you need to call multiple language models in one step. However, in browser use, you then need to make more assumptions about the HTML code. You need to perform more cleaning of the HTML code, you need to extract the interactive elements which you want to interact with to not overwhelm the agent because you cannot simply give the agent all the HTML code and the screenshot and all the instructions. one agent is simply going to be overwhelmed in that sense. All right. So now I want to present to you the architecture we designed, and as we already mentioned, it's built on lavage. And this is basically the environment. So the input we have is once the target website, so the URL of the website we want to send the application to, and the user data. On the one hand, as in file format, as most application websites still require you to to upload a physical file, for example, the resume or the CV. And we also need the parsed user information for filling out each form. And then we have our Selenium driver, which allows us to interact with the browser. So for example, click on elements, setting values and input fields, or selecting something from a dropdown. And this is done in a browser which is hosted in a Selenium grid, so basically running in a Docker container in the cloud. So at any point in time, we can basically modify elements in the browser, we can get the current screenshot of the website, and we can get the HTML code of the website. So what is now happening basically at each step is, well, that was the wrong direction, we take the screenshot and pass it to our supervision engine, which is basically nothing more than a wrapper around a multimodal model. So the only requirement here, it needs to be able to process images. And this supervision engine, based on the screenshot, decides on what to do next. So it sees the application page and it basically, for example, decides, OK, I need to fill in the Nama mux in for Nama, just as an example. And it basically, given this instruction, it calls then the suitable engine. So we have multiple action engine. Actually, we only have two action engines. This is a bit of a simplified version compared to Lavache, which has more action engines. But for our use case, these two engines are actually sufficient. So on the one hand, we have the interaction engine, which is used for complex interaction within the browser. And this is basically just a wrapper around the language model, or is it access to a language model. So what the interaction engine basically gets is the HTML code, or at least chunks of the HTML code. So you see there is a retriever in between, so we don't pass the complete HTML code to the interaction engine, but rather we chunk the HTML code up, and based on the instruction which was generated by the supervision engine, we only retrieve the most relevant parts of the HTML code based on keyword or semantic matching. And then the interaction engine basically generates code for the Selenium driver. The navigation engine is just a more lightweighted engine which is used every time we don't actually need to interact with elements. So for example, if we just want to scroll on the website, if we want to go back on the website, or if we want to switch tops, just as a few examples. So this is then basically happening at each step, right? So we execute our action in the browser. We take a new screenshot, the screenshot is analyzed by the supervision engine and it goes on and on like this until the agent reaches the goal or until like a maximum number of steps is reached. And the important part is really here that only the interaction engine has access to the HTML code while the supervision engine only is based on the visuals of the website. So we have the separation of tasks here. All right. Now I want to take some time to talk about our developing journey and one of the key key aspects of this was prompt engineering, so defining the prompts for the agents. And I have defined basically three categories which took up most of our time. The first one being defining the environment. So you need to make really clear to the agent what is the environment it is operating in. So what data is available, for example, the user data you want to apply with, the file path which the agent can upload, or the current date, what capabilities does the agent have and what available actions are there. So for example, for the supervision engine, what engines can it call and what can these engines actually do and what effects do these actions have? So this is really important to prevent the model from simply making stuff up or making assumptions about the environment which are simply not true. The next part is to clearly outline guidelines. So which rules should the agent actually follow? So you don't need to define rules for every case. You want the agent to be able to dynamically adapt to certain use cases, but you want to give it this kind of structure it can go along with. So for example, how should it handle cookie banners? You want to have one strategy to do that, otherwise the agent is going to take a different strategy each time, which makes it hard to debug and kind of have a robust setup. And the third one is defining a notion of state. So the agent actually needs to know what has been happening, what is the current state and what has it already done to prevent the agent from going into loops. And one essential part of this is to give it access to, for example, previous failures and to give it as much information about these previous failures as possible. So this might be a bit abstract now, so I want to make this a bit more concrete by giving you the actual example, or at least parts of the prompt we are using for our supervision engine, just to make these points a bit more clear and to show you the complexity of the prompt. So as you see, first we kind of define what is actually happening. You're a supervising agent, you're supervising an agentic system that performs a web action, and so on. And then we first give it basically the information about the job which it's supposed to apply to. So this is important because most job application pages actually start with a listing of different jobs or the job is obsolete and it's not even there anymore. So the agent basically needs to know what it actually should apply for. And then we basically define all the capabilities our supervision engine has. So what commands are available, which engines can it call. So here is the interaction engine, the navigation engine, also the commands of the navigation engine. And there are two special engines for example the stopped engine or the success engine So I think the success engine is quite self explaining and the stopped engine is basically a way for the agent to self Terminate if it's kind of stuck in a loop and because the agent doesn't really know the concept of a loop in most cases We kind of define some cases where we we know that it cannot go further So for example if a mail verification is required just as one example The next part of the prompt is basically all these guidelines, which also are quite equivalent to all our lessons learned during the development. So while you run these agents on many websites, you kind of see the agent failing on many of these websites. And you kind of start gathering some high-level guidelines, which you then put into the prompt so the agent is not failing on future websites again. So there are some general guidelines. I'm just going to read one aloud. You do not have an existing account on the website. the website so otherwise the agent is because of its training data it is just an LLM at the core so because of the training data we observed that sometimes the agent is just assuming that it has an account on the website and tries to sign in with some arbitrary user data so we give it that as a guideline for example for the navigation engine we tell it if the whole page has been scanned there is no need to scan it again because we need to define the concept of what a scan in our in our setup actually means for the agent and this further tip section is basically like a lessons learned all the stuff we we have been seeing going wrong or where the agent got confused. So, for example, I already mentioned this, always first handle pop-ups and cookie banners by accepting or closing them to give it kind of a consistency across runs so it always follows the same strategies. And the last part of the prompt is then the environment and the state. So what is the agent actually operating in and what is the current state of the process? So here's like the user data, the current data, but also like the previous steps. And there we also give it information if the previous step failed. We give it its previous thought so it can, like, at each step, see what it has previously done and thought in that sense. And finally, we give it, like, our expected output format. So I already mentioned this. The agent is always prompted to output its thought process. Why do we do this? For one, it's very helpful for debugging because you better understand what the agent is actually doing or why it is doing it. But it's also, it has also been shown that this language model, they perform better if you force them to really show them your reasoning process and make it explicit. That's the same reason why we let it evaluate the previous instruction so the model is forced to basically reflect what it has done previously and whether it was going well or not. And then it outputs the next engine and the instruction. Right. So now I want to give you basically an overview because you can run this agentic setup with any model which is out there. And I just want to give you here a vibe check, and be careful, it's a bit of a subjective overview because it's hard to do like an empirical study of how these models perform and they change really quickly, but I just want to give you a feeling of in which categories which model are better than others. And I took basically the strongest models from the three main provider, okay, OpenAI maybe has some newer models, but basically I took GPT-4.0, CloudSonic 3.7, and Gemini 2.5 Pro. And I try to evaluate them among four categories. The first one being GUI understanding. So how well are these models at understanding the graphical user interface, the website which is presented to them? Because they're not explicitly trained to do that, right? Normally they're language models. It's just a side product which apparently they work really good at. And we basically observe there that while GPT-4.0 and Gemini 2.5 Pro perform really well there, Anthropic still has some potential for improvements there because it often misses powers of the UI, or it doesn't understand validation errors. So one common example is sometimes validation errors, so for example if you put in the phone number in the wrong format, are not shown explicitly, but the field is only rendered in red. For us humans, that's intuitive. A red field, okay, I need to fix something there. For a language model, not always. The second category is instruction following. So how well do these models actually stick to all these guidelines we give to them? And here, actually, Anthropic is the only one which consistently follows all the instruction and not goes on different roads. Because like the other two models, they sometimes just ignore one instruction or do something other than we actually prompted it to do, because there are so many instructions they need to follow. The third category is maybe not the most important one. It's effectiveness. So how effective are they in achieving the goal? So the overall goal is to submit a valid application. And most of the time, there's not one path to this, but there are several paths you can take, maybe you can upload a resume, which also fills some of the field. We observed the Gemini 2.5 Pro definitely performs best here because it always tries to take the shortest path and combines action whenever possible. And the third or the fourth category is agentic competence, which is basically how well do these models work within these agentic systems. Like how well do they understand that they are just one part of the process and there's an overall goal to achieve. So the main criteria here is how often do they get stuck in a loop. So that's a negative part. And actually, Cloud Sonnet 3.7 is the best here, because if it gets stuck in a loop, it's the only model which can get itself out of the loop as far as we observed so far. So if you see these pictures, there are two points maybe to take away from this. So first of all, no model really fails at one of these tasks. I always put a zero, so a yellow there. They're not perfect, but it works. And the second takeaway from this diagram basically is that there is not one model at the current state of time which is perfect among all these tasks. So basically you want to kind of have a mixture of all of these models and want to make them all available in your system. All right. Let me finish off with our main lessons learned. So the first lessons learned is that the web landscape is really diverse. So we started the project and we basically took five application pages and we cloned them locally, and we worked with that over a month, and then we went into the real world and everything failed. This is because there are so many websites out there and they all do it differently. JavaScript basically allows you to handle everything differently on a website, so it's really important to get as fast as you can out there and explore the web landscapes to really get a representative set of challenges. The next thing is that it's really hard in this project to quantify improvements, because So if you do changes to one part of the systems, then this may enhance one feature, but may worsen another feature. And that's just because the testing is so hard in the setup. Because first of all, it's quite expensive. Running the agent on one website can cost up to, like, $3. So if you scale that up to 1,000 websites, you can do the math. And yeah, it's just like you cannot simply after each commit you do, like, do a full test run. The third one is that the GUI interpretation capabilities is still a challenge, and it's still causing a lot of issues that the models cannot perfectly identify what is happening on the on the website based on the screenshot and the last basically lessons learned I also talked about this a lot is the critical role of prompt engineering so there's a significant effort not only in the software development part but actually into optimizing and continuously better making these prompts better for the agent so we get the most out of the system so you can get a lot of performance boost just from this all All right. I think I have some time left, and we can also take it a bit into the Q&A session. We can just let it run in the background. But I want to give you just quickly a small demonstration of how this actually looks like. So this is always with live demonstrations, so it might fail, but we'll give it a shot. Okay. So basically what I did here is I've started the agent and it's going to open a browser in just a second and then we can see how it just looks like to get a feeling of what is actually happening and what the agent is outputting here. So you see this is the Selenium grid where our browser is running. So I'm just going to debug in here. So there we basically see, can you see it? I think it's quite sufficient. It's kind of hard because like this is a browser in a browser. It's inception. Yeah, but it's not too important to see what is happening here. You basically see, so first it scanned the website. It wanted to understand what is oh, so it actually took a wrong decision here. That's bad for me. That's bad for the line. Because it actually should have clicked on the apply now button. But it kind of decided to go back to the job listing. So we're going to let it run a few more steps to see. But this is one danger. Because it might now go into the wrong direction and actually apply for the wrong job. So we'll see. We'll see. It's I tested it five times. Do you think it happened once this no? But it's okay So actually this is the job I could also like in the prompt we tell it which job it should apply to so I'm not 100% sure, but I think this is a job. It should actually apply to I'm not quite sure why it so here now It actually says like I can see the la la la Actually Self-recognized that it's not the position it wants to apply to so let's see what it actually does I'm going to restart it otherwise. It's a bit cheating, but I'm pretty sure the next time it's not going to do that mistake. We can have a look, actually. Yeah. What? The what? The question is not in the screen. Ah, okay. So I was asked if it's actually correct that it should apply for the middle-aged teacher. But I'm just going to cheat for a second. I'm going to finish this run. We can have a short look of what it actually should apply to because I didn't tell you that. So it should apply for here. This is basically what it should apply for. So it was actually correct. Yeah. I don't know why. Let me restart it one more time. We can also do the Q&A session in parallel. And I hope this time it's not going to go on some random path. But the good thing is it didn't apply for the wrong position. I don't know why it didn't work, but it didn't apply for the wrong position. All right. We can start with the questions, I guess.

Speaker 3 [25:05]

So, there are many questions on Slido. Why does your client need a bot which applies for jobs online?

Speaker 1 [25:16]

Yeah, basically our client is a platform which basically tries to support humans by finding a job, by giving them all the jobs available, showing them suggestions, and they want to make it as easy as possible for them to fill in all the forms. and often it's like they're also active in the US and often we have forms like for 3, 4, 5, 6, 7, 8 pages where like you have to fill in like 50, 100 input fields and that's really cumbersome work for their clients and they want to make them use their time for a better thing.

Speaker 3 [25:54]

Okay, the next one is, how do you iterate on the prompts? Do we have an automated test suite in place?

Speaker 2 [26:02]

I can answer, but I have the microphone anyways. So we don't have an automated test framework for the prompts in place. So actually what we do is quite easy in that sense. We try to improve the prompts, and then we regularly run benchmarks where we just take a lot of websites, and we kind of see whether basically our goal currently is to increase our success rate, so the number of websites we actually can successfully apply to. We also evaluate all the data the agent is inputting, because sometimes the agent is actually inputting wrong information. it just puts like the wrong email that's not happening too frequently but selecting wrong values from drop down is a frequent case so we also we evaluate whether the agent was successful and we evaluate all the data the agent inserted and whether it was correct or not

Speaker 3 [26:45]

Okay, to the next. Have you considered combining language models regarding wipe check since you mentioned a mixture would be best?

Speaker 2 [26:53]

would be best? Yeah. So what we basically do, it's not currently implemented, but it's on our backlog, is to give these agentic system basically a method to switch models. So if we kind of define that or if we identify that we are stuck in a loop, and there are several ways to identify this, you can use some heuristic that the same instruction is asked over and over again, but the language model, if you force them to think about it, are also able to identify that they're in a loop, we can basically give it like kind of a switch and use another language model to maybe then improve or break out of the loop in that sense. But what you can also do maybe to add this, as I said, we have different components. So you could also, what we also did, is you could use a different model for the supervision engine than for the interaction engine. So for the supervision engine, we usually use a very strong model. But for the interaction engine, which only its task is to, from the HTML code, extract the next action, you can use a cheaper and more lightweighted model. And I just want to mention the agent successfully applied. I don't know if you followed in the background, I hope, but basically what was happening, it took nine steps and now it identified that it has successfully applied. And what is happening now is the judge engine, this is what I said before, so now we evaluate whether the inputted information was actually correct, and finally we generate a success report. Ignore that, that's never happened before.

Speaker 3 [28:15]

Okay, the next one. In a new project, would you use an agent-based system or refer back to traditional methods?

Speaker 2 [28:23]

Again, what was the last part?

Speaker 3 [28:24]

to go back to traditional methods

Speaker 1 [28:30]

so honestly it's already challenging to do this with some agent and we see definitely progress with the models being better with their capabilities and also improving our system I can't imagine to make such a system in in a not agentic system I just don't think that this would be possible

Speaker 2 [28:51]

I agree.

Speaker 3 [28:53]

Okay. Are you only accessing Internet pages? I assume most public sites apply anti-bot protection? Is that an issue for you?

Speaker 2 [29:04]

I can answer that. I actually had a whole separate slide regarding captures because some of you might have thought, hey, there's actually a technique to prevent what we're doing, and that is right, but this is like a cat and mouse game. So there are also a lot of providers which handle captures, and at least from my perspective, we're not trying to insert garbage information. What we are doing is currently done by humans manually, so they take information about applicants and insert it into the website. We're just trying to define a process which does that automatically, but we're still aiming for high-quality information in these application pages and it's still the user data which is inserted so it's not made up information

Speaker 1 [29:39]

Yeah, so it's no internal pages real pages in the internet on any website and yeah But that makes it tricky to to develop Yeah, because you don't want to make real applications all the time during testing so you need to be careful there

Speaker 3 [29:56]

Yeah, thanks for your good talk and great talks. And yeah, give a applause to Nico and Arne.