Type Errors for Better Agent-Assisted Development

Speaker 1 [01:35]

Good afternoon, everyone. So we are ready for our second session of this afternoon here. So now we're going to have the session Type Errors for Improved Agent-Assisted Development with Kyle Intu.

Speaker 2 [01:57]

Thanks everyone for coming. Today I would like to tell you about type errors and how they can improve your agent flows with Python. First of all, who am I? My name is Kyle. For the past four years-ish, I've been working at Meta trying to improve Python language services for them. Generally, we make the type checkers, we make the language services that people use as they develop day to day. So our job is to make sure they're fast, reliable, and accurate. A little bit of background. We used to work on this type checker called Pyre. It was too slow for our developers, so recently, and I'll be talking about this in this talk, we started working on this type checker PyreFly. It's on my shirt. We have a booth down at the bottom, and hopefully you'll learn more about that soon. The other thing I've been doing during my job is coding with coding agents a lot. And if you're something like me, you might use Claude, and your interaction might look something like this. Claude, and hopefully that's big enough for you in the back, Claude, please refactor this function to add an argument, I might ask. Claude will go in. It'll search around. It'll find all the usages. It'll hopefully update those usages. And then it says it's done. All done. The refactor has been applied, and all call sites have been updated, it says. But 47 call sites, that's kind of a lot. And the question is, will I go in and check all those call sites? Hopefully. But I don't know if I will. And let's say I do. That's 47 spots I've checked. But what about the spots I didn't check? There's maybe other call sites that I might have missed. Not sure. But let's say I trust it. Let's just merge this code, see what happens. Looks like it missed a spot. That's OK. I have a strategy for adding arguments to functions. We just tried it. Let's do it again. Let's tell Claude, oh, you forgot about this one spot. And Claude might go in and fix that. But eventually, you might start losing trust in your agents, because they're mercenaries. We pay them in tokens. They deliver us features. And like all of us, they might need some guidance. You wouldn't trust a mercenary with right access to your whole computer, would you? Some people trust Cloud with that. So let's talk about how these agents work. And this is PyData as well, so maybe a lot of you know more than me about this. But this is how I think about it. These agents do their thinking, which is using tools to read the files on your computer, gathering contests, coming up with a plan. Then they go in and edit your code. This is with the edit tool. They'll go and make the changes they want. Eventually, they'll validate the program. And if it thinks it has done a correct job, which it did in our first example, it has updated all the call sites, it then might be finished. If it doesn't think it's finished, it'll do this loop over and over again until it does think it's finished. And in our example, validation succeeded in Cloud when maybe it shouldn't have. And especially as more agents are doing more things at your companies, validation and verification is the best spot we can focus on ensuring these agents don't do things we don't want them to do. So let's talk about the types of validation and verification. As software developers, we're probably accustomed to a lot of this. Starting on the left, we have linting. That's like best practices. We might set up rules for our code base. We want different functions to be organized in certain ways. we want to avoid these pitfalls, that runs pretty fast. And it's really useful to run that on our code, especially with agents, too. As we move to the right, these verification signals become more and more expensive. They end up taking longer, generally. So type checking, while it's still quick on the left side of the screen, it can catch more bugs than linting. I'll go into a few examples soon, and we'll get into that. But I just want to highlight right here, I think there's a good balance between very, very fast and able to catch a lot of bugs. So then the other ones we have are automated tests, which should be ran quite often, but they do take a while. So that's the downside. I'll generally run my automated tests when I'm done with a feature. And it takes minutes to run that test suite. And then we have manual tests and code view, which require humans in the loop, which might take longer. So generally, what I do is when I'm saving and editing my code in my IDE, I always have these turned on, where I'll see the squiggles for the lints and the type checks, I'll know immediately when I need to fix something there. On a completion of a task, I'll run my automated tests, and then before I push my code to production, I might do manual tests or code review. The thing I want to focus on today is this type checking step. And I think it sits at a really good balance where it's both fast enough and gives us enough to work with that I think it's really, really great for agents to integrate into their loop. So before we talk about it, I want to tell you about Pyrefly and what I work on. Pyrefly is just one of many type checkers. Other examples are MyPy, Pyrite, And what a type checker does is it will validate your program to ensure the types agree. So if you try adding a number to a string, Pyrefly will tell you you can't do that. Python has these type hints that you can do. You may have seen them, like x colon int equals 5. But it doesn't actually get checked unless you run one of these type checkers. So a lot of you are probably familiar with mypy. Pyrefly is a new one that we've been working on that's really, really fast. And it's also focused on type inference, so we try figuring out a lot about your program. It is also a language server, and it was written with that from the beginning to be really, really fast in your editor. So a language server is what's used in VS Code, let's say, when you command-click on things, right-click, find references. All of those features are because it's a language server. It's also open source, written in Rust. You can see the source code on GitHub. We have a pretty big open source presence. We have a booth downstairs. come to these conferences, and if there's ever an issue with it, we're happy to talk to you, make a GitHub issue, happy to accept contributors too, people can open PRs, we'll look at all those. So what kind of bugs can a type checker like Pyrefly catch? We talked about the example earlier where Claude was trying to refactor a function to add an argument, that is one type of bug of function arity, where if a function is called with the incorrect number of arguments, a type checker like Pyrefly will catch that. Here's another example that is a little more complicated, though they get much more complicated than this, that kind of shows how this type checker can help specifically for these agents, potentially. So I have three code snippets here. The first is an API to get a user, where it fetches something from a database for a user ID. The second is the usage of that user that we got, I guess, sorry, this is the usage of the API above, and then down in the bottom we access the name attribute of that user. So let's say this fetch API actually returns an optional type. It is unclear maybe if when we access a user ID from the database if it will return something or not. Then our get user API might actually be wrong where we claim it returns a user but it doesn't Because we might not have a type checker set up, we might not know that. So down in the usage of our user, we actually find out that it isn't necessarily a user. It could also be none. What would happen is if we access the name attribute, we would get a runtime error. It would say that, hey, this isn't actually a user. This is a type none. It doesn't have the name attribute. Very, very bad. If we turn on a type checker, we would see the bug up top. And the type checker like Pyrefly or any of the other ones might tell you that actually user isn't necessarily an attribute on none. So in this example, we can see that you may have many, many usages of this user. Catching those bugs in production is pretty difficult. If you have a really good test suite, you might catch them. But seeing the error in the one place you can make a change to fix it can be useful in kind of shifting errors to the left, we call it. like seeing the error earlier, acting on it, and we think this is really, really good for Python programs. So the first thing we have to do is set up Pyrefly. And I'll give you a quick intro on that. I don't want to focus on it too much. But you need to install Pyrefly. Once you've installed it, you run pyrefly init, the command, which will enable it in your project directory. If you use mypy or another one, we we'll try transferring as many configs as we can from that to match our config. It uses both pyrefly.toml, and we also support pyproject.toml as well. And then the next thing we suggest is running pyrefly suppress, which if you're setting up your code base for the first time with type checking, you may have type errors throughout it, but it might be a difficult haul to go in and fix them all at once. So what you can do is run pyrefly suppress, which will insert the error suppression comments in your code, so you make a pull request that's just turn on Pyrefly, run Pyrefly suppress, merge that, now if Pyrefly is turned on, any new code that introduces type errors will be in error in Pyrefly, but pre-existing ones won't. And we think this has been useful for onboarding projects. Once you've done this, you can go in and one by one fix all the errors. But you can do it in your own time. You don't have to block everything from merging, because you're really touching all the code in the whole project. You just do this once quickly, and then fix them in your own time. And that's really all you need to do to set up Pyrefly. Initialize turns it on, and then suppressing them makes it clean so that your agents or you can learn stuff from the new code that you write. The other optional things you can do are put our extension in VS Code so you can see the squiggles in real time, or put it on CI so that we actually have enforcement of it to prevent errors from getting merged accidentally. So we know about Pyrefly now. We've set up Pyrefly. Let's see how it can help our agents. Well, one way we can use Pyrefly with our agents, grab our code, paste it in ChatGPT, grab Pyrefly's type error, paste it in ChatGPT, see what happens. That might have been something we would have done last year, but it's the future now. We have computers writing code for us. So what if we make those computers also do more for us and type check? Today I want to talk about three ways you can add type checking or any other signals, honestly, to your agentic loops, one of which is skills, one of which is hooks, and then the third is through the language server protocol. So I'll go over each of these next. The first of which is a skill, and you may have heard of these before if you've set up skills with different agents. I use Claude, so these skills are specific to Claude, but they should work on any agent you use, if it's Gemini or Codex or anything. So the way a skill works is it is this markdown file. You call it the name of the skill, and within the markdown file, you put the name. So we're making a skill to run the Pyrefly command line, And that's what we'll call it. Also, I have these slides on the website for PyCon DE. You can find them all there. I link to a GitHub in these slides at the bottom with all these examples. So if you're curious for looking up them later, you don't have to remember them. But we're making this skill for the Pyrefly command line. We've called it something. Now let's describe what it does. So pretty simple, runs the type checker. The other thing you want to put in the description is when a skill is used. That's what they say to do. So it runs the type checker. We should use it when the user asks to type check Python code or when you need to verify type correctness after making changes. That's what we put in the description. And then the actual content of this skill is what the agent should do to run it. Luckily, Pyrefly has a very, very simple API for humans, which translates to a simple one for these agents. So our content is really just run pyrofly check at the root of the project. Once you've done all the setup, really all you do is that. It'll tell you if there's issues with what you've typed. So once you've done that, you want to copy it into your .agent slash skills directory, which is where you put all your skills. You'll organize it like this. In isolation, this skill, even though it says use when the user asks or when you need to verify. I've found that generally, Claude, at least, does not run this skill very frequently. So they claim that the description is when to use it. Even if you put in the description something like, always use this, use this every time, no matter what, I found that it generally doesn't use it all the time. So the other thing you want to do, general tips for any skill maybe, put instructions in your agents.md. So you'll see this folder layout. We have the .cloud folder for Cloud. Also applies for other agents, probably. You have your skill here. You have your agent.md. Always ensure Python code type checks clean before a feature. And I have found this setup, which I linked this GitHub here in the talk you can find online. This setup works really well for ensuring that it actually runs the type checker. And what would happen is you tell an agent to make a change, like refactor this argument. It will do the refactor. It will run the type check because it knows about it from the skill. It knows about it from the agents.md. And then it will actually go in and fix these things. And it's pretty magical. You sit there. You watch it do it. And it can validate what it does. So these are skills. They're really up to the agent's discretion on when to use them. We suggest it, but they're not enforced. Another thing you might want to consider is a hook. A hook is similar to a pre-commit hook in Git, where that is like every time you make a Git commit or however you have it set up, it will run a set of checks. These hooks are ways of running a set of checks on agent events. So over here, I have an event. Also, this code is online. You can find it there. I have some events on the stop hook. Let me use this laser pointer. On the stop hook, we would like a command, which is a type of called event. I want to run the pyrefly check command. You will notice two things at the end of the pyrefly-chub command, which are specific to Claude, and this may change for different agents, this API might change, so this is what it is today, I am not sure about the future. Claude says that you must put all output in standard error, and you must return exit code 2. If you use Claude, this is what you will want to do today. I think in the future we might add a pyrefly hook command so that you don't have to do this, but that's at some point when they kind of consolidate all these APIs and make them the same, or if you guys find it useful. So what will happen in this is Claude will guarantee at the stop event, which is the event when Claude's done doing its work, it's ready to tell me it's done, it'll every single time run Pyrefly check, and then it can fix the errors from that. Another hopefully that's big enough, another type of hook you can add is an agent hook. And whereas the command hook, you specify a command to run that Cloud sees the output of, the agent hook is really like a subjective prompt that you'll give it. And this hook also runs every time the agent is in the stop event, but it runs the prompt that you say here. Unfortunately, you cannot reference a skill from a prompt, which would be nice. As of now, I'm sure these things are always changing, so maybe there will soon be a way. But for me, I just said run the Pyrefly checker as the prompt, and this worked pretty well for me. So there is also one called prompt. As of right now, you need to use type agent to have access to skills like command line usage, which like Pyrefly, so make sure you use the type agent there instead of type prompt. And here's a little table of all of them. Do not use the prompt if you're doing Pyrefly, but you can use command or agent. And hopefully you have permissions set up in some way to only give these agents access to certain command line tools, and you would allow list Pyrefly, however you have that set up. But it'll tell you how to do that when you try this. So, the third way I want to talk about is language servers. Pyrefly is a language server. We've found when humans use Pyrefly, the best and easiest and fastest way to get these type errors in front of you is to show them as you're typing. So, can we use this language server in agents like Claude in order to give them diagnostics? That's our question. And when we attempt to implement that, we might think about a skill. We've found skills useful for the command line. Command line's a lot easier of an interface. But we could probably tell Claude or another agent how to use this language server. So we might try describing it in a skill. We tell it that this is the Pyrefly LSP skill. We configure it to run Pyrefly through LSP, when to use it. We start describing the language server protocol. And this is the protocol that Pyrefly talks to VS Code with. It gets pretty complicated pretty quickly. We need to tell it how to send these requests. We need to tell it which order stuff has to go in. We need to tell it a lot of stuff. It's not worth doing this as a skill, I've found. Like these agents, they're not going to understand it. It's just not that fun. So luckily, there's two better ways in Claude. And I'm not sure what the ways are in other agents. But Claude pretty newly added this lsp.lsp.json that you can use, and that lets you specify a command to run to start a language server from a process, and it will know how to communicate back and forth with it. So that's one way to do it. Another way is through the Claude extension, and if you install that, Claude will actually have access to the Pyrefly diagnostics if you have the Pyrefly extension. So these are two ways you can do it. I will say, though, that they're fairly easy to set up, but they are complicated for the agent. So if you're aware of your token costs, you'll find that these things need to use this protocol back and forth. It's a lot more complicated to run a language server to communicate that way than it is to use a command line. It uses more tokens, it's more context, it's more text. The other issue with using the language server is synchronization. So, if you think as I'm typing, I'm making edits in my program, as I make an edit, if I save a file, eventually the diagnostics will come from that file and I'll see them. But there is no standard for how many diagnostics you can send. If a language server sends many diagnostics, it might come all at once, it might come over time. So, for Cloud to really be correct in understanding diagnostics, it will either wait until it It has everything, which might take too long, and when does it have everything? Or it might start with the diagnostics it has, which aren't yet correct. So we've really found that using the language server for type checking has a lot more issues than the other two approaches. And for that reason, we really recommend one of the other two approaches. So if you want to set up Pyrefly or any other type checker through your agent, we recommend a command line skill, which we talked about which is using, adding that skill, adding it to your agents.md, or we recommend a hook, which is adding an agent prompt hook or a command prompt hook, which the hook you would use if you really, really want it to execute no matter what, and then the skill you would use if you wanted to use its discretion. So developer tooling is changing. It used to be that humans were the only ones using Pyrefly. Now a lot of the time, things not made of flesh are executing our tool. In addition to all of the normal work we're doing on Pyrefly, like making it fast, making it reliable, making it really great for humans, we're also starting to think about how can we make it even better for these agents. So I've shown you a few ways to integrate it, but we're only really at the beginning of how we can make it even better. So we're focusing on a few questions, I guess, and trying to figure these out. Like one, how can we make the content of the type error actionable and clear for the agent? Two, when and how is type checking most effective with agents? And three is, can we make this integration even simpler? So I invite you guys to try out our type checker. Try it out with agents. And let us know if you have thoughts, feedback. We're down at the Meta booth, near all the booths. And I think that we're still at the beginning of this research. We're doing a bunch of experiments with different stuff. We will have blog posts out soon. But really, we're starting to understand how we can make it even better for all of this. So I'll leave this slide up at the end, are recommendations for how to use agents. But really, I think we're only getting started with ensuring the validation of these agents is really good. So that's all I have for you today. Thank you very much. And I think we have time for a few questions.

Speaker 1 [25:25]

Thank you very much for your presentation and also for the good time management. It was very good. Sweet. We have some questions for you. Nice. The one that was most requested was, how do you judge Pyrefly versus TI? TY. TY? Yeah, yeah. Why should I use either? Does the OpenAI acquisition change that?

Speaker 2 [25:53]

Yeah, those are all good questions.

Speaker 1 [25:54]

Yeah.

Speaker 2 [25:56]

To the first part of that, how should you judge the difference between them? I think TY is a very great tool that they've made, they've worked hard on, Astral has made other really great tools. We do have some blog posts up on our website comparing them. So there's a blog post about performance, there's a blog post about typing spec conformance, there's a blog post about memory use. All of these are in big tables you can take a look at if you want. I think also you should just try them both, they're both very good, and see what you think. Second part of the question...

Speaker 1 [26:33]

Why should I use either? Does the OpenAI acquisition change that?

Speaker 2 [26:37]

changed it. Yeah, so the why you should use either, I think we can catch a class of issues that you wouldn't be able to catch otherwise, and Meta made Pyrefly and Pyre originally for Instagram. If we didn't have a type checker like this, it would cost a lot of money with preventable problems. Like, if you have many developers working on a big project, it's kind of necessary at some point to add these guardrails in place. The third question, does the OpenAI acquisition of Astral make a difference? I think that might be a better question for them. So for context, Astral makes TY, which is the other type checker that is new and fast. And I think, and OpenAI just bought them a few weeks ago. So they have a blog post out about that on their website. And OpenAI has one on their website. So you should read those if you're curious.

Speaker 1 [27:30]

Okay, thank you. Another question is, for humans, the LSP is especially interesting for the things like go to or show all references and refactoring, rename, move, which currently agents all handle via grep, etc. Without using the Python AST, making this available to the agent better than now via LSP seems really important. Yes.

Speaker 2 [27:58]

Yeah, that's also a great comment. I did not cover in this talk how we can expose other semantics that we understand about your Python program to these agents. And I think that's another area that we need to work on more. A lot of people have had success using the, where is it, LSP, you can use .lsp.json and you can make a skill that says when searching for a certain thing or when doing a certain thing, please use LSP for that. I will warn you, though, that it does sometimes use more memory, like basically you can't control what Claude's going to do, so I think before we make real recommendations, we want to do more experiments with that. Like again, humans use gotodefinition because we don't understand what the definition is of things sometimes, and it's because it's slow for humans to read a bunch of code. But for these agents, reading a whole file that's 10,000 lines doesn't take as long. So some of them may be more or less useful than other ones, and I think we don't yet understand which ones are really, really useful. I will say, though, like, find references, the question brought up is a good example. As humans, when we refactor something, we might find references to know where we want to refactor. That is kind of equivalent to making the change and then fixing the type errors that come from it. So I think the type checker on its own does have really good strategies kind of built in for some of these. But yeah, I think it'll take a lot to understand really which ones are really useful. And definitely come to the meta booth if you have tried any of this and have anything to tell us.

Speaker 1 [29:45]

Thank you, thanks, which of the ways to integrate parafly do you personally use yes?

Speaker 2 [29:51]

Yeah, so I've used all of them to set it up. I always have a skill available, because I will sometimes tell the agent, please type check, and I'll hit enter, and then it knows what to do. And because of that, I've been making modifications to the agents.md to tell it when I want it to type check. So short answer, I use skills always. Medium answer, though, I think people have found a lot of use with hooks, especially you if a code base is used by a lot of people, you can do complicated things with them, but it gets into a lot of engineering. You might have to make a shell script that will execute on a hook, and I also work on projects where people do complicated hooks, like a hook that might run a lot of stuff. So easier, harder, both of them are good. This is probably worth always doing.

Speaker 1 [30:45]

Thank you. Another question, what do you use as a linter? Roof?

Speaker 2 [30:51]

Pyrefly is written in Rust, so we don't use Python linters, but I do think Ruff is a very good linter that Astro makes.

Speaker 1 [31:03]

Are LSP features like rename exposed via the CLI?

Speaker 2 [31:09]

Not right now, but that's a really good question too. So in this talk I kind of mentioned how the command line is easier and uses less tokens than the language server for these agents, and one option for the future is can we expose some things that humans use in an editor in a simpler format that's better for agents. Now there's been a lot of agent protocols for back and forth, but really a simple one One that we could do is a command line for find references. And we have done that in the past for other ones for testing, because it is really easy to write tests for gotodefinition if you have a command line for that. So yeah, really good suggestion, and that's something we're thinking about.

Speaker 1 [31:53]

Thank you. If agents struggle with non-type sensitive languages or perform better with strict type hints, why not move the entire workflow to a more formal language like Rust from the

Speaker 2 [32:07]

Yeah, that's more of a question for you guys, I guess.

Speaker 1 [32:07]

start?

Speaker 2 [32:14]

I think... I mean, yeah, Pyrefly is written in Rust. I do see a lot of value in having the ability to write things without types in addition to with types, and I think depending on, like, the velocity speed of your team or whatever, there's really, really valid reasons for using a lot of different technologies.

Speaker 1 [32:35]

Thank you. Another question. Have you thought about MCP to integrate type checking?

Speaker 2 [32:40]

Do you say ACP? MCP. Oh, MCP. Yeah, sorry. So, yeah, so MCP, the model context protocol, has been somewhat thought about for integrating with type errors. I think there are definitely tools that use Pyrefly through MCP. They are just more complicated, in my opinion, than these two options. And the question is, like, do you get anything more out of adding that complexity? And I have not used them directly to say. But I think some people may have found use out of that.

Speaker 1 [33:22]

thank you another question would be an even deeper integration possible in open models where they else LSP could filter only working tokens and change the log it log it

Speaker 2 [33:38]

What's the last word? How do you spell it? Logic. Oh, logic. Was there an open... I'm not sure what filtering tokens is. Maybe whoever asked that, just come to the Meta Booth and I will answer that better, probably. Can we ask them a clarifying question or no?

Speaker 1 [33:58]

Yeah, maybe the person that asked the question can afterwards come to you and ask you the question in more detail and you can have a conversation on that. Sure. Why do you recommend the agent prompt hook instant of command hook?

Speaker 2 [34:15]

Yeah, I guess this slide is up. I recommend the agent prompt hook just because it's easier. Right now, to do the command hook, you see you need to add the redirect to standard error and exit with code 2. You can mess that up. It's easier just not to do that and to specify, tell the agent exactly what to run. That's the only reason. I think that's very likely to change though soon, either if they change their interface or if we add a Pyrefly hook command. I think that could be a good option.

Speaker 1 [34:52]

Thanks. There's a last question. What did you learn from developing Parafly in Rust, a typed and compiled language, to make type checking in Python better? Yeah.

Speaker 2 [35:05]

Yeah, that's a good question. We actually have a blog post about this on our website, pyrefly.org. It's like lessons learned from writing Pyre, which was our old type checker we wrote in OCaml, which was a typed language. I think that has some good answers to this. From my personal experience, I don't know, I find a lot of confidence writing Rust that I hope to have when I write Python. so I think I'm really focused on making our type checker very reliable and I want people writing Python to have the same confidence that I have writing Rust

Speaker 1 [35:45]

Thank you very much.