Process, Analyze, and Transform Python Code with ASTs
Abstract Syntax Trees (ASTs) represent the structure of Python source code as a tree of language constructs, such as modules, classes, and functions. In the Python compilation process, the parser creates an AST as an intermediary step before compiling source code into bytecode. This structure allows tools like Ruff, Black, MyPy, and Bandit to analyze, lint, and format code by treating the program as a hierarchical data structure rather than raw text.
The Python standard library provides the `ast` module to programmatically interact with these trees. The `ast.parse()` function converts syntactically correct source code into an AST rooted at a module node. Developers can inspect these structures using `ast.dump()` or traverse them using `ast.iter_fields()` and `ast.iter_child_nodes()`. For deeper analysis, the module offers three primary traversal methods: `ast.walk()`, which visits nodes in no specific order; `ast.NodeVisitor`, which implements a depth-first traversal for observing nodes; and `ast.NodeTransformer`, which allows for the modification of the tree.
Practical applications include building linters to detect anti-patterns, such as bare `except` blocks or missing docstrings. By subclassing `NodeVisitor`, a tool can flag generic exceptions by inspecting `Raise` and `ExceptHandler` nodes. To transform code, `NodeTransformer` can replace a `try-except-pass` block with a `contextlib.suppress` context manager. Because modified nodes lack positional metadata, `ast.fix_missing_locations()` must be called before the tree is compiled back into a code object via `compile()` or converted back to source code using `ast.unparse()`.
This description was generated by Open-Source AI using the transcript of the session and the original submission contents.
This session took place in track Python Language & Ecosystem and was classified suitable for novice domain / intermediate python by the speaker.
Submission
The proposal as submitted by the speaker before the conference.
This tutorial will be a roughly 50/50 split of lecture and exercises. Attendees will get hands-on experience working with ASTs in Python, using only the standard library. By recreating common code-quality checks from scratch, attendees will both learn how common tools work under the hood and how to work with the AST in an easy-to-understand fashion.
Topics covered:
- Introduction to the term and concept of Abstract Syntax Trees (ASTs)
- Some of the ways ASTs are used by Python itself and by popular tools
- Parsing code into an AST and inspecting it
- Walking the tree:
ast.iter_fields(),ast.iter_child_nodes(),ast.walk() - Modifying the code before running it
- Converting an AST into source code again with
ast.unparse()and its caveats - Finding missing docstrings
ast.NodeVisitorandast.NodeTransformergeneric_visit()method — what it does and why we need it using animation- 4 exercise breaks spread throughout accounting for ~45 minutes
Transcript (auto)
Auto-generated from the recording utilizing Open-Source AI. Speaker labels (Speaker 1, Speaker 2) reflect diarization, not identity. Timestamps refer to the recording.
Speaker 1 [00:27]
your questions during the speech so feel free i give you the micro because to be more integrated to the session maybe it's better to talk with our speaker today instead of writing in the channel so i give you the microphone and our speaker today is stephanie moly and she is a software engineer at Bloomberg from New York City, works in information security, and she is co-developer of NumPyDocs and creator of NumPyDocs pre-commit hook. Also, she is the author of Data Analysis with pandas and she has some background in machine learning. So I hand over the speech to Stephanie and so the stage is yours.
Speaker 2 [02:22]
editor, your choice. Make sure you, and this is also in the pre-talks information, that repository, this one, I have no idea what's going on with all of that text. This screen is very weird, but you do have the slides here where they will be with these weird lines through them, and so you can follow along with me here. Here we have the larger code examples will be in here, so you'll have those for reference and to continue when we do the exercises. And then I also want you to open this link here. This is to the AST documentation on the Python's there at library. So that looks like this. We're not reading through all this obviously and part of the benefit of it sort of like this is that you don't have to read through all this so you'll understand how to consult it for what you need. So we'll come back to that in a little bit. So everyone have those links? Can I just get like a thumbs up if we're good? Okay, a few people who care to vote are good, so that's all that matters to me. So for the agenda for today, so this is gonna be a half version of a larger workshop that I've recently built. So we're definitely gonna get through the first two sections, which is gonna introduce you to what ASTs are, where they're placed in the Python languages, how you generate them, and then the basics of how you work with them. The final part on building an import linter is going to take step-by-step through building a more complex example, learning additional things that might help if you're gonna actually build one of these linters or analyzers of your own. We likely won't get into any of that, but this link will be public and you will have access to that if you want to look at that afterwards. So let's dive in now to the introduction to the ASTs. So AST stands for Abstract Syntax Tree. It represents the structure of the source code, your Python code, as a tree. The nodes in this tree are language constructs, so module, class, function, and in fact, if I just peek over here to the docs, you can kind of see that already, right? We have module, we have function definitions, we have class definitions, returns, right? are all language constructs we're familiar with. So each node in this tree has one parent node. So for example, a class would be a child in a single module. If you have the same class in different modules, it's actually a different class. Parent nodes can then have multiple children. So example, in that module you can have multiple classes. In a class you can have multiple methods. So to get a feel for what an AST is, let's take a look at this very simple snippet here, which is found in the snippet directory of that repo, and it's in greet.py. So it's just a simple class called greeter, and it has two methods, an init method and a greet method, both of which are also very simple. So this is the AST representation of that code in Python specifically. So you can see there's several types nodes. So up here we have the module So only one because we had the one file. We have our class definition, which is the greeter, which was the only class Then we have its init method The init method is arguments, information about the arguments, default values. We also have the argument name and then also the return annotation and then also for the greet method we have a similar structure here you can see this is actually the logic of the function all that in that single return f-string so ASTs are all around if you've used any of these tools like rough or black for linting and formatting you get you've used ASTs also in documentation tools like Sphinx and NumPy doc which we were just talking about. Upgrading tools like PyUpgrade, Marimo for next generation notebooks, type checkers like MyPy, code security tools like Bandit, coverage and testing tools, also testing frameworks, hypothesis and PyTest. And specifically in Python, an AST represents syntactically correct Python code. So you You cannot generate the AST if you have syntax errors. The AST is actually created by the parser in Python as an intermediary step when it's going to compile your source code into bytecode. That is part of the process that actually runs it. And I've linked here to, this is a link to the, the screen is like getting worse and worse. That link is to a page in the CPython repo, which goes through the process of how they actually parse the code and create the AST and all that, so it's an interesting thing. And the AST access, if you want to program with it, is via the AST module in the standard library. So in today's tutorial, we are only going to use the standard library and have access to all this. So let's see our first steps of using the AST module. So if we want to parse Python code into an AST, The first thing we have to do is read the code in so that we have a string of all the source code, okay? So here I'm just using pathlib, so I import path from pathlib, I provide the path that I wanna read in, which is that greet.py file that we just looked at, call the read text, so now source code is just the contents of that file. And we saw already that that was syntactically valid code, So I can read this in. I'm gonna import AST, and then we use the parse function, passing in the string of the source code. This, again, parses syntactically correct code and returns back our AST, our tree. And in this case, for all that we're gonna be using today, it will be rooted at a module node, and you can see that right here. So one way of looking into it, Unfortunately, the visual I showed you is not native to the standard library, so your way of looking through it will be through this dump function. So you can use ast.dump, you pass in the tree that you want to inspect, and here the indentation is just so that you have some logical flow of how it's structured. So the first thing we have is our module, which we saw was the root of the tree. The module then has a body, and that is the contents of that module. So you can see in here that
Speaker 3 [11:54]
So this is a very basic level question. So can you explain to me what
Speaker 2 [12:06]
syntax tree
Speaker 3 [13:04]
You say little using it? How do you preserve all this white space and the comments and the stuff? I'm going to talk about that later. There's a reason they're not in there. Yeah. Maybe I misremember, but I think you said that AST is used by Python when it compiles to my code, right? But it's not that exact library, right? Or is it maybe because C Python is written in C++, right? Yeah, this is an interface into it. I'll show you an example where we can actually take the code, turn it into AST, change it, compile it back, and then run it. So you can do it with this, but... So is the AST library actually under the hood C or a C program?
Speaker 2 [14:04]
I think the ST, like,
Speaker 1 [15:39]
quick hint that we are very sorry about the screen the problem is with the project and we can solve it right now but the slides are perfect in those two screens so if someone needs to have a better view I recommend to use those two big screens at the back Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. The last minute of the exercise. So if there is any other question.
Speaker 2 [21:38]
error so you can now see the the symmetry between those two things and how this is part of that process you will see an interesting thing here where like the new underlining where the issue came from how it doesn't quite understand how to deal with it in this case right they're all misaligned right it should be misaligned it should be aligned to here but you can see that it is correct up here so it's like the
Speaker 3 [22:23]
I made an F-statement with triple equals.
Speaker 2 [22:53]
and you never defined a you get a name error right that a it doesn't know what a is so i can parse that and there's absolutely no issue with it right there's there's nothing wrong with it syntactically you can add five to a whatever a is but it doesn't know that a is not defined so that's also like a thing you can think about like if you're in an editor it's able to tell you you have an unused variable or you have a variable that was never defined and part of the way it's doing that is with ASTs. So you can detect that, you can track where things are referenced, which is a little preview of what that third section is about. And you can make those decisions that flag things that aren't used or flag things that weren't ever defined. Okay, so now moving on to where we're going to spend the most of the session, which is actually working with those ASTs. So now that we know how to create them, how do we go about traversing them, manipulating them, and looking at the code. So the first thing I want to talk about is the fact that these ASTs, there's a lot going on in each of those nodes. So we already saw how the tree massively expanded even for very, very simple pieces of code. There's also the element that when we were talking about the modules had a body and the class had a body, but not everything has a body. Sometimes they have a different attribute that also has AST nodes that need to be explored. So you can't really rely on grabbing a node and then manually accessing each of these fields. There needs to be a different way.
Speaker 3 [25:40]
that tree again because I
Speaker 4 [25:45]
that has two parents and one son.
Speaker 3 [25:51]
four principles that no, no.
Speaker 4 [25:52]
no node has two parents.
Speaker 3 [26:01]
It's just the visualization. The tree actually does not have...
Speaker 2 [26:04]
uh so if you traverse you will end up at the load thing but it's like it's just that you're loading and you're loading this same x twice oh okay thanks okay so going about how we actually traverse the ast so first thing we need to do is read in that code again so we're going to work with that assert dot pi snippet so i'm just the same thing we've covered before we import the ast we import path we read it in and we have our tree parsed so the first function that we can use is iter fields so each of those nodes has like a underscore fields attribute that tells like what are the things that it has available on it and this will just allow you to go through it so for example if i call this and pass in the root of our tree which which is the module, it tells me that there's a body, which is what we were looking at before, and this type thing that I was excluding from the previous visual. So there's really not much on the module itself. But if I then grab from the body, the only thing in that list, which is the function definition, then we can use iter fields on that function definition, and now we get more information. So this is the information that was on the function node. So you can see that some of these are not AST things, and then these particular three are AST nodes that we would then want to look at. So this is giving you, like, given a node, what does it have? It's nice for getting your bearings when you're working with a new node type. So we can see that functions would have the args, and this would be an arguments node that would need to be explored, and then the body, which has the assert and the return, which is what is actually going on in that function. So then we have iter child nodes. So this lets you grab all the things that were AST nodes. So if we looked at the output here, we see those three are highlighted. If we just ran iter child nodes on the function definition, we get those three pulled out. So we could iterate at that level of what the function definition has, all of its child ASTs. nodes, but we're not going to its grandchildren or great-grandchildren. So direct only. So if we want to actually go beyond that one level, we need to do recursive behavior. So there's three ways of doing this. You can walk the tree, or you can use a node visitor or node transformer. We're going to go through each of those, but know that it's building upon those two functions that we just saw. So to look at how we would walk the tree. This is recursively going to go and visit each of those nodes, but it's not going to do it in any specific order. Every time it finds something, it's going to go and off and do something, and then come back and add things. You can't rely on the order, and we'll talk about what you need to do if you need the order. What we're going to do is we're going to look for any asserts that don't provide a message, and we're going to inject a message into the code. We're going to start up here and we want to have something that looks like this. We can do this with the walk function. We simply create a for loop and we're going to pass in the tree to the walk. We get a node from the tree at each iteration. Then we check if that node is an assert node and also if it has no message. Again, you have to know how the node behaves or how the node attributes has so that you know okay message is the one that I want right and that's also where having the ASC docs bookmarked is very helpful or using something like iter fields because you can quickly see okay what does this node have what am i what field am I interested in and then you can write that so here if it's an assert and it has no message then we are going to set the message so if you just want to put string somewhere you can use the constant node and it just passed your value, so string or a singular value as well. So we set that message, and then since we're actually going to run this code, we have to call this fix missing locations utility function that they have. So we injected this constant node, but as far as the AST is aware, it's like, where does this exist? It has no line number, no offset. So if you try to compile that, it won't work because it doesn't understand what that is. So this just goes from that root node that you're giving it down the subtree and fixes adding those line numbers that are missing for you. So once this loop finishes, we've updated all of the asserts that we care about. And then we can take this tree and compile it. So you have a built-in function called compile. you can pass in the AST here you just have a have a string which is the module so when you see trace back this is like modules standard in like that that's coming from this type of thing and then the mode that we want to do so just put exact this gives you back a code object and then you can call exact on that so by doing this even though we've never defined that duplicate list function The second you run this, it now exists and you can call it. So here, I don't get a name error. I pass an input that will fail the assert, so I can get that the assert failed, but I don't get a name error. It knew that function exists, and you can also see that the assertion error provided the message that I had injected at the top. So that's one way that you can go about editing the code, intercepting it before it's run, change the behavior and then run it yourself okay so now going back to a question we had earlier about comments formatting in the file so how do we convert things back once we have an AST can we go back to source code so there is an unparse function so here you can see I've taken the tree we've just manipulated I can unparse it and that looks exactly kind of what I told you it look like right but it's not always gonna work out so nicely so for one it's not recommended that you use this if you have large trees because it can run into recursion limits and then the other is that it's going to be equivalent so some things are represented
Speaker 4 [34:58]
Thank you. We saw that the double quotes were changed to single quotes, and I think some people are wondering what happens if the comment itself contains a single quote. If the comment was removed. Sorry, or the password itself, or the doc string itself contains a single quote.
Speaker 2 [36:11]
through the snippet ingredients.
Speaker 1 [36:29]
Okay, so we go to the second round of the exercise. Please raise your hand if you have a question Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. 5 minutes for dealing with the exercise. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Okay. the last minute so if there isn't any question and if you are ready we move on to the
Speaker 3 [45:47]
from looking at the AST.dump output, it looks like a docstring really is nothing. There's no docstring, no type, it's just really a string constant that happens to be at the right position, right? Is that correct? That is exactly it, yeah, yeah. What's also interesting is, if you've done things like with Sphinx, how you can put a docstring underneath just an attribute, and it's able to read that in, this does not know that it exists, right? So that also gives you an appreciation for some of the other things that happen, yeah. A question, the AST has a lot of functions, but I looked at the objects, I would expect the nodes to have a lot of methods, because they all seem to inherit from the AST class, and this class would provide all the methods, but they decided it has functions. Do you have any idea why this sign is like this? I do not. You don't know? No. It could be that there's a reason for this, because it doesn't work for something like this? Because this object doesn't expect to have base loss?
Speaker 1 [47:13]
So we move on. Thanks.
Speaker 2 [48:11]
in that snippet, and then actually walking the tree. So I have the suppressing bit here, so with context lib.suppress. I know I'm gonna get a typer if there actually isn't allowed to be a docstring, so there's no field that will store the docstring. But if I'm able to get it, then I will say, get the docstring on the node, and I'm just checking if it's falsy, so it's possible someone had a docstring with just a space, and I'm not removing that, but that would be also something to look for. And then also the bit here about, I wanna say what was missing, right? So you need some kind of name for the node. So the module node does not have a name because when you're just giving it code, you're not necessarily giving it the file to read and it doesn't know where it came from. So I'm just putting that default value module here so that when run it through, we see that there are four missing doc strings in this file. The module's missing one, the greeter class, and then both of its methods all lack doc strings. So hopefully everyone.
Speaker 3 [49:27]
Why was the type error suppression needed? So if I call this on any of the nodes that cannot have a docstring, like a constant, like the docstring itself that I'm looking at will be, oh, it can't have it. So that's going to raise the type error when it hits this. So you only get through if it's allowed to have a docstring. That's instead of explicitly checking that it's a function.
Speaker 2 [49:58]
You could say, there's a bunch of ways you could do it, right? You can test the specific node type, and then you would probably say, if it's an instance of module, class def, function def, async function def, that's the tricky one. So it's like, you never know when they're going to add a new construct, so this is the easiest way to do it. You just, okay, I just want to look through all of it, right? Thank you. We'll also see more suppressing later. Okay, so I mentioned before that the AST walk is any order kind of just wanders around. When you want to have the order, and also in the case of docstring, maybe you want to flag the init method as not having a docstring, only if the class also doesn't have a docstring. So you might want to know information about its ancestry or its descendants. So in order to do that, you need to do a depth-first traversal. There's two ways to do this. One is with a node visitor, so you go to the nodes, you observe, but you don't make changes. And then there's a node transformer, which is gonna have the same behavior as the visitor, except that you're allowed to change the AST as you're walking, or as you're traversing. So we'll start with the node visitor. So to create one, you're gonna subclass the AST's node visitor class. And then for each node type that you want to visit, you're just gonna create a method that's visit underscore, and then you just put the node type exactly how it is. So it's gonna look funky, you're gonna have weird mix of snake and pascal case or chemicals, whatever you wanna call it, and it's gonna look pretty nasty. In the past, a lot of linters didn't like that format, but that is how you actually target these things. And so by creating that, the node visitor's automatically gonna know that whenever it counters a node of that type, it needs to call that method on that node. So that's your way into it. So to see this, we're going to do the following. So we're gonna take a look at try, accept blocks, and we're gonna be detecting this anti-pattern of doing try, accept, pass, okay? And instead, we want to encourage the use of ContextLib, and you will find linters today doing this exact suggestion. So we want to change this to be like this. So with ContextLib.suppress, and we would just map the error here, and then we try to delete the password. So we've just reformatted that code, it does the exact same thing. So from that, we now know we need to visit try nodes. And for each try node, we want to look at the accept information, and specifically we're gonna look at what is it accepting, because we're gonna need that information, and then what is it doing in the bot.
Speaker 1 [55:09]
Any guess?
Speaker 3 [55:14]
It hits the first, the outer triangle. It doesn't, it says it, I won't look.
Speaker 2 [55:26]
it hits that try block and then because we didn't tell it to do anything else it stops and then the traversal goes back up so the depth stops at that point and goes back up so here's the traversal that's actually happening so we're going through and then when we hit the try here we just move on so there was a whole another part of this tree that never got explored so for this we have the generic visit method so whenever we don't define the visit method so we define visit try how did it visit all those other nodes it's calling generic visit which is just going to go to the node go down all the way until we find something that we have a special method for so because we didn't call it that it stopped when it hit our node and then that ended the traverse so all we have to do is say at the end that, oh, and also visit everything else from this sub-tree. So that's this generic visit. So that allows the traversal to continue. Also notice it's not in the if. We want to go there regardless. We don't want to suddenly stop, okay? So with this small change, if we now run that code, it now finds the inner one that needs to be replaced on line five. So That is a key, key thing to remember. Whenever you're writing these, that's probably the first thing you should write just so you don't accidentally forget and get confused about what's going on. This is the actual tree. It's significantly different from the other ones. If you look, this was the trial we stopped at before. This was the one that had the issue. All of this was not being explored. That brings us to our third exercise. This is going to be the most complicated one of the session. So now we're going to create a generic exception visitor. So we're going to be looking for two things. I have examples. It's not exhaustive here of what you're looking for. So for one, we want to flag whenever you have a bare except. So you just say except, and then there's nothing there. We also want to flag the use of generic exceptions. So if I say raise
Speaker 1 [58:48]
So more 15 minutes for doing another exercise
Speaker 3 [58:57]
What would be a sensible way to make sure that I visit all the nodes that I want to visit? So you can use generic, but also can I just say I want to visit everything and I want to make sure I visit everything? Is there some kind of measure that tells me that I didn't forget anything? So you could override the visit. Well, there's an example. I don't know if we'll see it here, but you can override visit. You can override generic visit. Either way, you're going to go to everything as long as you don't have a visit thing and forget to call that. Yeah, so I could go to everything, count how many nodes I visit, and then I write another one, and then see how many I've lost. Sometimes I can do some statistics, how many nodes I've lost.
Speaker 2 [59:46]
than super visits.
Speaker 3 [59:48]
That would be interesting how many nodes you have and if you do them code changes, how you increase the number of nodes a lot or something like this. That would be, maybe it's a use case to see how your code changes, like complexity of your code.
Speaker 1 [64:35]
It's very nice to hear the keyboard sound so it means that you are dealing with exercise
Speaker 2 [64:44]
more nine minutes
Speaker 1 [64:46]
Any question, please raise your hand.
Speaker 3 [69:53]
Visit underscore myself. I try to do visit except
Speaker 2 [70:15]
This would be your best bet.
Speaker 1 [72:50]
If there is any other question we move to the last 15 minutes of the talk and For sure if someone is interested more or any other Explanation you need for sure you can reach them
Speaker 2 [73:49]
So I just had a curiosity, was anyone able to do the get source segment part? Raise your hand. Okay, good, some people got it, cool. Okay, so here I have an example solution. You can also find this in the examples folder. So here, just to format things nicely, I'm also going to import from text wrap. So I have the dedent and indent. And I'm going to start with the declaration of the class, so the generic exception visitor inherits from the node visitor. Now to make easier work of that get source segment, I'm actually going to change the init, so instead of accepting nothing, I'm now going to accept the source code string. I'm going to store that for later, and then I'm going to parse the AST directly in here. So both of those are now stored. To actually print that segment, I'm going to have this helper method that's going to receive a node. it's going to call that get source segment passing in that source code that I'm storing and then the node and this bit is just to get the indentation so it's easier to align everything because we're going to have multi line and then I would just print that bit for the actual AST processing logic we have a visit raise method so this is going to handle the raise parts that are either either a raising exception or a raising exception with a message. So this bit captures just raise exception. So we particularly look if the exception is a name node. In that case, we just can check the ID and check that it's exception. The other case is where we have raise exception and then something passed in. So that now is a call node and then it has a function which is then a name and then has an ID so we can grab that here. So those handle those two cases, and then we would just print that we encountered that generic exception, we print the line number, and then we also use that helper method to print a piece of the code that was violating that. And then we also have our friend generic visit, right? We don't forget to continue the traversal. So that handles the raise part. Then we have our visit accept handler. So we have to handle, again, two cases here. The first is we check that it's not a bare accept. So in this case, if we had our exception handler and there was no type, so it was none, that means that we were not given anything and it is a bare accept. So we can provide that information. And then if we did have it, so that was what the walrus operator did up here, so this is now the exception type, and so in this case it was not none. So it is a name and we have our ID and we can check. So also we're not handling anything like exception groups but there are separate nodes for that as well. You can see in the docs. And then again we have our call to generic visit. To make things a little easier since we already have the tree in the class now, we have a run method. You can override this and you can do all that. I just prefer to leave that alone and not mess with it too much. I do have examples later in here where I am overriding both visit and generic visit to do some more complicated behavior, but I still have a run method, which is like the outer interface that I'm using here. So when I actually call this, we have similar logic to before, where we read in the file, and here we instantiate, but now we pass in the source code, and we still will call the run instead of visit, so that passes in the tree. And here you can see examples of the types of things it caught in that snippet, and you can also see that it's formatting and this is the piece from the code where it's showing me where the particular thing happened so this is the bear except this is generic exception in the except and then generic exception in arrays and also here so you can see different examples of the types of things it was catching okay so that now brings us to the other way of traversing the final way which is the node transformer this is the one that can actually change the nodes. The visitor cannot change them. So this inherits from the visitor, it's using a lot of similar things, but again it can actually change things. So if you've noticed in all of those visit methods so far, we didn't return anything, right? Which means we're returning none. So if you just swap out your node visitor, like inheriting from node visitor to inheriting from node transformer, you will find that you end up deleting everything you visited. So So the behavior here is that when you visit a particular node, you have to return something. So you return none to delete that node and the entire subtree that goes from it. Or you can change it in some way, return the changed node, and that then will update the AST. Or you can just return it untouched, and then it stays in the AST. So if you see any weird behavior, it's probably because you forgot to return a node. is a big gotcha here. So if we go back to our try-accept-pass detector, we can change it from being the node visitor, which was just able to look at this and say, hey, you should use context lib, to actually doing this transformation itself. So taking the code on the top and converting it to the code on the bottom. So here, again, we're going to have the same imports, AST, text wrap again for formatting. Now we are going to inherit from node transformer and not node visitor so we have our try accept transformer. I'm not gonna print the source code in any way so here I'm just gonna store the tree I'm not storing the source code but I'm also storing this boolean has changed so here we're just gonna be very simplistic about it if I've changed the code I'm just gonna know that I need to insert that import context lib I'm not gonna to check if it's already there. There are further slides where you can see how to work with imports, but we're just going to ignore that bit for now. So has changed will tell us at the end, do we need to add the import? So now also the question was like, kind of what the question was before, like how do you know which nodes to work with? So one easy way of getting this to work is don't even try to deal with constructing an AST for the width block. Like we want a width block like this so what we can do instead is create this helper that is going to generate it for us by first taking this code that this is what we know a suppress should look like we're gonna have with context of suppress we're gonna have in here some exception type and we're gonna have some body so for now these are placeholders pass and exception are placeholders so we're going to read this in and I'm also remember we read this in this is now a module which we don't want but it's a module of a width block a width node and we want the width node so that's why we take this the body and we take zero so that's the only thing inside so now this is the AST representation of that bit on the top so to actually change it so given that node that we want to rewrite this will be a try, right? So we're gonna take the try node, look at its handlers, the first one of its type. If it has a type, we're going to update this bit in the with block to be that exception type. And we do it this way because if there's no type, we need to have something here so that just stays unchanged. So either if this is a keyer, this becomes keyer. If it was nothing, it stays like this. And And then we now need to rewrite the body. So whatever was in the body of the trie is now the body of the width. So we just move that over into here. And then we will return the width block, because that's what we actually need to return for the AST to be updated. So then we have our visit trie. The first thing we're going to do is go down the tree. So handle all the descendants all the way to the bottom. That will bubble back up. And once we get here, we're now just replacing at this particular node. We have our logic mostly the same from before. The exception is that now here we have this bit, where we now say, okay, get the suppress block that we need for this, passing in the node. So now this is the new node. It goes in as a try node, and it comes out as a with node. Then we know we've changed, and we return that with node. then we have our run right so i mentioned before we'll need to add an import so we're still going to visit so we visit the whole tree we've updated the tree this now is the tree with the tri nodes replaced with the width where it needs to happen and then if we see that we've made a change we need to add reject that import so we'll just update the body of the tree which remember the The tree here is the module node, so we're just adding, at the very top, an import of ContextLin. And then we add the rest. And then what I mentioned before about making sure the line numbers are correct in case you want to actually run this code, we just handle that here. This works from the subtree, so it will do the entire tree, wherever we've added, and return that. So at this point, what is returned here is the altered AST. So if we go about running this, we read in the code, we instantiate with source code, and then we call the run method. You can see that we're still printing that we detected. So we detected a try except block line 2.
Speaker 1 [84:50]
Okay, so five minutes for the last exercise Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. the last minute for the exercise then we are gonna to close the session so any question or sharing ideas
Speaker 4 [88:40]
My understanding is that Python itself is parsed via ASTs. I'm seeing a lot of examples here of using Python to parse Python. I guess the thing that I've never been able to reconcile in my head is, like, it's kind of a chicken and egg thing. Like, is it, at what point is it, like, something else? or is it just the fact that these tools are sort of so sophisticated now or so mature that we just have a Python representation of ASTs now but at some point I guess it was something else.
Speaker 1 [90:23]
1-2 minutes maximum
Speaker 2 [90:54]
exact same thing we had with the walk except we're storing the result back in node so then we can return the node and update the AST. Okay so I mentioned before that there's a whole other section in here if you want to get more into the weeds on things you can do like tracking the ancestry and I have an example here where you look at finding unused imports and missing and masked names and that uses stacks to track ancestry and scope and so you can see that built up step by step so that is available to you and I also have a keynote that I did last year at PyCon Lithuania about using ASTs and that's specifically focusing on doc strings and type annotations and signatures so if you want another
Speaker 1 [92:06]
the interesting presentation and thank you all for your attention your attendance enjoy the rest of the conference and it's
