"Easy Python": lies, damned lies, and metaclasses

Speaker 1 [00:02]

Hello. Can you hear me? Yeah. That's me again. Probably you have heard me the last day. I talk always about Django. But right now, surprise, I don't want to speak about Django. I want to speak about Python. Who knows what Python is not only the animal who works with Python? Yeah. Thanks. Thanks. It's Python conference. And right now, I want to speak about Python. If easy Python is lie, then lie, or this is lie only about metaclasses. Who knows? Probably me, yeah? And at first, I want to present two authors who worked with this presentation. First speaker should be Gregory Petrov. He is right now in Russia. They cannot come to us. That's why only me here. My name is Maxim Danilov. If somebody have any question, please ask us after this presentation or you can ask us by email or in our networks all contacts you can see here and slides you can download by link and a little bit about us gregory works for company euron gregory is development relations and this is a big software company which creates software by demand. My name is Maxim Danilov, I work for, I am owner from Veprsoft company, we solve complex tasks on Python. We both work with different technologies, we both We both work with big projects, really big projects, more than 100,000 lines in Coderbase. And we both every day work with complex code. In this talk, I will share our experience about complex code and how we can work with that. But the first question, what is complex code? Somebody knows what is it? Or somebody here works with complex code, for example, me, who works with complex code. What is complex code? For me personally, complex code is hard to understand. At the second, for me personally, complex code is hard to maintain. And for me personally, complex code is hard to change. Of course, I know we should not change something that exists, we should only add something. But sometimes it happens we should change something. And complex code is really hard to change. And if I summarize these sentences, I can tell complex code requires a lot of my working memory and according to Miller's law we all have a short time memory and this memory can contain from five to seven elements at a time. Who knows about Miller law? Yeah, thanks. Miller law from five to seven elements for example for So, developers, seven, for me, for example, from fear to five elements in one time. And somebody can tell me, hey, Max, do you can use visual programming? But I can answer. Visual programming is a pseudo-value. It cannot help me to work successfully with my short-time memory. For example, as a pseudo-value for developer, alcohol, once from them. But visual programming, it's perfect, transparent. We can draw the small pictures, we can draw connections in between. It's really readable. But what happens if I start to draw more than 1,000 elements and more than 15,000 connections? I have one project with that. It's impossible to maintain. It's impossible to work. That's why I call visual programming is pseudo-value for big projects. The next pseudo-value can be easy language with fast learning speed. For example, language with low number of instructions. Of course, brain fog. Only one instruction. But nobody can tell me what happens here. Who knows what is what? Okay, this is Hello World on dialect from Brian Fagg. Once more time, Pseudo-Value cannot help me to work with complex code, but every language offers us different things to work with our short-time memory in an efficient manner. For example, we all have the ability to split big blocks of code into small parts. For example, spaghetti code. Who knows about spaghetti code? Of course I know. And we can split spaghetti code into functions. After some time, we start to create more and more functions, and we should organize these functions into, for example, classes. We organize functions in classes, classes we can organize in modules. We already organize it, but complexity or projects is big and complex and I can say we can organize modules in libraries, but this This is not enough. And right now, we see the software complexity problem. Look at this picture. This is perfect example of developer. Developer can see only the small part of whole project. And this small part, it's easy to work. But the developer cannot see the whole complexity from the whole project. And if we don't start to reduce the complexity, this project starts to eat us. This is the short way for burnout from complexity in project. Who knows about burnout because the project was too complex? I know. Yeah. I hope it's all okay with you and with me. And every language like Python can simplify our work, this big project. But how Python, exactly Python do it? I would like to go to Gwida tweet. Gwida tells us, deep magic things hide in some places and it's designed to quietly help us. Python hidden magic, as known as layers of abstraction. And Python has more features to fight with software complexity problem. And I made a review of important abstraction layers. The first famous abstraction in Python is special syntaxes. Python offers us to write perfect, clear, readable code without additional elements. It's perfect. Also, Python gives us the new operands or operators, for example, Valorous operator. Valorous operator designed to work in logical constructions or internally operators. It's perfect. I can create a new valuer in one line. But what if this can help me to decrease the code base? To make the current part of my code, to simplify current part of my code. But what if I start Valorous operator in function call? For this example, I emulate named arguments. It seems already not too transparent what happens here. But what if I start to use Olros in lists or in sets? It already has low transparency. And if I start to use Valorous Operator in comprehensions, it's really not transparent. The complexity from the whole project grows if I start to use Valorous Operator in different manner. The next syntax element is argument separation. This is a perfect way to document how happens function call. At first positional only, at second positional and keywords, at last keywords only. But every time if I meet this construction in my code, I should remember at first positional, after that positional and keywords, and after that keywords. Every time I should take in my short time memory this rule. That's why I can say complexity from whole project grows if I start to use it. The next powerful thing, which I love, comprehensions. This is my real love. We can different art of comprehensions, list comprehensions, set comprehensions, dictionary comprehensions. But what if I start to use it all art of comprehensions in one time? We can receive something like that. This is example from SQL Alchemy. I think the developer from this code cannot tell me what happens here. from our project grows. The next thing, pattern matching. It was a perfect talk before yesterday, I think, about pattern matching. But for me, the transparency from pattern matching, low. This example I take from specification from pattern matching. And for me, it's already hard to understand what happens here. Of course, probably I don't work with pattern matching too much. Probably I should work more. But complexity, if I start to use this construction everywhere, complexity from our project grows. The next abstraction layer. Next feature which designed to help us to simplify our project. Types. Types. Types has a complex specification, and types help us to document our code already in code, but if we start to use types without strict guideline how I should use it, we can receive something like the example from stdlib as a standard library in Python, and I can say this code is not more transparent. We lose transparency in this code. Of course, the developer of this code can tell me what happens here, but not everybody. The perfect abstraction which we can use to simplify our project as a ground, as our basis, Python data model. Python data model works everywhere in Python and we have more than 125 Dunder methods and If we start to use it or to override it, including delete, including subclass checks, set names and more, of course, transparency of our code goes down and complexity of our code grows. The next feature which we can override is, for example, multiplication, double multiplication, math operators, ellipses, none, we can override every element here. And we can receive some domain-specific language. It was lightning talk yesterday about creation of domain-specific language. Perfect lightning talk. For example, we have NumPy domain-specific language. It's easy to work if I work with that every day, but on the other hand, if I receive the complex project without documentation, and in this project, developer starts to create domain-specific language, and I cannot contact that developer, I cannot answer what happens here. knows what happens here on this code line? I don't know. I cannot answer. The best possibility to work with this code, we should override it. It's not possible to work. I don't know what should happen here. The next perfect abstraction which knows everybody is scopes. This probably is a perfect way to split, to decompose our code in blocks by five elements. We have, for example, local scopes. But sometimes in code, we need to use non-local scopes. And sometimes we use also, we should not use, we receive a global. And after that, we also have Boolean. And after that, we can use inheritance. And we should remember about method resolution order. And after that, we should use context manager. And it's already complex to remember all these features from Python. And around these methods, we have already an established ecosystem. What I mean. For example, with new syntaxes async await, Python gives us a possibility to work with async content manager. But if I don't know how synchrone content manager works, I cannot answer how works asynchrone Content Manager. This example I take from IOTTP, and probably not everybody can tell me what happens here, but I know what is it. The next abstraction layer, perfect abstraction for us is metaprogramming. For example, decorators. Everybody knows about decorators. It's perfect tool to change one object into other object. One function into other function. One method into other method. But if we start to use it in our code without strict guideline, we can achieve something like that. Sorry again about Django, this is an example from one big Django project, this is a test case, and transparency from this code is too weak. I cannot read it, I cannot understand it, of course I know what should happen, but sometimes it not works. This code is already complex. The next feature which should help us to write easy code is metaclasses. Who works with metaclasses every day? Really? Okay. For example, if somebody works with Django. Django use metaclasses in models and in forms, somewhere else. But what is metaclasses? Metaclasses has complexity by default. Why? If I start to work with metaclasses in my developer environment, I should install for PyLint additional plug-in. Why? PyLint not work with metaclasses alone. He needs help from me or from plug-ins. And if we start to use metaclasses together with context managers, we can achieve something like domain-specific language. This example I take from PyQt and probably nobody can tell me what happens here. Here I want to create a graphical user interface with PyQt. But this code is already complex. Complex by default. And probably we can try to use something else to fight with complexity. And the next thing, next feature which I can use is reflection API. API we can use if we want to catch some exceptions and we want to know a little bit more why this exception happens we should to introspect into threads into function there happens there shit happens and we can collect all variables But if I start to use this code block in Sandia in my project, complexity grows automatically. Why? Not every developer in my team can answer me easy what do this code block. The next abstraction, which I want to describe, this is structured concurrency. What I mean is that if I start to write async code with callbacks, this is normal async code, but this is a short way to receive callbacks help. That's why Python offers us the other art of syntaxes, I think await syntaxes. And after that, these syntaxes, I can write structured code. For example, this example I take from Trio, there's a perfect library which helps us to write structured code if else await for function call. It seems like normal code, but it works asynchronously. And complexity is I don't know which part of code in which time works. It's a little bit difficult to manage it or to find error in this code. The next complex thing in Python is async coroutines. If I call, if I await other coroutines from those coroutines and the next coroutines returns me async generator, it's already complex to understand what happens. Probably it's already understand my sentence. This is already complex. Ask Shilova, this tool helped me to organize all my async parts of code and this tool starts all functions which I want to start. But answer how it works, I can answer, but I know complexity from those code is too high also for me. And if you start to use in additional async protocols like I await, I enter, I exit, complexity and transparency from our code grows incredibly. And somewhere these codes start to beat us or start to eat us, devour us. The next complexity which I want to describe is dependency management. For example, complexity in dependency management has their own symbol, platypus. That's a symbol of complexity in dependencies. What I mean with that? It's complex not only for junior, for middle developers it's also complex. Why? For example, I want to create installable library. I want to, I should to start with setup API. After that, I should not forget about requirements text there. After that, probably I should add PIP file. Probably I should add an additional PyProject tomo. OK, I create it. I create it. After that, I should to think how I want to install it. I should use Python or not. Python 2 or Python 3? Ah, OK. Right now, Python 3. Python 3 means . But I forget something. What I forget? Probably, at first, I forget about the virtual environment. And after that, I should use PIPX. After that, I can wrote documentation, but sometimes we have documentation which comes too late. These documentations don't describe the modern technology which we have in our project. It's hard to work with documentations, but sometimes our install simply doesn't work. On my operation system, I cannot compile some libraries. For example, you can try to install uvvizgi on Windows or PyGraph Wizard on Windows. It is really hard to compile. After that, we receive relative import and absolute import. After that, we can meet circular import. Somebody knows about circular imports, yeah, hard problem. Sometimes we can meet import log. And after that, I should update all in my libraries to use a new feature which I can which I already integrated in my model. sometimes I want to change import on fly and this all in dependencies made make our dependencies complex the next abstraction which help us to work with big complex code is this is exception error handling we have exception I have exception which I want to catch, I have exception which I want with this exception I want to fall. I leave this part exception which I want to catch. If I work this exception from my project, I know about these exceptions. But what if I start to work these exceptions from another library which I cannot introspect. What I want to catch? Somebody knows? I don't know. And this is already complex to find what I want to catch. The next abstraction layer which Python offers us is a standard standard library. The standard library is a perfect package from abstract collections. For example, we can import func tools, library to work with iterable objects. We can import text wrap, perfect library to work with raw text. We can import something from future. We can use a num to collect all flags in our project in one container. We use timezone and daytime. And this all we start to use in one project, this all increased our complexity. The last abstraction which I want to describe in this talk, this is our Pythonic way. Some tricks, code snippets which we use every day in our project. I think this symbol, Hydra, this is perfect example from our Pythonic way. We can do many things in different ways. Every Pythonista knows how it works. We want to receive element from iterable object or none. Or we want to receive an element from a hashable object, or we want to receive none. We want to check if this object is none or not. Or we want simply to check if this object is something like true or not. Sometimes we use string concatenations. Strange string concatenations, especially in vertical lines. In vertical lists. the comma is omitted and shit happens. We can use callable objects like attribute container. We can change our closure after this closure created. I can send a special argument to change closure. We can use class body statements and achieve these statements in instance of class or simply in class. Every code block is a container and sometimes we want to maintain to operate this performance from every container in our code block. If we start to use our Pythonic way, we increase our complexity in our project and we increase complexity in our work. I try to summarize all sentences from this presentation. If our project grows, in a moment, sorry, if our project grows, the complexity grows automatically. If we start to use more abstraction tools, the complexity in our projects grows automatically. The next idea I want to tell, small complex things multiply complexity. This is an example from a computer game Path of Exile. Who knows this game? Yeah. Good luck. And in this game, we can collect different damage modificator. And this is only a small part of our damage. But we small things multiply. And we can achieve till one hundred million damage per second in this game. Small complex things in our project multiply complexity. Python is a complex and it's designed to create complex project and it's important. Microservices cannot help us. It's not the answer I can imagine how I work with five million rows in quarter in project but I for me it's difficult to imagine how I should work with thousands of microservices in one in one time and at less at last Python can be easy but Python is not easy and that is intended and let us to discuss it. The second speaker comes right now and we try to answer on your questions. Gregory are you here?

Speaker 2 [33:14]

Yes, our obligatory Zoom spiritual check, so I can see what you can see me, but can you hear me?

Speaker 1 [33:27]

Yeah.

Speaker 2 [33:34]

We have some millers here you can recollect that from slides alongside with procrastination that stumbled upon us while we prepared our talk. But Maxim, thank you for being my copy on this stage that was amazing. And now we both can answer some questions about Python complexity and how it helps to handle big complex projects.

Speaker 1 [34:13]

We are ready.

Speaker 3 [34:13]

Yeah, thank you. Thank you, Maxim, first of all. That was a fantastic talk. Thank you also, Gregory, for joining us for doing the Q&A. Really great to have you both. So we have some questions, and I can see some more coming in. The first one here is, simply put, what's the solution?

Speaker 1 [34:14]

Yeah. If I summarize my talk, my presentation, the first solution personally for me, strict guideline for project. This is solution which can help me to set borders for complexity in my project. Yeah. Gregory?

Speaker 2 [35:05]

Yes, as a hobby neurophysiologist, I can also tap on the flavor of evolution and gradual things. You know, it was mentioned in the talk that Python, it has gradual types, but we are not limited to gradual types. We can also use this gradual approach to our complexity. So if our project is small or we are working on the small part of our project or even on some micro service, we can exclude all complex things like meta classes, like decorators, like advanced things. we can start simple and keep our code simple. But the trick, the solution, when our complexity grows, when our code grows, we add advanced Python features, we make our code more complex, But at the same time, we fight complexity and we make our code more readable.

Speaker 3 [36:27]

So, Gregory, do you like Java more?

Speaker 1 [36:32]

Rust problem.

Speaker 2 [36:35]

um uh actually um uh i have uh python and uh i have uh ruby uh here i really do enjoy uh ruby but uh you know java and go programming uh language they were mentioned too in our talk My personal opinion is that language simplicity is a pseudo-valuer, it's not the solution. If your language is simple, really simple at its core, like Java or Go, you can enhance adoption. You can attract lots of developers as Java and Go did at the time. But if your project became millions of lines of code developers, they start to struggle. Go developers, they never create a Go project with millions lines of code because Go is for microservices but with java as an enterprise language uh it was intended to create such big projects but language itself didn't offer any advanced tools to fight complexity result uh software patterns um patterns of software uh architecture dreaded patterns would break havoc to our industry and stack traces with hundreds of layers of abstraction that's not acceptable and not maintainable.

Speaker 3 [38:26]

Yeah, thank you. Okay. I think we were running a bit over folks, but let's try maybe do two more questions So please vote which ones you? Want to hear the most there's a lot of questions. So maybe folks can also come and ask you max at the end. I'm The next highest voted one is does it mean the programming is hard and you need to learn a lot to be a programmer?

Speaker 1 [38:49]

For me, programming is my life. I like my life. That's why I cannot see the complexity of my life. It's fun. For me, programming is fun. That's why it's all for me easy. try to receive fun in your work and probably it can help you to decrease complexity probably i don't know

Speaker 3 [39:34]

Okay, we'll take one more. Gregory, do you like the KISS principle, keep it simple, stupid?

Speaker 2 [39:42]

Oh, yes. It's one of the really amazing software development principles. As a hobby neurophysiologist, I can say that our brain, our prefrontal cortex, it's one ultimate abstraction tool. It keeps millions of ideas, the tree of ideas called cognitome. And it's natural for our brain to abstract things. And when we write our code, we automatically try to abstract, to introduce abstraction, to make things general, to generalize them. But that adds complexity that aligns our code with our brain. But different software developers, we all have different brains, different cognitomes. So making things complex without a need makes code not so good. and it's a really good rule of thumb to constrain ourselves to keep things simple and to add complexity only if it's required if we see that our project part of our project is really big and we need to add complexity to make it maintainable. So this complexity addition, let it be our choice, not our gut reaction.

Speaker 3 [41:26]

very well said well thank you both so much I personally have to say I feel very seen by your talk and I'm sure many people also are feeling the same way very positive vibes and and everyone can you please join me in a round of applause for Maxim and Gregory