AsyncApp. My contribution to hype Pythons asyncio a bit more

Speaker 1 [00:06]

Thank you very much for having me. This is my second talk on this conference this year, quite exciting. If you want to follow along the presentation, there's the QR code, simply scan it and you will directly land at these pages, they live on my GitHub repository in the GitHub pages branch. And they are doing that, that's great. All the effort was worth it. So why are we here? I would like to talk about asyncio a bit. My impression was asyncio is there for quite a long time now, since Python version 3.4, and I still don't see it that often in beginners' projects. And there must be a reason for that, I think. I think the way asyncio works and the idea behind it came into Python in a very specific in a very specific time, and it wasn't very obvious. It's much related in the way how concurrency works, how parallel computing works, and there is this threading idea in Python, and there is the global interpreter log, and hardly anyone really believes in doing concurrency and parallelism in Python. So maybe asyncio wasn't a thing everywhere. It's probably available in all web frameworks and it has its advantages there, of course, but when you really start off your new application doing whatever thing there is, you probably won't start using asyncIO from the very beginning. My aim is to possibly put or shed some light on that and find a way how to get you into AsyncIO using that from the very beginning and give a different perspective on how these things work behind the scenes and maybe motivate things. So what's the motivation behind it? That's the personal talk. It's nothing related with the work I do for the company I'm involved with. And I personally, I am very much into Internet of Things-like applications. these simple things that you often see, these temperature and humidity sensors that you have maybe lying around that show you the data for each and every hour, every second of your life and broadcasting this somewhere and then you have a graph somewhere and you store all the data, can go back in time, make your analysis. It's a pretty exciting thing and I'm always into, I want to understand how things work behind the scenes, how they implemented it what does it take really to make these applications and for that the temperature thing it was really temperature sensors I distributed around my house just to see in all the rooms what happens over the course of a year or pool temperature in the summer when will be the first day to get safely into the pool and power consumption is a thing and the yield of my balcony power plant that thing that we have in Germany here like Balkonkraftwerk and by the way there is another talk by Yannis he's sitting there in the row in a couple of hours I guess dealing just with that thing and it might be very exciting to go there as well and I also like to recreate things that already exist just to get an idea how these exist like a telescope control or creating time lapse videos or simply remote controlling toy cars. So how do these things look like? That's a dashboard, so to speak, for my temperature sensors, even with some test and dummy implementations in it. And this, by the way, works by having an ESP32 microcontroller running MicroPython, and it feeds all the data to a phased API web service, which delivers things, stores things somewhere else and the display is done using PyScript so that web assembly based implementation for Python, so it runs in your web browser and does everything for you, and that's a quite exciting topic in itself that tracking of the power usage for my power meter is a thing, and that's a very simple implementation based on a Raspberry Pi and simply an infrared photodiode and a resistor that you connect to the serial port of your Raspberry Pi because what the power meter delivers is simply a serial interface in an optical way. And that already is enough to decode data that power meter delivers from house and that you can then interpret, store and enjoy. the balcony power plant pretty much the same way it's a micro inverter that delivers the data broadcasts these things with with some some sending and receiving device that you can attach to a Raspberry Pi pretty much the same architecture and all these things have been done over the course of a couple of years and I tended to always reinvent the wheel because I started off doing these things playing around with the versions of the libraries that were around at these times and it was always a proper way to do these things following all these available tutorials in the internet and if you go for that very same project the next year things might have been changed different things might be hype now and you restart the thing again so I did quite a few of these implementations and there's this time-lapse project that I'm doing basically the Raspberry Pi camera attached to a Raspberry Pi with an awesome camera streamer application that lets you let's the Pi camera act to be just like a normal webcam with these high quality streams that you get and a and a snapshot URL where you can retrieve the photos, attach different things to it to process data afterwards. And what I'm doing is simply one hour before sunrise to one hour after sunset, every 30 seconds, grab a photo, make a time-lapse video of it, and by simply feeding it into the FFmpeg process, that creates the video already, and that's what I'm enjoying right before I go to bed in the evening to see what happens when me and my wife worked hard all the day and had no notice of what is going on outside. And the telescope control, that's simply a stepper motor attached to one of the axes, waiting for commands it receives via Redis, and these commands will be produced by another Raspberry Pi attached to that and that gamepad. And that's the communication way of things. And toy cars, pretty much the same thing, four mortals attached to a Raspberry Pi. That very same gaypad can be used to get you directions, speed and everything. And as you can see, these are very, very comparable. It's always the same thing. There's some input device, some sensor delivering data. There's some instant that it has to distribute the commands or the data somewhere else. And there is a sink somewhere, might be a storage system, might be a display, might be a motor acting on these things. They pretty much all can be based on a very simple, very effective implementation, and I was at the point where the next project was interesting for me, was in the row, and I remember that I had all these implementations in the years before, and it was very difficult to maintain them, to really understand them. If there's code that runs every day for two years, you don't touch it, then you read it again and say, what the heck is going on behind the scenes there? So I was up for finding a way to rebase all these things, to do a refactoring and put it all on the same base. And that's what we're talking about today. So the plan was made to streamline all these implementations, find suitable base for all of them to be the common one, and integrate monitoring and configuration concepts in it. And I wanted to follow the traditional Unix philosophy behind it. It was very important for me that you can read down by Malcolm Douglas McElroy, the inventor of the Unix pipes, who said, this is the Unix philosophy. Write programs that do one thing and do it well. Write programs to work together, write programs to handle text streams because that is a universal interface. I said it quite a long time ago, but it's very, very true today. And in the end, this is probably the thing to really work on, to find that minimum viable application, that minimum viable product that just does that thing that you want it to do and make it as atomic as possible and maybe do many of these applications that interact with each other in some standard interfacing way. That could be pipes, absolutely possible today, or Redis, for example, and using the PubSub pattern behind it to simply publish some JSON-encoded records and have on the other side the subscribers who consume this and buy on these things act and do the commands. That was the way that I wanted to follow. Use Redis as my central communication instance, use systemd on the Linux system, mainly on the Raspberry Pis to aggregate starting all the applications that I needed, and let Linux decide what happens in an error case. So I quite often do not need to handle errors that occur in the applications because they simply then stop and fail and have systemd restarting this and everything is fine. So that's a quite nice thing to handle errors if you're not in production. And I wanted to use asyncio because, you know, why not? It's quite simple interfaces if you just have that one application and you do not plan to have those many concurrent algorithms running in one application and you can stay within that one process, one thread and solve concurrency things in a different way. So for that to understand that maybe now the new view on things is to get this not too short introduction into asyncio. So asyncio IO is one of the three implementations for parallel and, in this specific case, for concurrent processing in Python's standard library since 3.4. It can execute many things in a single thread and single process. That is what makes it specific and makes it differ from real threading and multiprocessing. For that to work, the workload has to be in a specific category, like IO-related. It should involve a lot of waiting on things and must be interruptible. That's the key thing. Whereas multiprocessing threading can deal with classic Python code, iSyncIO needs a suitable implementation for all the core functionality that's ready to be interruptible. When you have a closer look on these things and the documentation in Python's standard library, you see exactly these things. It's like all these sections covering all the functionality of Python, and there is is a section for asyncio with everything repeated for that in the asyncio fashion, the asyncio way. I don't like this approach of the Python documentation. I think it would be nowadays possibly better to split this into the synchronous Python thing and the asynchronous Python thing because it's pretty much all resembled. There's one thing for time sleep and asyncio sleep that's for sub-process handling. For every resource handling, there's a synchronized standard blocking version and there's an asynchronous version. So I think the major step is towards using asyncio to get into the idea of a main loop executing things. If you ever were to implement graphical user implementations, that idea isn't new. It's your graphical user interface popping up, waiting all the time for you to interact, and you decide by quitting it that now things are done. And it's quite different for many Python applications just following to execute a specific task. Once that's done, Python is going down, that's it. That is the linearity in the scripts that we implement. And with asyncio, you can resemble it, but you have to have the idea and the background that there is a main loop doing these things, and you have to ask the main loop to quit. It's not that your last action that you really implemented will do this. You have to do that step for you. So let's have a look. The first things that are necessary to understand for async.io is this concept of functions and coroutines. And if we look at it, what we need is we need to import async.io. And I would like to demo some workload by simply waiting to a time sleep. That's why we need the time module. And there is this function blocking wait for that just sleeps for some certain time and tells you that it's now done with it and returns that specific time that it waited for. And you can know that this is a function because there's simply the def keyword and that's it. That is what makes this thing in Python a function. And executing it, most obviously, just gives the expected results. And there's a coroutine version that you can create for it by defining this as an async def. Using async def, you introduce that thing that, well, that is the special thing that is this async able, this interruptible version of what we're doing, and you have to handle that in a special case. So now we are awaiting an async IO sleep, which actually does the same thing, but lets the system, lets the Python interpreter from time to time interrupt just waiting that, looking for other work, and if there's things to do, simply doing these and come back to that function. So then you can create that coroutine by simply calling that friendly wait for, it's not executing this thing now. You have to await that and that really tells the Python interpreter to now do that thing and execute that code and then the same thing happens. And then there is another layer on top of that that gives you some wrapping to let it look like, let it look, let it be seen in a different interface that creates tasks and these tasks can be managed from the outside and they can be queried from the outside. You can ask if these actions that you programmed are now possibly done and what the result of it was and you can cancel these things. But it's just another layer on top of that async functionality where you can use different ways to manage the functionality behind it. You are creating these tasks using async create task and at the specific time you're doing this that coroutine is instantiated and executed. Now your action is going to happen. Create the task and you're good to go. If you have a long running task then from the outside as you just gain control at that very specific section of your code you can ask for the state and you can cancel these things and thereby manage what's happening in that in that section of your code. And if you have classic implementations that are blocking then you can from the asyncio world just push them to another thread and that gives you the same interface even though it's a completely different implementation but you have the same control on these things you can query these things but you might not be able to cancel them because if your function is not prepared for to be cancelled then there's no way in doing so might end up in some zombie process or something that is going around and finally to run things concurrently you simply you create these tasks and you group awaiting by gathering the results for example that's So that's a way to do this. You have a lot of tasks that should be done concurrently, and you simply wait for all the results to happen. And if you are interested to have the things in order of execution, so you have some tasks that are finished quicker than others, then you can use asung.io as completed for that. And that gives you the things in that execution order. So, exception handling is a thing as well, what happens if I do not have that specific control in that specific line of code, but it works pretty much the same way, as soon as you have created your task as the coroutine runs, then you can simply catch your exception and handle it, and due to the task interface, even afterwards, query what was the reason and handle upon that. There was a very, very quick overview on the asyncio functionality. If you were into deep diving a bit more, I would really recommend to read the complete series by Jason Brownlee for Python asyncio, the complete guide. It's awesome material and very, very good to follow. So what I did in the asyncio framework, I wanted to be following Alan Case. he says that simple things should be simple and complex things should be possible and I had a to-do list for my application execution that I wanted to run functionality concurrently I wanted to have long running jobs I wanted to have periodically running jobs like reading temperatures every minute every few seconds put it somewhere and I wanted to have event listeners like listening for that gamepad for events and handling all these things. And you can imagine that thing is a blocking thing. So you need to find ways how you can live with that in an async world. And regarding execution tracing, I wanted to be configurable for both. I wanted to have logging in it. And when I start developing things, I want to watch it very, very closely. And I have the idea that now it will be okay in everyday work. I want to silence things. And, regarding configuration, I wanted to be able to simply write my code down and execute it, but I also wanted to have command line parameters, like for if I have periodical tasks, I wanted to state on the command line what this periodicity would be, and I wanted also to be able to have configuration files, like tunnel-based files, and possibly also environment variables. Luckily, with our ecosystem, there's quite simple ways to get there. There's a lot of things that are related to these. If you follow the slides, there are entry points for all of these technologies, and you will find simple examples in it to get your first ideas, not for today, not for this talk. As time is really, really almost over, I would like to skip a few details that you can follow along if you reread these slides or even maybe play with these things. The most important thing for me was that I wanted to have a simple interface for the Async Promise app that lets me, for everything that I wanted to do concurrently, create the same interface. and all I do is creating a dictionary with a task description and the task description says I have a simple or a continuously running task where things in a blocking way or something interruptible will forever run and there might as well be things to do periodically so I have periodical tasks that for maybe every two seconds every 10 seconds every 30 seconds like for the time-lapsing project do things and I wanted to offer some monitoring functionality that I can see how many tasks are running which state they are actually and wanted to be able to exit things there. So this is how the task descriptions look like in that case. It's just the Python dictionary and that async app application lets you add these things. Once you're done configuring all this you simply can await the application to run and the magic happens behind the scenes. And that for this talk is pretty much all that there is. A very brief overview. I hope that you are looking at the slides again and then find all the details that there are in it. And for the rest of it, thank you very much for being here, for having me. And that QR code leads you to the repo with the actual implementation of the async app framework that i hope you will be visiting and finding all the bugs for me that there are in so that we can together work on that very simple interface for internet of things like applications and async io usage so thank you very very much. There must be a lot of questions, I think.

Speaker 2 [23:29]

Thank you very much. We have actually some questions. Yes. So the first question we had was actually to the introduction. Somebody wanted to know more about your balcony power plant and your Python controlled telescope.

Speaker 1 [23:43]

So I'm using a balcony power plan that is in the area of 1.4 kilowatts peak, over-provisioned, and a simple Heumann's micro-inverter, and some sending and receiving interface so that I can query the outcome and write it to some simple file storage and enjoy it by this pie Excuse me.

Speaker 2 [24:14]

Second question was, what is a good practice to allow the interruption of a function that runs in a separate thread?

Speaker 1 [24:22]

If you are able to modify that function think of, that was one of the examples that I skipped over actually, think of a time sleep for a very long time. That's one thing that you can accept that there is, but you can always interrupt that by sleeping just parts of that time in a for loop or in a while loop or something and have this simple await async IO sleep some very very tiny amount of time like a thousandth of a second or something, and that will already help the Python interpreter to find that very slot where it can interrupt and do other things. So if you're able to modify the code also slightly, then that will be the thing to do.

Speaker 2 [25:06]

Okay. Next question is, to what extent are asyncIO tasks different to futures, for example, as in the concurrent standard library?

Speaker 1 [25:18]

It's very, very related, actually. If you look at how things developed from these generators to the coroutines, then there's this concurrent futures interface, and task is something related to all that. It's not very different. Actually, you're dealing with futures, with promises, so to speak, that your coroutines give you. It's very, very much comparable, actually. And you find ways, like when you're using the multiprocessing toolkit for process pools and there are thread pools, they give you an interface to concurrency and threading-based solutions that are almost exchangeable. So that's another topic I had last year, was the recommendation to play with that. because then you can really feel what your algorithms are doing if you're simply switching the engine below it. And that might be very, very interesting for trying things and get the best performance. But here it's not at all about performance. It's just having a few resources and be able to have concurrency there. Raspberry Pi.

Speaker 2 [26:37]

How do you handle the addition and execution of tasks? Are you using a queue here? In parentheses, Redis.

Speaker 1 [26:45]

Yeah, so I have very simple tasks that do not interact in all directions, it's just one way. So I can publish things to Redis and I consume it from Redis and it's all fine. If I would need to have interactions going back that has an influence on these things or even communicate within the application, I would use queues for that and simply push things on the queue and pop things from there up until it's empty to work in it. But Redis is unbelievably fast, so it is a drop-in replacement, simply publish everything or set it there, even with JSON encoding, it just doesn't take any time really. Or use, how it's now called, Valky, the other implementation, the drop-in replacement, all the others, they all work the same, they're all very fast.

Speaker 2 [27:37]

Okay, great. Do you recommend starting all projects directly with asyncIO? Yes. Just the ones that are IO-bound? Yes.

Speaker 1 [27:45]

Yes, do it because there are so many large frameworks that switched over the years and it's hard to

Speaker 2 [27:45]

Yes.

Speaker 1 [27:51]

Get into them because they work so differently and if you start with with this simple idea Like if you also want to follow these things boost your own temperature cells and do these things It's just these concurrent tasks doing simple things and you get the idea how things work And then you will be able to recognize the patterns in other frameworks as well that is a lot simpler that way than if you start with the blocking thing and I find well that's that's 15 lines of Python code script that's doing these things but it's actually blocking but it's fine. The asynchro way might be 10 lines more but it gives you a lot of more insights and how things really work.

Speaker 2 [28:32]

Okay. Have you tried using Curio instead of AsyncIO? If yes, your experience with that?

Speaker 1 [28:39]

with that? All of them. I'm a huge fan of all these libraries and especially the people behind them. These are true Python heroes and try all these things. Curio and Trio, they are absolutely awesome. There are great talks behind them that show you the principles behind asyncio and all the IO-related things like the simple Unix select that is the magic, the real magic behind it. And to recognize this and to find these these things is awesome and NEIO is another wrapper on top of all that where you can simply exchange the engines so that's the entry of the rabbit holes and there are a lot

Speaker 2 [29:21]

Maybe one very quick last question. How do you handle logging across async tasks?

Speaker 1 [29:25]

Logging is single and in my application it is that single thread, single process, there is no issue. And if I want to have many applications in my stack logging these things, then one thing is a central logging instance that you are configuring and send all your logging items to there. Or have simple different files for that and merge them afterwards. We all are data scientists so we find ways to.

Speaker 2 [29:55]

Okay, thank you very much. I think we're already a bit over time, so I know there's some questions, but maybe you can go and ask them afterwards or in the break. And again, put your hands together for Jens.