IoT using Python on Linux: Lessons Learned

Speaker 1 [00:04]

Welcome everybody. I'm glad to see you here. We are going to talk about doing IoT development using Python. First, let me introduce myself. I'm Matthias Schmidt. I'm in the industry for quite some time, starting off as a C and Assembler code monkey, doing a lot of device drivers and stuff, low-level programming. I was with Sun Microsystems for some 12 years and I'm interested in programming languages and compilers and stuff like that. Low level stuff beyond any reason. I'm with the company Diva Ace in some 5 years.

Speaker 2 [00:54]

Okay, now I'll introduce myself. My name is Thomas, and I'm a rather young developer. I started development in 2007 with the LAMP stack, and since then I've moved on over to C Sharp and Java and others, as well as Python, of course. In fact, it was one of the first things I learned when I started my career in 2013 in the SysOp industry. and with Dewey since 2015. And now I want to go into what's covered and what isn't by this talk. We really don't want to talk about Python all too much, really, but everything that surrounds it. So we won't talk about the code we used, but we really want to talk about the architecture we chose, why we chose Python 3 in an embedded situation, how that worked out, and a little bit about the project.

Speaker 1 [02:03]

Here we go. So first and foremost, I want to give you some sort of an overview about the system we're talking about, the whole system, which is a gateway, which you see here in the middle of it, a gateway passing events, passing sensor events between a radio network into Java-based back-end systems called the cloud, yeah? Because of the headless nature of this gateway, we had to use a mobile software for Android and iOS to kind of kick off things and do the initial setup and do the reconfiguration and also passing on alarms from the sensor network to the mobile phones. So this is the overall picture of the system. passing events back and forth between proprietary network and sensor stuff into the cloud software and back again. And now let's talk about the gateway itself. We will stick with the gateway since this is the IoT part of it. We're dealing with some sort of a Raspberry Pi kind of similar uh device here arm v7 uh linux based system so not a traditional microcontroller no no no arduino something like that or admiral or something um but it's it's in terms of programming it's a it's a full-fledged linux um with a lot of communication facilities on board with many things going on in parallel, custom hardware, which is the receiver you can see here, real-time clocks, communication, LAN, Bluetooth chips, and stuff like that. On top of it, Debian Linux. For communication purposes, we went the Debus way, and the application itself is written in Python 3, running in process of Python 3 and this is the the app portion here so this is the the embedded embedded system we we used as a gateway and now let's take a look at the architecture we've chosen it's a kind of a traditional one a traditional star like architecture for the software where we have more than four modules here trying to do something trying to do some only one thing and do this right so we have a module for for accessing the partner for accessing LEDs for accessing the the radio network accessing the the the internet and this star-like architecture for this for this device brought a lot of benefits one one is that you can plug out one individual module supply it with some sort of testing context here and test it individually so we can go down our network decoders protocol decoders and and and security stuff and and test each individual on its own um which yeah worked out quite well and um this is yeah um shown here by by mock you can mock pretty much every module and um you're good to go and on the other hand uh since we are um highly uh dependent on hardware and a special special hardware devices like a a push-button LED simulator stuff. We also exchange for running everything in the CI environments. We exchange certain modules because they rely on hardware. To give you an example, in the CI environment, the LED module simply doesn't do any GPIO programming, but prints out, I switched on the green LED. to give you an example so we could run pretty much all of the code in in the CI environment

Speaker 2 [06:40]

Next, actually before I introduce you on why we chose Python 3, I want to tell you which values Matthias and I shared during development, which should help you to understand why we did things the way we did them. We really believe in don't repeat yourself, dry. It's a really good pattern, no duplication if possible. So we really believe in the right tool for the job, so we won't just jump on a new tool because it's new and cool. And small is beautiful, really. So we are talking about a device that's in service for around 10 years, and we want to be able that someone else can maintain the product. And for this, you really need to cut down on the things you do and you don't have to go into abstraction heaven. And then we coined something which we call the Feng Shui software development, which pretty much just means that if you are on the correct abstraction layer, at least that's what we believe, and you have all those puzzle pieces flying about, you know where they go. It will just feel right, and it will be an indicator that something's wrong when it doesn't feel right anymore. okay so now over to why we chose python 3 exactly so first of all let's look at what was a given thing for us when we started development on the device um we knew that we have an arm v7 running debian linux um we knew that we we were relatively low powered and now don't get me wrong i think if someone uh here in the audience is into embedded he will he will internally shout at me you have got so many resources and just don't know it and yeah you're right but we're talking about a raspberry pi a class device um we're really not doing much we're just a collector and an emitter we collect stuff from the radio network and we send it out to the internet so we're relatively dumb for a smart device um there's lots of parallel things going on so for example you can talk to the device via ble You can see the LEDs and which state it is and stuff like that. And one of the things we also knew that we had to do, there is this receiver part which talks as a radio network and it's fully proprietary even to us. So we just had a C library and they said, okay, this is a C library, go with that. And with that we knew that we had to choose something that understands C or that can interface with C rather well. So what are the options? The options are C and C++, of course. I think that's a given when you're in embedded. It's one of the most common options. Then Go was an option for us. We wanted to seek out for new languages, safer languages. Mattia is a big fan of Go, and I always wanted to learn it, so it was an option too. Rust is something we both liked, or still like, but back then it was still not ready for Prime, so we really didn't use it. And then when we looked around, we stumbled upon Python. We both knew it. And, well, it has a very long-term support through the Debian community. They don't mind patching it five-plus years. It is quickly to write and easy to read, even for people who are new to python and who have previous programming experience in other languages what's really good about python is that it's batteries included so you can keep your external dependencies way down because most things are already in the standard library and of course there's a simple way to interface with c through the use of cffi and you don't have to do manual memory management which is a huge plus for me as well and another thing we didn't really need a high performance application because as i said we we just are collector and emitter this doesn't have to go in the microsecond sort of range if it takes a second to transmit an event that's okay with us it worked out rather well let's now talk about one of the central components we've used um and it's called dbus which stands for desktop bus it's an asynchronous message bus which is normally used for desktop apps in the gnome environment so if you want to see what uh what dbus does on a gnome system just boot one up kill all dbus and then you'll see a black screen um that's pretty much how important that is the communication with dbus is standardized there are various implementations of the standard um but we just use the standard one the reference implementation, so to say, started in 2002 and was stable in 2006. And here you can see in the picture that we had Blues, which we used for Bluetooth, and we had Network Manager, which we used for networking, and our app connected to the bus, and they talked to each other, and it was rather nice. Now, I promised you some problems in the description for this talk, so I'll give you some. There were really no problems. with DBus itself. So DBus just worked fine at a very low footprint. To my amazement, it is even fast. Yeah, and it had standard Python bindings, but they're really not well-suited for using nowadays. They are known to be broken beyond repair even. And yeah, they're deprecated. They tell you don't use them, but we did as everyone else seems to do on the internet. and it was not the best idea we had. So the lessons we've learned from that was switch to something that works, like PyDBus, which is based on GDBus, so the GNOME implementation of DBus. And I think you can trust them a bit on this because they use it so heavily themselves. Applications that use something like DBus really need to be designed around it because what will happen is that you need to run a main loop, which they provide, and it will devour your entire application. And you will only run when the main loop idles, which is also a plus because events then become first-class citizens in your system. There is a DBus monitor you can use to monitor the DBus and monitor messages, check out timings, maybe get used to working with these tools before you have a problem. It makes things easier, I can tell you. And one other thing we really learned is that GDB is a really cool tool for debugging Python and native parts, especially when they're working together because you can just view them and you can browse through dumps that you've created. It's rather nice. So check it out if you haven't. and also do this maybe before you've got the problems might help.

Speaker 1 [13:57]

Okay, now I'd like to talk a little bit about Bluezee. I don't know how to spell it correctly. This is the standard way of doing Bluetooth and Bluetooth Low Energy communication in Linux. So it's some sort of a given. It's around since nearly 20 years or so. It's made up of some kernel facilities, some kernel modules, and a userland daemon. doing the communication to the hardware devices you can either use it in an in a synchronous fashion doing a linkage against the C library and it has as well supported a D bus plug so you you you basically most of the people use it in a D bus asynchronous fashion so and we had to use Bluetooth since we are we had to come up with a mobile software uh doing initial setup and and and reconfiguration of of the device um and um it it turned out to be quite a rough ride um bluetooth itself basic bluetooth is stable uh even in under linux since since many many years so connecting your your your wireless keyboard or your wireless mouse works works quite good but um with bluetooth low energy we really did some sort of the the bleeding edge here and uh our experience was um if you um if you go with this bleeding edge technology um as probably always go with the latest version of uh of your your tool chain of your libraries of your your blues stack so things got better in the long run but when we started off it was quite a mess we had cordons we had hang-ups we had this and that pretty much everything but on the other hand i i was really i was really amazed about the help we've got from the community here. Another thing we were facing, we have been quite naive in the first place the Bluetooth low energy is not always Bluetooth low energy so connecting thousands of different Android devices to your Bluetooth low energy turned out to be quite tricky and on the Android side we had to introduce many ifs and then and and doing special handling of certain devices so uh i mean especially on android um you you have a lot of uh testing here um but as i said uh things got better now i would consider um um consider the um the the blue stack and the bluetooth low energy part of it as a stable thing but we learned the hard way um if if the maintainers take something as experimental they are not kidding um so um but on the other hand as i said before um people have been really tremendously helpful on on the mailing list the card car commuters fixed uh um many stuff nearly instantly and We could circumvent a lot of stuff in in the the Android and iOS mobile devices here, so But be prepared if you do due to slow energy with with Linux really Take into account you you have to do a lot of testing manual testing most of the time since all Android devices behave differently when when it comes to Bluetooth this is because different chipsets different versions of androids different microcode loaded into the chipsets and this and that you cannot imagine so you cannot rely on something like a spec I programmed against the spec and this all to people this this is this kind of work no it doesn't

Speaker 2 [18:21]

So let's now talk to the other participant on the bus, which is Network Manager, a rather unconventional technology to be used with an embedded device. Now, I have a question for you all. Who is using a Linux-based OS on their notebook? Who of you is using, well, most of the room for the video. Who is using Wi-Fi on their notebooks? Also most of the room. And who is using Network Manager on their laptops? a couple of people at least um now the thing is that network manager is regarded as this desktop solution and this laptop solution and you cannot use it headless or something like that it's actually not true consumes very little resources um it abstracts all your network configuration into so-called connections and it will then just cater uh to whatever device is best fit to fulfill this connection so it makes things really easy there's a dbus and a there's a dbus connection and a c library um it can emit status change events whenever something changes on the network which makes uh working uh with uh the network quite easy because uh i think um you might think to yourselves now well you can just do if config uh the interface and then add an ip address and that's about it and i can tell you know that's pretty much not about it um you really have to monitor the device you have to peek and poke proc that's what i called it uh you have to do so many things and they are so hard to get right um and we we've tried to really because we wanted to keep the dependencies down uh but it wasn't really worth it it's network manager and others really support all the things so what were the problems well we didn't really have problems with the network manager now i'm not going to say that network manager doesn't have any problems but we weren't facing any uh but what we did face were strange wi-fi hardware issues now to the guys who use uh linux on their notebook have you ever had issues with wi-fi a couple of heads nodding um and i can tell you that most of the time it's either something in between the network manager and the device or it's uh just the driver itself or even the kernel really not uh not so much fun of handling those errors uh and that's only so much you can do so that's pretty much it and the lessons learned were really don't roll your own network management code. There are people who have done this for decades and they do it a lot better than you can do it, off the cuff at least. Network manager is small enough to run in a headless situation and stuff like that. And brittle drivers can, of course, ruin your day when you really just try to get things done. Let's now switch gears and talk about a general issue. This slide is called RTFM, which, as you know, is the politest way. A programmer can ask you to read the documentation before you span the IRC channel. And here I've pulled up the timer objects documentation. As you can see, it's quite simple. It's just define a method, then you start it, and presto, right? Well, most of the times that's actually true. but uh we have faced something uh rather unfortunate um now we already mentioned that we connect using bluetooth low energy and we don't want to keep the connection open all the time so what's the easiest solution to uh get rid of the connection well you just set a timer 10 minutes from now and uh yeah um close the connection um so far so good this worked out quite well until one day when another developer on the team gave me a log file and said, yeah, Bluetooth turned on midst configuration. And I was like, what? I turned off, yeah, I turned off midst configuration. And I was like, what? How can this happen? He sent me the logs, everything looked normal. And then he showed it to me. And yes, midst configuration. So when we weren't really done, but when we had connected to the internet, we were turning off Bluetooth. well turns out that uh well we have an ntp running and in the first version of the prototype board our rtc was faulty so we pretty much jumped days ahead which resetted the timers because they are not monotonic in python at least on linux um so yeah so the timers just did latch and they did what they were supposed to do, just not what we expected of them. The root cause for this really is a missing parameter, as far as I've understood this, for glibc and the kernel. This bug has been open since 2012, by the way, and not so much work has gone into it. And now you might ask yourself, okay, so now that we know it's broken and what's broken, what can we do? Well, first of all, you can buy a working RTC. That's one solution, and a certainly good one. You can also try to roll your own kernel, roll your own glipsy, roll your own Python 3, which may be fun, but in a time-critical situation, it's probably not the thing you want to do. So what else can we do to mitigate this? We can do a workaround in Python, actually. So what we've done is we've made a sliced timer, so to say. So if you had a timer for 10 minutes, you can imagine it like so. Then you just had slices, one minute each. And if the time jumped ahead, you would only lose a minute at max, which could be unfortunate if you're in the last minute, but wouldn't be so bad if you're in the other parts of the minutes. But it's pretty much a dirty hack, so don't tell anyone that we've did this. But it works fine. The lessons we've learned here is that documentation is one thing and can be beautifully written, but really you need to look down into the implementation to see what's really going on. And even if you're not trying to do C or try to circumvent it some way, You need to be able to look into these things so you can find out what's actually wrong. And it's crucial to a project like this to be able to be doing this.

Speaker 1 [25:21]

Yeah, another thing we've experienced, but something you might be aware of already, is this. Although we both really love Python and love working in the REPL and everything about Python, this highly dynamic nature of Python comes with some sort of a cost. And here's an example which really happened in the field in our project. We introduced, we changed the signature of a method. We introduced a parameter in between, which is not a wise idea anyway. But nevertheless, we had to introduce this parameter here. And one of the callers, we simply forgot to update the caller. And this is what happens if you don't go with typeins or stuff like that. um as you can see if you call it uh in in the traditional way uh python rolls out um the table of the list whatever into your your your private key variable here and no front-end compiler nothing no runtime will stop you from doing this so you're ending up and having some sort of a string or whatever as your RSE private key have I really have fun doing the debugging online here and then trying to figure this out this is this happens so easily and this is I call it this the cost of doing dynamic languages but yeah luckily there's there's something you can do about it simply use type hints I I cannot stress this point more, do, even for the simplest methods, guard yourself against these kinds of problems by doing type hints in the first place. And you should check this using your CI systems. So you sort of combine the quick and easy style of doing Python coding with checks and balances on the back end and the CI systems. Just as an example, what we discovered on the way, and this was the only downside in our decision to use Python for this embedded device, but honestly a downside you can work around. okay so um to wrap things up um after this project what what are the the the big takeaways here what uh what our our lessons learned um really tried to do um um ci and testing from day one i mean we are all lazy and we say okay i'll do the testing i'll do the ci integration next week and next week and next week and at the end of the day um um you you're running out of time you never do it do yourself a favor do it on the very first day spend spend some time up front to um to get your tooling right to get your um your your whole tool chain up and running and and get yourself acquainted with those kinds of things even if it's um it it turns out to be a hard fight against some managers and trying to educate them that this is really necessary this really pays off this pays off in the long run and um you're you're good to go so um can you do test first sure you can i'm not not religious about testing here but especially if you go if you run with a dynamic programming language like Python, you should equip yourself with lots of tests. In our case, protocol decoders, cryptography stuff. So we did tests first, not for all modules, but many of them. And this also really pays off. Code coverage. Introduce code coverage tools and again here, don't be religious about it. Don't shoot for having 100% code coverage. In our case, for example, doing code coverage on a module which turns on LEDs or so is completely pointless. But when it comes to your own coding, your own kind of algorithmic stuff, try to keep up good code coverage and cover everything with tests. I know I might be...

Speaker 2 [30:18]

destroyed.

Speaker 1 [30:19]

stress this a bit too much but it really pays off tie pins linting all those tools set them up upfront in your CI system and and really equip yourself and the last thing which really paid off was try to to have all the the different phases in your build run, make it enable to call them up individually. So we used a traditional make file and we have been able to kick in various places. This also makes your CI integration quite easy. If people complain and say, okay, I have a lot of trouble, I cannot put this into a CI system, usually this is just an indicator that your build system is too complex or too many individual steps or too many manual steps are involved. So if you prepare yourself with a good makefile or whatever tool you want to use here, And if you separate individual steps and phases and you can start them individually, it's a no brainer to plug in the CI system of your choice. So these are probably kind of boring. not hacker-like recommendations, but they pay off.

Speaker 2 [32:03]

I also have a couple of things for the wrap-up. As we said in the beginning, and Matthias told you just now, tooling is important. It needs to work, and broken tooling is a big no-no. Trust me, if a project gets into a bit of a stressy situation, time pressure and everything, you'll just let it slide. You'll do what you shouldn't do. And in the end, this just makes it less productive and makes it less likely to get things done in a timely manner. Then what we also learned and what I really appreciated about this project, try to figure out what is the most important thing about what you're doing here and what are the most important concepts of your design. For us, this was communication and security. and putting them at the core of everything we did made it A, fun to work with the system because things just fell where they needed to fall, and also it made talking about things really easy because you really didn't need to think about it. When someone asked you about security, you just knew what the system was and what you got it against, and it made also some questions just evaporate in thin air because they were already answered by our software. You can try to introduce security late in your code, but let me put it this way, it won't be a smooth ride. And then the last point, which was very important for me, is that really we're engineers, and there's a bit more to engineering than there is to programming. And in the end, what we do here, it's just software, it's just technology, and given the time you will most likely figure out even the weirdest issue or you'll find a way to circumvent it but what you what you will not find is a way to regain trust if you if you lost it to your customer if you lost it to your teammates so in the opening today it was mentioned that we need to be a nice community and this is not also true for open source which was very nice to us, but also very true in the workplace and for your teammates. So always keep this in mind. It's mentioned way too less, in my opinion. Lastly, we do have a quote here that really basically illustrates this.

Speaker 1 [34:49]

That's about it.

Speaker 2 [34:58]

Now, are there any questions? Yeah?

Speaker 1 [35:18]

The question was how debugging in my day-to-day works. The answer is single-step it in GDB, try to get debug symbols for your Python and for all the C modules you have, and bite the bullet. To be honest, this is nothing I do on a day-to-day base, since if you prepare yourself up front with testing and that, you normally don't end up in GDP each and every day. But if it comes down to really nasty bugs, you mind doing that way. And then you can use any front-end to GDP you want to use or you do like me, the traditional command line version.

Speaker 2 [36:26]

And the setup was actually always, I don't know, were you talking about the manual testing? Yeah. Okay, so I think the question was that if we ever thought to automate the manual testing, right? Yes, we did. And well, in our setup, we actually accounted for all of these things so that we can really do this. But in the end, we never got around to it because the test setup, getting this all working, is a bit complex. So in the end, we just skimmed on that and just did the manual testing in a coordinated fashion using TestRail.

Speaker 1 [37:12]

When it comes to testing, in other areas, we really did some automatic testing with external devices. We had, for the sensor network and this proprietary radio protocol, we had emitters which we could program in Python as well. And so we create some sort of a feedback loop, issuing some sort of radio commands and catching them in our software and see whether the software reacts as it should. So to my belief, really try to do as much as possible in an automatic fashion. But if you have a matrix of, in our case, more than 30 or 40 Android devices, you really have to do it on your own. Okay. I have a question regarding the typing part you mentioned. Okay. The type hinting solves the problem in your case, but it sort of only solves it coincidentally, right? because if the two parameters have to have the same type, then you won't have it at that point. You're right! Why didn't you use a keyword-only argument with your Python 3? Yes, you're right.

Speaker 2 [38:34]

So for the video, the question was, why didn't we use keyword-based arguments instead of the type hints? Because they would only solve it coincidentally. Because if we had the same types, it wouldn't help anything. Yeah, the answer was, yeah, it would have been a solution. We used it in a couple of places, but not everywhere. And in my experience, getting quirks with keyword arguments can also be quite troublesome. them and at least you'll get an error when you miss out on a parameter for the if you just list them out but there were some some places where you don't get an error if you use keyword arguments

Speaker 1 [39:29]

Yes, the question was whether we use mocking for several devices. Yes, for example, we have a hardware button attached to the device which turns on or off a Bluetooth low energy. And the mocked version is basically an identical module which is not connected to the actual hardware, but which you can issue a unique signal against. And a certain signal means pushing the button or pushing it long or something like that. And these were the kind of tricks we used to do mocking. So it's not this sort of mocking in a traditional way where you're facing a big framework and you try to get things done and you have to mock all the kind of logical entities of your code. We did more of mocking away our hardware to keep on running on CI systems and to simulate certain behavior.

Speaker 2 [40:54]

So that's a good question. The question was, why didn't we look into, or if we looked into spawning an access point instead of doing Bluetooth Low Energy for configuring the device? And the answer to this is clearly yes, but we did this a little bit late in the project. We were really only trying to circumvent the BLE problems we were facing. and well, it then turned out that we could just circumvent the BLE problem so we went with that but yeah, it was definitely one of the things we had in mind and it would have been the thing that we would have done if BLE wouldn't have come together that nicely that it now does.

Speaker 1 [41:50]

Coverage the standard coverage to high coverage Yeah, yeah, it's good enough. I mean there's probably some more fancy stuff around but Coverage did fine for us Last question Okay, thank you guys