Driving 3D Printers with Python: Lessons Learned

Speaker 1 [00:04]

Hi, everyone. My name is Gina Heuske. You can also find me on Twitter as either Fusel or with my project account OctoPrint3D. And OctoPrint is actually what I'm going to be talking about today. This is an open source project that I have been maintaining now since almost seven years. And it's basically like a web interface slash remote controlling solution for consumer 3D printers like you can by now buy them even in hardware stores and all that. the question of course arises why would you need something like that for 3d printers in the first place because like with a paper printer you don't need a remote monitoring solution or anything like that and the thing with 3d printers first of all who of you has uh yeah experience with 3d printers in any kind okay quite a number cool okay um so um this was my very first 3d printer that i bought back in November of 2012. And as you can see, it took quite a bit of space up in my home office. And the problem with 3D printers is not only do they take up space, but the way that they work is, imagine a hot glue gun mounted to three stepper motors that are able to drive it in X and Y direction and also in Z direction in order to build the model that is being printed up from molten plastic. so you have stepper motors that constantly make some noise and you also have the smell of molten plastic and considering that this whole thing can take quite a while you will be in this situation quite a while unless you find a way to put that thing somewhere else and monitor it from afar why would you even want to monitor it? well these things aren't as foolproof as some of the ads would like us to believe because if you do not keep an eye on a running print job, it might end up like this. Something goes wrong somewhere during the multi-hour to multi-day print and instead of what you actually wanted to have, you end up with modern art. And this is basically where Octoprint comes in. And this is also why Octoprint was born because I was in the situation that I basically wanted my desk back. So I wanted to be able to put the printer somewhere else, monitor it from afar, ideally with a webcam and via web browser because that way I didn't also have to find some way to put a client application on every single device that I might want to monitor things from. What I ended up with is after figuring out that apparently there was no ready-made solution available already for doing exactly that. I built my own. And this is how it currently looks. You basically have a web interface where you see the current print progress. Currently, there's no print job running. We have a webcam tab where you can see, where you can actually look at the printer and see what is happening. You can monitor the temperature. You can take a look at the model being printed behind the G-code viewer tab. You can see what is actually being sent to the printer and received from the printer on a terminal tab, which is very important when you want to debug certain situations about which I will talk a bit as well. And you also can do time-lapse recordings of your prints. So it makes snapshots via the webcam every 10 seconds or whatever you set it to. And in the end, you basically see your model building up from the build plate, which is not only something fun to look at, but if something goes wrong, you actually can maybe figure out what went wrong by looking at the time-lapse recording afterwards. why am i even talking about this here well octoprint is powered by python so the whole server side of octoprint is developed completely in python there's a ton of stuff going in going on in there about which i do not want to go into too much detail here but we basically have a back end that talks to the printer while the serial interface Most of the consumer 3D printers have a more or less well-defined protocol that is being talked over serial plaintext that allows to plug in various plugins that users can write themselves. Some file system access various internal components. And on the front-end facing side, we have a web server, Tornado in this case, which also using Flask provides a REST API so that the web interface component that runs in the front-end and is completely written in HTML, JavaScript, and CSS can talk to that stuff over a web socket, also get push information, updates on the ongoing print and all that. And yeah, so this is the rough architecture of how everything fits together. the target audience of something like OctoPrint is basically people who own a 3D printer, which roughly translates to makers, tinkerers, and hobbyists, but also some businesses. So there are also architect offices who have 3D printers in order to print models for the buildings that they are currently working on and stuff like that. Overall, you can say that OctoPrint targets end users, snot developers. People who use OctoPrint usually are not developers. They might have some experience, but it's more the exception to the rule than any kind of rule. And with regards to the platform that I'm targeting here, there are years that you run OctoPrint on a single PCB computer like a Raspberry Pi, very cheap, 35 euros. You plug the printer in there, put the webcam in there, install OctoPrint on it, and then you can remotely monitor your printer. I've been doing this since almost seven years now, and over the years, I discovered some challenges in running such an end-user-facing project in general, with Python specifically, and some of these I now want to talk about a bit more. Challenge number one is how do you even get the software to the user? So how do you get the hurdle of the initial installation over? And also, how do you keep the software up to date? Because, of course, you will do bug fixes, you will add new functionality, and maybe there are also security fixes at some point or another that you need to somehow roll out. And yeah, this can be a real challenge. The problem with the initial installation is, well, I mean, usually we would just say, well, yeah, bootstrap Python environment and then do pip install OctoPrint and then everything runs. But this is something that for this target audience of users is too much. So a lot of people already start screaming at the point of bootstrap Python. And then when they hear pip, they completely have no idea anymore what I'm talking about. And I'm like, oh, command line, what now? No. so the solution here was as I'm primarily targeting the Raspberry Pi was basically to provide a pre-configured image with everything on it where all they have to do is edit a single text file to add their Wi-Fi credentials, they flash this image on an SD card, put the SD card into the Raspberry Pi boot it, access it and then a wizard will do the rest for them basically so this worked great but yeah what about installing on something that is not a raspberry pi a problem here is yeah this is something that i that i don't focus as much on but it's something that i also always have to keep in mind usually i approach it from that direction that if someone wants to tread the not so well trodden path in that regard they are left more or less on their own but i still do not want to build software that is not able to run on all the platforms that usually get used for something like that and there can be some surprises for example when you try to do ipv6 dual stack stuff under windows where you suddenly have to hack around with some stuff so this is also something to keep in mind regarding keeping things up to date again the usual approach would be pip install that you octoprint that is again the issue with the command line and a linux shell maybe and something that a lot of the people in the target audience simply cannot cope with. So I had to build in a whole software update mechanism into OctoPrint that automatically will detect when there's a new release published, will offer to install it, and when the user says, yes, please install it now, it will do basically pip install-u OctoPrint and some other stuff, and also display a changelog, which, by the way, is also a good idea, because then you can hope that people will read it and at least put some heads up regarding any breaking issues or something like that in there. There, of course, is also the problem with the rest of the environment. I mean, I'm distributing a whole image with a ton of other stuff on there. How do I keep this up to date? I'm not doing this from within OctoPrint. Some users do it on their own, some don't. And there's also this problem I mentioned that I update OctoPrint through PIP. For debugging purposes, I also print the output of pip in a little update window. And when users see this message, which is probably something that a lot of you have already seen during your development life with Python as well, they get a bit confused and sometimes a bit panicky because OctoPrint is now telling them to update something they don't know about and is telling them to run some commands they don't know where to even enter. And this causes a lot of support overhead as well. So long story short, the rest of the environment can be in a very unknown state, sometimes partially broken because some components were updated by the user, some components were not updated by the user. Maybe there is some mismatch situation going on. And yeah, this results in situations that are very hard to test for because I have to anticipate a lot of possible combinations. And this is also something that causes a ton of additional support overhead. Ideally, it would be great if users would just update the image itself if a new version is released, but most of them have heard this never touch a running system thing and adhere strictly to it. And that has the result of the software stack that I'm developing against because OctoPrint keeps updating itself on ancient images, outdates very quickly with setup tools versions, which cause issues with third-party dependencies of third-party dependencies of third-party dependencies. And that can get really funny in the field. The current approach that I'm using to solve this issue is something that I'm still evaluating, but it appears to work, is to basically have OctoPrint detect if the environment it's running on is fairly oldish. And if so, say, by the way, please update your image. I refuse to update on this version because it's simply too old and I do not know what will explode when I do. Most people are not entirely happy with this situation, but I've at least offered them a way to quickly back up stuff and then restore. And so it is fairly fast now. So long story short, lessons learned from this is try to control the runtime environment as much as you can because a lot of various versions and stuff like that from dependencies can cause a lot of additional support overhead with a project this size. Try also to stay flexible so that you can cope with a lot of the situation because people will always do something that you did not anticipate and update parts even though they shouldn't and stuff like that. And, yeah, use features, new features on your bug fixes, basically new versions as a motivator to please touch a running system, even though it might be counterintuitive. Challenge number two, maintaining backwards compatibility for third-party plugins. I mentioned earlier that Octoprint has a plugin interface. It has had that now since 2015, which was version 1.2.0. Currently, we are 1.3.11. Um, uh, currently the official repository has over 170 plugins. So there is a fairly established plugin system, which allows you to do stuff like telegram notifications or select notifications when a print job is done. And people get very creative there. Um, the good thing about established and established plugin ecosystem is that, yeah, basically, um, a ton of functionality that otherwise I, as the maintainer would have to write now gets written by the community. The downside is that suddenly my tiny little open source project is no longer just a project, but actually a platform that I now have to keep backwards compatibility in mind. Because if I don't, then I have disgruntled plugin authors who no longer want to write plugins maybe because of that. Then I don't have an ecosystem anymore. And that leads to disgruntled users because suddenly they do not have like a Slack plugin or something like that anymore. the challenges with regards to keeping backwards compatibility that I faced over the years is I do not only have to look at my own code but it might also happen that some third party dependency releases a new version which suddenly removes something that you thought was a feature but actually turned out to be a bug and now they fixed it and now your template system is broken and now you have to pin this version so these are hard to debug situations and how to anticipate situations that you constantly have to keep an eye on. Another problem, of course, is code refactoring. If I change a typo in some kind of API that is part of the documented API and so the plugin authors might actually be using it, I have to write wrappers and all of that. And something very current and kind of pressing is this whole Python 2 versus 3 situation. OctoPrint actually is still on Python 2. I know, I know, I'm working on that. the whole plugin ecosystem such is also on Python 2 and telling plugin authors who might have made their first development steps with Noctoprint plugin that they now need to be 2 and 3 compatible yeah this is a bit of a problem that I hope I have figured out by now long story short if you have to keep something backwards compatible read the changelogs of all the third party dependencies that you use If you can find them, it turns out this is sometimes really tricky, aggressively do version pinning for your requirements because otherwise someone two weeks later than your past release will try to install your application and it won't install anymore because some third-party dependency introduced some backwards incompatible breaking changes because they did not do semantic versioning and everything is broken now. and make life easy for plug-in authors because they are who provide your whole platform with a running ecosystem and are a huge motivator for users to actually use it and also to contribute back. Challenge number three, performance. Yeah, the problem is I have a web interface. I have several clients that are connected. I have to feed them data through a web socket. I also have to keep a printer running and I have plug-ins that also do all of that And it's a huge ton of stuff that's running there. And with all of that, I'm basically locked to one thread at the same time, thanks to the GIL. I have a lot of CPU-bound tasks like message parsing, checksum calculation, file analysis and all of that. So I cannot simply say, well, it's just, yeah, they are not IO-bound. So this gets a bit tricky. and multiple processes to use as a solution here is also non-trivial because, again, of the plug-in situation, people might get a bit overwhelmed if they now need to do inter-process communication in order to communicate shared state between various processes and all of that. So, yeah, also not the ideal situation. And also the underlying platform that I'm targeting is often rather limited because in RPi, even now, the newer versions, they are still ARM quad-core CPUs And if you run on only one core and do all of that, it can sometimes get a bit slow. What I learned from that is try to extract CPU bound tasks as much as you can into their own sub processes. Try to always keep an eye on resource usage in your code, even though you might not think that this will become a bottleneck. it might still become a bottleneck. And also, yeah, try to explore IPC options because in the long run, I think I won't have much choice other than that. Challenge number four, I mentioned parsing being a bit of a tricky thing and a CPU-bound situation. And this is actually because the communication with 3D printers is not based on a proper protocol specification. So there is no RFC that provides a well-defined protocol that you can talk to a printer, but instead we pretty much have a wiki page that everyone can edit and that only specifies which commands you can send to the printer but doesn't lose even a sentence about how the responses look like. In order to figure this out, you have to read firmware source code. In a nutshell, there'll be dragons here, and they can be quite huge dragons. New printer models cause issues again and again because some vendor just fiddles around with something and break something in the process they don't even know of. There are also issues with very old printer models on the market because the vendors forked the firmware somewhere six years ago but never did a Git merge upstream again. And so ancient bugs always stay in the system. And all of that introduces a significant parser overhead and maintenance nightmares because you just have to try to match against everything that was sold between 2011 and now. So, yeah, the lessons learned are a bit depressive, maybe. Just try to keep up with everything and use plugins for quick workarounds, which thankfully works here. My final challenge that I want to talk about here is support overhead. As I said, I'm targeting end users here, people who might not ever have written a line of code in their life, who do not know how an issue template is to be filled out, who do not know how to collect logs. In the beginning, this was different when I started this project. Yeah, most of the people who had 3D printers were also quite tinker-happy, were maybe some people who even developed their own stuff for it and such. At least they were very, very capable of using Google to find a possible solution for a problem, and then also maybe to provide not only a bug report with all the info that I needed to fix something, but sometimes even the actual fix. These days, with printers being as cheap as they are, thanks to China, You can get a printer starting at 150 euros, but I would not recommend it because that one can burn your house down. Instead, maybe get the one for 400. Many of the owners getting these things, they had never to tinker before with something. They usually do not even have to build the printer anymore. When I started, I still had to build my own printer. And OctoPrint might actually be the first conscious interaction with open source. Of course, if they are on the internet, they all the time interact with open source. but that might be the first time that they actually have to participate on an issue tracker or do anything like that. And on top of things, most of them for some reason, or at least many of them think for some reason, that this is a multinational company that runs this project and not me. So there's a bit of expectancy management that you have to do here. What worked really great for me in that regard was launching a community forum, being very active there in the beginning to coach people and in general always when there was any kind of issue that was not an actual issue but more of an environment situation like i don't know i bought this cheap webcam off of ebay and how do i use that now i always directed them to the forum i always said go there keep the issue tracker to actual bugs with with vlogs and all that and always go there because there you also have other people that might be able to help you with these problems and that way by now I have basically a whole community that supports each other instead of always looking at me and expecting me to answer all questions and this is actually what I strongly would like to recommend if you plan on running any kind of larger open source project even if it's not end user focused because at some point 80% of my time we're just providing support. And this reduced this back to manageable levels. I could probably talk for another couple of hours about this topic because I've been doing this for a long time now. But I'm now at the end of my talk here. I'll still be around the conference if you have anything that you would like to discuss further or questions or something. But I think we also have some minutes now left for a short Q&A here.

Speaker 2 [21:50]

Thanks for an interesting talk.

Speaker 1 [21:51]

interesting talk. So if you have a question, please raise your hand.

Speaker 2 [22:05]

Hi, thanks for the talk and especially thanks for OctoPrint. I wouldn't ever want to go back to SD cards and the annoying BP menu on my printer. Great. I was wondering, you mentioned that you think you found a solution now to convince plugin maintainers to update to Python 3 compatible. What is that solution?

Speaker 1 [22:24]

And not necessarily a solution to convince them to update, but rather at least a solution to be able to detect if a plugin is possibly working on the Python 3 or not. I simply introduced a feature flag that plugin authors will have to set on new releases. If they have tested their plugin with Python 3 and found it working, then they can just set that in the plugin definition and then OctoPrint will also load it if it is running under Python 3 as well. the plugin repository has also been adjusted to include such a flag so that at least it will not be a situation will not arise where people install a plugin and everything breaks hopefully at least that is the hope right now and yeah I hope that the adoption will also be okay-ish let's hope for the best more questions okay so thank you Jinan now we're there

Speaker 2 [23:24]

Now we have a lunch break until 2 p.m.

Speaker 1 [23:26]

Perfect.

Speaker 2 [23:27]

Perfect.