Building versatile operating setups for real world use and testing with Python and the Raspberry Pi

Speaker 1 [00:07]

This is about building versatile operating setups for real-world use and testing with Python and the Raspberry Pi. So let me have a few words about the ROSE Next where I work for. We see ourselves as visionary architects of progress. So ROSE Next is a forward-thinking technology group, and it was founded by Hermann Rosen in 2023. And although we are looking for decades of experience, so there's a similarity in the name between Rosen Next and the Rosen Group, which is also sponsoring here, and the musical talk is from one of the colleagues at Rosen. And actually, we are a spinoff, so to speak. So in the age of 77 years, Hermann Rosen decided that there are some parts of the company that are not quite fitting the old business model and decided to start over again and build the same thing again. So that's ROSE Next. And what we are doing is we transform latest technology into advanced solutions for harsh environments, meaning latest technology, we are working cutting edge technology actually. And it's about advanced solutions, so it's tricky basically. And harsh environments, it's complicated what we do. So it's very, very challenging, these things. Although we are a young company, we are already around 530 colleagues. And you already find us all over the world. And these are the business fields and areas. So you see some things that are pretty related to pipeline inspection, the core business model of ROSEN. But they are very special. They are corner cases in the old business model, very challenging. It's still all business-to-business, B2B, but there's also one venture, as we call them, that is B2C, so that's creating really end-consumer products. So what it's all about today? We build versatile components to create various assemblies for several business fields, as we've seen already. These include sensors, actors, displays, motors, rudders, foils, battery packs, so pretty much everything that is needed for, say, AUVs or seagoing vessels, fun gadgets. And almost all components are developed, designed and built in-house. And as a short note, besides that, I will show you some AI-made images on these components that we are building because, you know, you can't get it without any AI in your project. So I thought that would be fun. And on the right side, there's already something that I imagined, what a thrust motor for underwater use could look like. Pretty fun, I guess. And I think there should be no worries for us doing the right thing. So I will take some time to get there. So these assemblies that we built, they must undergo rigorous testing and are part of long development and optimization cycles. And that has an impact on how we work in our venture. They take a long time. we expect a time frame from probably 10 years to be ready for a product and so we are starting off with the typical startup idea like move fast and break things and then build up maturity and in the final very end have a product that is ready for the market and here to the left is what AI thinks hydrofoil so the foil that you can use below water to lift you out of the water could look like reminds me of a boat basically and these functional models they include a control unit to connect these components data wise so you have some sort of where commands for the functional mode to do something come from and these get operated some way and finally probably a motor is doing something for you and there is that one unit that connects all these things and has decisions made on how the program flow would look like and for those we have would like to have complete data stream for logging to know the state of the functional model at any point in time and use and for that platform decision made at the very beginning that was in 2018 we decided for using a Raspberry Pi based hardware and a Python based software because you know the Python thing is pretty optimal for starting off and playing around so you have quick really quick progress on your ideas and that on the left the pretty good image actually of a microcontroller I think so that's where I is at home and on the right is a battery pack I am engine for underwater use I'm not pretty sure about if it's the right direction to do but why the pie so you have to choose the hardware platform wisely and it's very commonly used by makers and research and industry these days and the state-of-the-art hardware is at a very attractive price tag the documentation and that is essential is absolutely excellent and you have excellent operating support very much feels like working on a regular computer which is essential for us because with the Linux operating system you have very efficient interactive development cycles and that what really i think puts us apart from the typical embedded linux development where you have these deployment cycles we really work interactively on the raspberry pi for the function models as we build them and that is the quite essential thing that i can fire up my my vi and have some some scripts written down and tested while the model is on change the configuration and see where where it is going without having the complete deployment or process of a new firmware for an embedded system. And that's the architecture behind it. To the left, like, AI imagines how a cockpit for such an environment should look like. And actually for us, it's an electronic platform based on the Raspberry Pi. It has standard interfaces. You probably all know them. these can be used to pretty much attach anything else and there is just a few key components that we were adding to that like the CANbus because that is a pretty essential interface for the industry and attach everything else that we built in-house to that and that is the main data chain and we also have some sensors for like the IMU the initial measurement unit so we have an idea of the orientation of our devices, why we use them, GPS, like when we are, when we're using these devices and see that we know how fast are we, where are we, are we in correct state and these things and we think about extension sockets to connect other things. We came a long way when we were developing it. We started in 2018 and that was the initial board that we built. that was for a Raspberry Pi, standard Raspberry Pi, like the Zero or the A+. With the connection with the GPIO header, the typical 40-pin header, and we just put everything that we imagined was of importance to it, so to decide later what really makes it in our functional models or in the use. Like the real-time clock that is pretty essential, you will know, you want to know the time of your data point when analyzing it later on. The CANBUS, GPS and IMU, environmental sensors for temperature, monitoring, humidity and these things. And also ADC and DAC converters. And we added the USB socket that you can see in the right top corner for adding Wi-Fi or Bluetooth. Our next iteration already was with a compute module in mind. So when you do a product for the market, if it's for industry use or end consumer use, then the standard Raspberry Pi is not what it's meant to be. This is the compute module that is for industry use, that is for end-user products where you build your own platform upon, and that's what we had to go for, and that was the board for in-between use. So it was still the idea of the GPIO header to attach a regular Pi, but on the downside, under the board, already was the connector for the compute module. And we already made some decisions, like having all possible MIPI ports like for DSi and CSi available that the Compute Module offers so there are two camera and two display ports available for the Compute Module and we had some use cases to use two cameras at the same time to display so that was necessary for us to do and also have reduced other things that we need well that wouldn't really make it in a product later on and And that's the version that we are currently using, exclusively for the Compute Module 3, already made with some decisions to no longer use some hardware. And there on the top image, in the lower part, there is this black high-speed socket, a high-rose connection for an extension board where we can essentially place everything that we've gotten rid of beforehand. So if it comes back, then there will be options. And now we are going for the compute module 5 and the PCB is still getting a bit, we shrunked it in size and we made some more decisions to get rid of components and on the underside, on the below side there is still an option to create extension boards and this will be the smallest footprint board that we ever made for that. and we do remote controls as well some at some point there are there are models where we have to remote control things and for these things we use the wonderful esp32 platform there are boards that you can buy off the shelf just add a lithium-ion battery and has displays on it add some sort of buttons to it and then you have a remote control and to use that you can use micropython. So the control unit, the software part, this is how the AI imagines a throttle grip would look like, that you can use for a gas handle to get the motor running. There's even a hand somewhere. And we use Python for the software stack, so every component attached is controlled by a dedicated application, that is really key. We didn't have in mind to have one application that drives it all so there's a dedicated application for throttle grip there's a dedicated application for the motor there's dedicated application for just the GPS chip and we connect it in some way later on and these components they are exchangeable so we have different variants to do the same job and we carefully chosen the API's for that so we can make them interchangeable So just by configuring. And for communication between these applications, we've chosen to use an in-memory database. And we just rely on a PubSub pattern to be available and regular getters and setters with a time to live support. And actually it's Redis, but that can easily be replaced because, you know, Redis. So that's how the architecture looks like. We pretty much have a group of possible sources and there are loops and sinks and that makes the data connection from some source like a throttle grip and there is commands going over the line into the Redis in-memory database and they might be transformed in some way depending on the input of other sources that is done in the loops and that finally ends up in a command set that goes down the line to the sinks. And that gives us some pretty much good configuration possibilities to build all sorts of setups that can do completely different things. There is data communication and storage. So to the left some samples for variable names made by AI. And these data we represent as records that are regular Python dicts basically. we implemented the key value store in these dicts by choosing variable names that have certain prefix in it so very simple decision just in the name of that to help distinguish between things that are of the kind requested or target or an actual value or some model value or some estimation and these components identify via namespaces like for what's what thought or what sort of a component is it? Is it a sink like the throttle grip or is it a source like a throttle grip or is it a sink like a thrust motor? So we can have an idea to what kind of component the data relates. And this can be worked out downstream in the whole loop on the device, but there is also the idea that we save the data for analysis later on and for that we merge it. We have done this differently before using each and every component has saved its own data but it's really a pain for time wise merge data afterwards. So we decided for just have one timeline on the functional model merge every data on it and live with the latency that this introduces in the data. And that in the end fits typical data frame processing pretty, pretty well. So the fun part, how does it look like? So there we see a functional model really on water and one of our test drivers just playing around with it and probably having a lot of fun, I guess. And to the right is the matching GPS track. And, you know, based on the battery capacity, the batteries that we built that let you run this device for, say, half an hour or three quarters of an hour, that is quite some fun you can generate. But we're letting this idea grow with a functional model operation. We were pretty happy. It was pretty robust. But the individual component testing is also a thing. You know, it's Pareto principle. It takes you to get to the final set. It's a lot of effort. And these standards computers usually are tested in developed environments with dedicated hardware and computers using MATLAB and LabVIEW. And we thought, well, the devices that we built, this control platform is pretty versatile. So why don't we try to use that for our measurements? And that worked out okay. And as a result, we have pretty much the same workflows for the operating functional models and experiment control. We have the same data models, so comparing data is very, very easy to do. And the same workflows for visualization and evaluation, and we do the same development. It's the same software stack for everything. That reduces efforts for us quite a lot. Now, what can we do with that? And to the left, we have our linear test rig. It's an 18-meter test rig where we can place our functional models all these components and pull them through the line. Pretty much looks like that. And the very end there is the carrier and currently the specimen is placed below water and shortly it will accelerate and get that system down the line to collect data. And behind the scenes is the Pi that is driving the sensors, the motors, this time it's just data collection and streaming the data to a cockpit so we can see the live data and try be happy with that and work with these data later on and with this rig we're doing roughly it's almost 5 000 pulls linear pulls a year currently and the same we can do with our friends at our reference rig in the university of rostock they have decades of experience they have their own setup but we try to use our own measurement setup there as well so we build a black box that you can hardly see on the carrier just coming to us where the pi is and this is pretty much empty volume and a small little pi is somewhere safe but with everything connected and then we can compare all these data sets and see if we're on the right path so we have a variety of test programs we can grant discrete or even grant carrier speeds measure lift drag thrust and side forces determine the electrical properties calculate component efficiencies compared to FEM calculations that is essential because the main development work you do in FEM calculations you can't do each and every parameter set up in real world and by that we can monitor improvements and that's what we're really looking at the top left is the PI script based dashboard that was also an essential step for us so that we do not have to deploy the necessary software to each and every notebook for the one who's doing the test, we just have the Pi, and the Pi is delivering the software to your browser and everything is working in the browser. That's quite neat. And it generates the same data mode, so we can do the same plots and then try to see if our imaginations work and if we're on the right path. So that's pretty much all I have to say today. So now to conclude. we had huge advantages with our approach it was just single hardware and software platform to develop and that was quite important with just a few people behind this and time is limited and there's a lot of work to do so we had to make decisions and that was there was a really what was really a good one and the data form and stay the same the evaluation workflows are the same data visualization is the same, and data handling and bookkeeping is the same. And there was my talk last year where I told you about how all the data that we collected, we built our own warehouse and have Jupyter Notebooks to process all the data and build actual books that show you the progress. So if you're interested in these things, have a look at the videos of last year's, so everything is just the same. And that's all for today. And I've spoken about that we have a lot of work to do, so we are hiring. If that was interesting for you, I expect all your applications by this evening, I guess. I would be very happy to talk about these topics to you. And yeah, thank you so much for listening.

Speaker 2 [19:57]

Thanks for this interesting talk. Let's have a look at the questions from the audience. So let's start. What do you use for storage in your Pi setups? SD cards are quite unreliable from my experience.

Speaker 1 [20:13]

We are using SD cards and they are not that unreliable as everyone thinks. It depends on the model actually. And I don't think we have lost very, very much SD cards in these years. And we are pulling off the battery. So we are not letting the system shut down properly. It's just taking off the energy and putting the next in and that works. But for the product it will not be an SD card and we also have the computer models with flash on it. That flash is more reliable, that is the way that we're going for. But we still have startups with SD cards in use and we will probably continue to do so. Choose the right model.

Speaker 2 [21:02]

Okay, next one. How do you upgrade the software of the Raspi in the field?

Speaker 1 [21:07]

We would be able to do this online, so it would be a Git pull to do so and restart. We have online connection via Wi-Fi access points, so that would be possible at the current state. Usually we prepare in the factory to have the setup that we would be testing, but we are able, the system has Wi-Fi. We were able to, when you remember that video with that guy on the sea, on the lake, we have a connection then. we can change the complete behavior of that device while he's using it. We don't do. Yeah, above water. So in seawater, things would be more complicated, and we had a lot of discussions and issues about where to place GPS and Wi-Fi antennas. We got this going.

Speaker 2 [21:59]

Next question. Can you give an example for a concrete product this is integrated in?

Speaker 1 [22:06]

Yeah, if you count one and one together and remember the business fields with the consumer thing, that might be one coming.

Speaker 2 [22:19]

How do you handle the eventual consistency of interop between apps? I'm assuming contracts, but maybe give us some details here.

Speaker 1 [22:30]

I don't think I got this. Duxo. Maybe a little complicated question, but after all, not that complicated. I was just wondering if you have this decoupled thing with multiple apps, and they communicate via a pub-sub system to each other. I would assume that over time, the formats of data change and evolve and how do you handle this? Ah yeah, so the idea was even though there are different applications for each and every component they all share the same base so if we change things in the base they will be available everywhere and we set it on a very robust simple setup it's just python dicts passed around json encoded in that stream and you put the JSON encoded things in the Redis database and you pull it from there, you poll it or even you subscribe to it depending on the latency that you expect and that works quite well really and to get it on storage it will be message pack in environment format but the same thing and that gets loaded and put in a data frame and then we are ready to go And we are reprocessing the whole data set. Remembering my last year's talk, the idea was to never touch the raw data. And when we change the model, the data model, for having advantages, then we reprocess everything that we did. Ah, there is a video for that. We can talk later on.

Speaker 2 [24:20]

Would you use Pyscript again?

Speaker 1 [24:22]

I do, I do. It is an amazing project, really. I try to place talk for that. This conference, last year's conference, not that popular. Hopefully at some time. It is a great opportunity these days to just have a JavaScript alternative that is so Pythonic. You can deliver the whole Python interpreter ecosystem to your browser and simply work with that. And the opportunities to modify the DOM, it's absolutely crazy to do that with Python. It's just pure fun. I would recommend doing more with that for everyone. That's really fun.

Speaker 2 [25:04]

Does running the display stack on the Pi have a measurable impact on data telemetry collection or control loops or are control loops out on microcontrollers?

Speaker 1 [25:17]

They are not on microcontrollers because we don't have any real-time needs currently. So when we look at these models that are basically driving motors in Walters, in Walters that is a slow process where things change. You hardly notice with frequent changes, hardly anything happens that you notice. So we have no real-time complaints there, and then everything works out quite well on the Raspberry Pi itself. but we could move to a microcontroller anytime and the motor controller itself, it is a microcontroller.

Speaker 2 [25:55]

Okay, did you have any problems integrating external or internal hardware into the system?

Speaker 1 [26:03]

Absolutely not, because the ecosystem for Python is amazingly good. So the Raspberry Pi itself, due to its documentation and the large effort that people put behind it, is an amazing platform. It's much easier to operate than the professional ones, I think. And then there is the hard work of Adafruit with CircuitPython. And there are so many common chips that they are ready-made drivers for Python that you can simply install and use. It's crazy. So it is very simple. In the very beginning of the project, I had done some drivers myself. And just recently, I've written a Linux driver for a display because there was none. We needed to have this operating. And that is an effort, but usually it's selecting your chip and there will be some support somewhere and you integrate it.

Speaker 2 [27:03]

What kind of pile are you using?

Speaker 1 [27:06]

All sorts of. We currently have in active use the Compute Module 3b plus and we're going for the Compute Module 5.

Speaker 2 [27:18]

deploy via git pull has versioning ever been a problem because of that which made you want to switch to installing your apps as packages instead

Speaker 1 [27:32]

I don't answer that without my lawyer. Was that your question? No, to answer it. We are operating based on the maturity of the project. So if you are in the mood of breaking things fast, then you are just changing things. And version control is a place to get things safe, remote, and that's it. And at some point, you get more professional, and then things change. So we have built pipelines. They create packages. They have package numbers. There are setups where that really matters, that things work out. But I'm most usually in the mood of breaking things fast.

Speaker 2 [28:25]

And the last question, what are the benefits to using a PubSub pattern with Redis instead of using gRPC as the inter-process communication on the Pi between the components?

Speaker 1 [28:39]

choose whatever suits you best so that I've decided to go with Redis was probably that I was playing around with Redis at this time and it could have been any other solution that just offers PubSub and has setters and getters and provides time-to-live support. Time-to-live is essential because there are these commands that go down the line if they fail to count then we need to have for safety reasons that the motor shuts down. So placing a variable in the red is saying 100% motor speed and keeping it that way while the environment is not safe is not a good idea. So these are the constraints and then this works out pretty well. And another thing that is of importance to maybe conclude with this, and I left it out probably in the talk, is that each and every component, the failure state of each and every component is basically covered by the Linux system, by systemd. So if something happens, I just let the application fail and go down, and systemd will restart it and automatically repair everything, so it works out afterwards. And that is also a nice thing to take with you when you're using Raspberry Pi in a typical Linux system, because the embedded guys have to do watch docs and cover everything for themselves and I just let the application fail and things will recover and people will come back from the lake.

Speaker 2 [30:12]

Yeah, let's thank the speaker again.

Speaker 1 [30:14]

speaker again.