Embedding Data Science in IoT devices with MicroPython and emlearn

Speaker 1 [00:02]

All right. Good morning. And so welcome to this session. And now we're moving from the big data centers to IoT. And I'm happy to introduce you to Jan Nordby, who is going to present a talk about embedding data science in IoT devices with MicroPython and Mlearn. So please give a warm welcome to Jan.

Speaker 2 [00:28]

Thank you so much. Yeah, so we will be talking about very small devices. I think it's one of the few talks about hardware related at this event. So if you didn't know, Python will actually run on a small device like this. You get smartphone watches or many other form factor devices you can put on battery or have around. So we'll see a little bit about this today. So just as a question, Has anyone here played with MicroPython before? Yeah, there is maybe a fourth, so you will hopefully learn something, and you that haven't played with it will definitely learn something. I work for a company called Sound Sensing. We provide sensors for monitoring ventilation units. Our customers are operators of buildings like this, so we notify the operations team when there's something weird. We use MicroPython for prototyping new sensors there, and also of course we use Python for all the machine learning, analysis, anomaly detection, and backend. But today I'll be talking about a side project of mine, which I have had now for seven years already, about doing machine learning on microcontrollers, and for two years we've had MicroPython support in this open source project. In this talk, we'll talk about making a smartwatch as a standalone IoT device. If you have a smartwatch today, they are very dependent on the phone and on cloud services. A very small part of the system is actually on the watch. So it's interesting to see, with Python, with MicroPython, and taking a local first approach, approach. How far can we go and what are the bottlenecks currently in the ecosystem? Because I'm trying to expand the ecosystem for data science on physical hardware with MicroPython. So we want to make some hackable device. We're not making a product, we're not making something super polished and we want to understand possibilities and constraints and we want it to run Python. We want to be able to do some machine learning with activity recognition so we can tell you what have you been doing in the last hours and days. You can do some statistics on that and it should have a few some days of data otherwise it's it's not that interesting. And that causes some challenges for a small device and it should be able to serve the user interface as well from itself so that you don't have to have an app or a server that you install at home or a cloud service etc. Ideally it should all be from the device. But optionally you should be able to integrate it, of course. If you want to, you should. So it's local first, but not necessarily. You can choose if it's only local. So hardware, there's lots of possibilities these days. If you get started with MicroPython, I would recommend going with some ESP32-based device. They are cheap, affordable, from like three euros to, this device is like maybe around 50 euros in total. It has a lot of sensors and devices and things like that, including touch screen etc. So this is on kind of like a higher end of a microcontroller. So it has 16 megabytes of flash for our program data etc. and 16 megabytes of ram which for microcontroller is a lot. But we'll see how far we get with that. And of course it has an oximeter which will be our primary sensor and wi-fi which will be our connectivity. So, MicroPython is an independent implementation of Python 3. It doesn't share any code with CPython. It has quite surprisingly good compatibility. It supports devices that have at least 128 kilobytes of RAM. It's mostly fun if you have like one megabyte or above. It's a bit tedious below that, in my opinion. It supports a lot of different devices. ESP32 is a great start. Raspberry Pi Pico, etc. is also a great start. I am not going to talk that much about MicroPython. I give a talk at FOSDEM if you are interested in that. The basic premise is that you install a pre-built firmware. For most devices, you will find that. Plug it into your device. The tools for ESP32 is in PyPI, so it's very easy to get if you're a Python developer. And also a tool for talking to your device called NPR remote. And then you flash it and you have a prompt, you have a REPL on your physical hardware. And it runs completely standalone on that device, which is quite amazing. And you can of course use an IDE if you want, but this will get you started. And there's some interesting, lots of functionality. It comes with, you could say, batteries included from the Python perspective. Interesting here is we want a file system so we can store data. And this MP Remote is useful for transferring, for example, our configuration, our machine learning model, etc. We can transfer via MP Remote. And it also has a package manager. They tried to use PyPI for a while, but now it's a dedicated tool and repository, which is very tailored to these small devices. So you can install packages like this, either from a common repository, or from a specific GitHub, or a specific file, just on HTTP. And you can also install native C modules at runtime. This is a very unique functionality. You can actually extend your, not just with Python code, but with the C module that exposes a Python interface. And this is very powerful. And I haven't seen it on other embedded devices. There is a little data science ecosystem. Because the C API is different from C Python, but also we have typically one megabyte of RAM. None of these libraries will run with one megabyte of ram for sure. They barely run with 100 megabytes of ram. So there are alternative implementations. I maintain emlearn micropython which is basically scikit-learn for micropython. There's something called openmv which is very similar to opencv. And there is uLab which is a very compatible implementation of numpy. And there's like a lot of tools around that we're building. And I'll show you a bit more today that we needed to build for this project. So EMLearn MicroPython has two components. There's a library that you install in Python so that you can convert your scikit-learn or Keras models onto efficient implementation, which you can deploy on the device. And we support both C deployment and MicroPython deployment. And it's a single .mpy file, so you can actually install that at runtime. You don't need to rebuild your firmware, etc., even though it's mostly implemented in C for the efficiency. Today we'll show just a little bit about this activity tracking. I've given two talks about that before, so I'm also going to skip through it quickly. Typically, this is the classical pipeline. You separate the orientation, so how are we standing or down, from the X-rometer data. The residual of that is basically an estimate for the motion. Without our gyro, you can't do very good positioning, but it's good enough for activity detection. output will be typically these kind of classes. There are examples in the repository for a couple of common data sets, so you can use that pre-trained classifier. Although make sure that sometimes you need to reconfigure the scale of your data, and also axis orientation might be different in different devices, so that can be critical. There are many other examples for EM-numeric pattern, of course, and these two talks go in detail about activity tracking with MicroPython. So on our wish list, how are we doing so far? So yes, it's possible to run Python on these kind of devices, which is already super cool and that opens a lot of possibilities if you already know Python. And you can get off-the-shelf hardware as well, so you don't have to be an electronics engineer, you don't have to do a lot of soldering to get started. You can just plug it in via USB and you're up and running. It's possible to do activity recognition with the EMR MicroPython. There's examples for that. But then how about the next pieces? For this project there's a repository that's linked in the conference talk page. There's a link to a repository where I've been experimenting. And I found very quickly that in order to have multiple days of data and then be able to query it and so on, we need some sort of representation which is not just like one flat buffer. We need to be able to continuously append to the end, like the current data, but be able to, when you open the UI, query back in time or select time. So I created a small time series database or time series data lake using Apache Hive style partitioning. So you partition based on the day, hour and potentially down to minute if you have very high resolution data. And then you can have different resources like this so that you might keep your raw data just for some hours, but then the outputs of your machine learning model you'll keep for several many days, maybe be even a month or maybe even longer so that you have this granularity and you can in the small space that we have fit both a long context but also a rich raw data. So this is in process of being moved to this repository so that we can have it as a general tool in the MicroPython data science land. I tried row based and columnar chunks and columnar chunks is nice for trying compression, etc. But it didn't work so well. This is just a typical API. We append all the raw data, and then when we have enough data for a whole window, typically like four seconds, that we're going to classify the activity, we will process the window, which is the pre-processing of data. We store those features that we have extracted, And we also do a prediction and we store the classes that we get. This is the main part, like we're pushing, pushing, pushing. And then we have an API which we can get long periods of data, which will be for our UI. So the key is performance, right? We don't have a lot of a big device. So I tested and we were able to read around 100 kilobytes to 250 kilobytes per second. Which is not that much if you think about it in the normal scale of things. But actually it's 100,000 data points per second. Which means you can do two hours of one resolution data per second. If you're willing to wait 10 seconds-ish, then you'll get a whole day. Or you can get one year of one hour resolution data with 10 columns. So this is already quite okay. I was hoping to maybe get 10x of this, but it's already at a useful level. And it's really IO-bound. So I found that reading these chunks, even though I tried to size them well according to the flash precision and so on, it's really the reading that takes time. And that also influenced compression. And raw accelerometer data, which is the highest resolution, takes the most space. I only got 5% net savings with the compression. So it was not that great. Let's see if we can do better. But it's possible to do 24 hours with raw data without compression and still have space for our other data and our frontend and our firmware, et cetera. So I found that very successful. we need to serve our web interface. There's a project called MicroDot, which is basically a flask-style web server implementation for MicroPython. It's super well documented, very robust, has really good features that you need for these kind of devices, like being able to do streaming, so you can chunk up your response, and being able to serve pre-compressed files, So you can serve the pre-GZipped static assets, for example, for your JavaScript, which tends to be quite big. I'm not going to go through much, but here's how to do streaming with that. It doesn't support asynchronous generators, which would be of course even nicer, but that's not supported in MicroPython, I think. Then you can plug it together, and you have an API that you can query to get this Grafana-like view. We're still in Hatch Hacker mode, it's not a very pretty interface, but this exposes the features that have been computed, as well as the predictions. you can choose what you're looking at. I'm testing it on the device and in a few seconds, you can load 24 hours or a few days of data. I use Plotly, which I like a lot. It's very powerful and nice for interactive. The default one is over one megabyte, so it really hurts on this device, but there is a minimal JavaScript as well, which has most of the things you want, and that's just 250 kilobytes, so it fits nice. And we use Preact for the UI, which is like React, but you don't have to do a build step. Important here is how we're doing multiple things at the same time. We are collecting data, analyzing data with machine learning, pushing that into a data store, and then someone might connect to the web interface and want to look at the data. So how is this done in practice? And super nice in MicroPython is that it supports asyncio. So you can do this, at least conceptually, in the same way that you would do with Python. But we do have typically more stringent timelines. So asyncio is cooperative scheduling, which means that it's important that the individual tasks are rather well behaved that they yield at some point so that other tasks can come in and do some work otherwise they will starve each other and this is especially critical for the sensor data readout because if you fill there's a buffer in the sensor itself and if that is filled up you are going to lose samples your that data is forever gone right there's no way of catching up at that point Thankfully, on this device, there is a rather large buffer that can take around 100 samples. That is around 4 seconds at a 25 Hz sample rate. That is actually quite comfortable, but you probably want... I have not added the user interface yet, but we will have on that. I want to have a responsitivity better than 4 seconds. It is a general recommendation that you practice to keep all kinds of chunks of work under 100 ms or even a bit lower, if you can. you can. But that means that if you have longer tasks, like for example when we are responding to like, give me all the data for the last weeks from the web user interface, that needs to be chunked up. So that's why the streaming is critical when you are fetching from the data store. So then yeah, the whole process takes maybe four seconds or maybe more, but it's done in a way that the microDOT will actually yield to the asyncio runtime in between chunks. So then the task that does the sensor readout get it just a chance to read. But it is critical here to use the buffer in the sensor. If you didn't do that you would basically have to your deadline would be now 25 Hertz you couldn't miss and even if you would a little bit of delay, you would actually get jitter on your data. So this is a really key part of this kind of system to make it feasible. There are ways to do it with interrupts, etc., but I would always recommend going with a FIFO-based approach for this kind of advice, even if you would do this in C. AsyncIO is very familiar, but keep deadlines in mind. and you actually want to guarantee the worst case and that if you have multiple tasks that can be multiple things before your critical task comes in and that's general embedded practice but do this and you will kind of be in an okay spot so that's a little bit further so yeah it is possible to store enough days of data when we do this Hive-style data lake type thing, and we can serve it without compromising the ability to read and analyze data. And it's possible to serve some browser UI. Basically you need to dedicate like one megabyte to your front end, which is a lot in firmware terms, but there is enough on this kind of device. What if we don't want to just local? What would we do then? Oops, that's missing stuff. I'll have to just explain it. The UI that you saw that gives you this Grafana style is of course backed by a HTTP API. You can just basically from your device copy that URL and as long as you're on the same network, which you need to see the web interface anyway, you can just curl that part and it will stream down a NumPy array for you. So then you can plug that straight into your ordinary data pipeline if you want to do some more analysis than you could do in this very basic web interface. So that already that's kind of your escape hatch right as a data scientist and that's very important to have this yeah you provide some standard functionality but we want a hackable device to just have a open hatch where you can get the raw data. So that's important and you can use that of course for backups or for syncing so you can keep around one day of data on this device which means that maybe you want to have a script that when you're at home it will transfer it to a longer. So if you want the raw data but you might want to integrate it like into some more of a system Then you could push data via HTTP or MQTT. You have a full Wi-Fi capable device. But here it's a little bit tricky if you want to make sure that you don't risk losing data. For example, if you have an external server, that could very well spend more than four seconds responding, especially in a downtime scenario. Or Wi-Fi can be unreliable, so you might fall on and off and so on so it's always good to to if you have data that you want to have guaranteed delivery keep them in a in-memory queue or even in a disk-based queue and then only remove it from the queue if you when you've gotten confirmation that you it's been sent and acknowledged by the server and you really want to implement timeouts to that so that you're not you have a bound for how bad this influences the rest of your system because there is just like one core there is no way for it to do anything else, as I mentioned with the asyncio. So you need to kind of kill it if it takes more than a second, for example. And unfortunately, there is a very nice asyncio-http package in MicroPython, AIO-http-styled, but there is no timeout support. There is a nice to do implement timeouts in there. So that's a possible contribution if someone wants to do that. So I unfortunately cannot recommend that, but that's like a Python wrapper on top of a socket implementation in MicroPython, and that has timeout support. So right now you would actually have to probably manually do your HTTP request with the socket if you want the timeout. In MQTT it's a little bit better. The simple MQTT server has timeouts built in. That would be typical for sending data to some IoT cloud platform. There are many nice ones that you can get like a free plan and integrate if you want. So a little bit of stuff that we should improve in the MicroPython community for these use cases. And that's the purpose of these kind of things to find what we need to fix to make it really good. It is possible to get or push with this, with some caveats, if you want to do it properly with the run timeouts. That is all I had. To summarize on this experiment. It is feasible to build a standalone IoT device with MicroPython. We have the hardware, easily accessible and affordable. There are a lot of open source libraries available that you would need for this. You have machine learning models implemented. You have web servers, you have UI elements that you can use, and with this grade of device, which is not crazy expensive, you can use quite standard things, like Plotly, for example. I was surprised that that actually worked out, that I could fit that, and you can use your existing skills as a data scientist. I think it was very good. One new thing we needed was this Hive-style data lake. That's now going up on GitHub, so that we can have more nice things in the future. Thank you very much.

Speaker 1 [23:46]

Yeah, thank you, Jan, for your great talk. We have some questions from the audience. So the first question, can I extend MicroPython with Rust as easily as with C?

Speaker 2 [24:01]

Yes, but you do need that C kind of wrapper, so that's generally the case. So you just need a little skeleton around with the C around your Rust code. There is also a project that actually does Rust to Wasm and has a generic binding layer that you will find that could be another way. That's a little more experimental, but could be interesting also.

Speaker 1 [24:30]

So next question, which data set did you use for training?

Speaker 2 [24:33]

I use the PAMAP2 dataset. There's also in the examples, I think, for UCI-HAR, which is also very popular. It's got all the not so big datasets, but they have typical everyday activities, at least little selection.

Speaker 1 [24:56]

All right, not really a question but like two people like asking like will you publish your slides on github, please? Yes

Speaker 2 [25:04]

Yes, they will be there for sure right after the talk.

Speaker 1 [25:08]

All right, yeah, and then it goes into that is your code snippets published for that prototype. I guess yeah, yes

Speaker 2 [25:13]

Yes, I think it's already linked from the link in the GitHub.

Speaker 1 [25:22]

And another question, what is the battery life with this setup?

Speaker 2 [25:26]

That depends a lot. We didn't go into power management, which is a big topic, but Wi-Fi will kill you. If you keep it running all the time, it will probably take two hours. So it's very bad. What needs to be done is to have a mode. It doesn't need Wi-Fi when you're just running around tracking. You just want to have a mode that is like, oh, I want to connect to it now. In AsyncIO there is no integration with the sleep functionality out of the box in MicroPython. It would be nice to have, but there is an alternative implementation called asyncio-alt from Peter Hinch. He has a super good tutorial about AsyncIO on MicroPython. And that one has power saving support, so you can swap it out. I haven't tested it for a full day, or full 24 hours, but I think we'll need that in order to reach 24 hours, or like 24 hours to 48 hours with Wi-Fi off, or at least off most of the time. You You can do, for example, wake up, five minutes, dump your data, and then go back or disable Wi-Fi again.

Speaker 1 [26:52]

All right, I think that's it. Thank you so much, Jan, for your talk. And, yeah, next up in this room is how we built an inclusive data organization, careers, community, and 50% women.