Using Micropython to develop an IoT multimode sensor platform with an Augmented Reality UI

Speaker 1 [00:04]

Can everyone hear me okay? Yeah. Okay, I'm a Scottish person, so I'll try and speak clearly. Okay, I work for Samsung. I'm part of the Research Institute in London. And the last nine months I've been working on some cool technology with augmented reality and IoT. So this is really, really cutting-edge stuff. And I'm going to try and do a demo. Hopefully it will work. And if anyone wants to, they can come up and... probably if anyone wants to they can come up and try it in one of the phones one of their phones but we need a volunteer who's got a modern phone uh so anyway it's cutting across lots of technologies so i'm going to really keep it as simple as possible and so i'm going to try and just show some of the code to show you know the overall view of how this works yeah so So what we're going to cover today is IoT and augmented reality, what's a microprocessor and micropython, sensors and how they work, a tiny, tiny web server which is going to be running one of these little IoT boards, we're going to dive into the hardware, a bit of a demo, and then do some real augmented reality, really, really simple, and then that's it so I will try and keep on track and I will take lots of questions and one other thing is on Saturday we're holding a workshop we'll have lots of boards to try out and some of the sensors so if anyone wants to try this out just come along it's on the main web portal for you know the organizers so go into the sprint section and you'll see it and what else that's it so the way I work my slide is I have little icons in the top right hand corner so you know what I'm talking about so overview is little sticky notes and then your idea with a symbol in the middle big picture with people icon and then the cogs for code and then how it works with the database icon just to keep you grounded again we're going to cross lots of technologies and keeping it really simple and I apologize if I'm not being clear enough I'm trying my best to make it clear right so something amazing before we start 1% of all the microprocessors are in your laptop or general-purpose computers and the rest are elsewhere so there's only a tiny fraction of the microprocessors that are available or actually on your PC so where's all the other ones this is just a word cloud to show where all these processes are what they're doing it's just to give you an idea that everywhere they're in microwaves they're in ovens and they're becoming more powerful and they're doing more stuff so even at Samsung we're looking at trying to get speech recognition on air conditioning units for example. Even the Mercedes S-Class, this is in 2010 I took this from, it has 63 microprocessors on it. It's even got processors on the lights so they're everywhere. And another thing that's driving this is Moore's law. I know this isn't linear anymore but it's just giving you an idea of that the processing power is a lot cheaper and the other one is the cost to transmit data is really really small now so this stops 2010 it's insignificant in fact I've just heard you can buy 10 narrowband IOT sims for 10 years for $10 so it's getting really cheap to transmit data right a bit about this project. So basically we were challenged at Samsung to try and gather data. So the AI teams want to build models and typically they buy data and so they'll buy in data sets and it costs a lot of money, hundreds of thousands of pounds a year to Samsung to buy in data sets. So they thought why don't we just try and put our own IoT devices all around the campus and get our own data sets. So that was the idea about how we're going to do that. So we thought about having a proof of concept of using some IoT hardware and we decided to use the the micro Python board it's really low power 47 milliamps weighs less than 9 grams tiny 34 by 28 millimeters in size but we had to try and think of getting a micro web server on there as well and how to interface to that device and typically embedded systems you don't have big real estate screens or keyboards you've got nothing it's really hard so the concepts we're thinking about we're using the phone to try and get an interface to the device does that make sense yep so wouldn't it be nice to just bring up your phone like i've shown you in a snapshot bring up your phone and then get some data from that device that's the kind of concept yeah right so what we're going to try and show today is this is it in a nutshell those are the six steps the tiny little web server running in the number 1.1 and that's going to spin up and then we're going to take data from these tiny little sensors and a phone will hit an endpoint and then get effectively a web page back in step four and then the phone will then try and render that page it will use AR.js in A-frame to render that page and then when this activates the phone is going to try and find a marker and when it identifies a marker it will try and display or render the data on the marker so the marker looks something like that here's a marker so it allows the phone to pick up the proper place to place an image. There's lots of different types of markers. There's markers with QR codes in as well. But we're just doing a demo to show that the concept hangs together. And number one and two here are the little micro-python boards which I'm going to hopefully show you. But I need to try and explain the overall picture first. Normally I do the demos at the start but this is such a big topic i kind of just try and explain the top level view so there's probably people in there that know about iot or know about augmented reality and so on but there's people are not so effectively microcontrollers started the iot revolution happened around 2005 and i've stopped around 2015 it's still going and it started off with arduino i don't know if any of you guys used arduinos so there was more of a toy and more of a prototyping tool and then embed for my arm came out and then you've got Esperino and a whole raft of others and then the Raspberry Pi which is really a general purpose computer really, it's got an operating system on it, it's not running bare metal either language like Python or JavaScript it's a full blown Linux OS on the Raspberry Pi so I put it in there because people understand it as an IoT device and at the very end I've got the MicroPython board which we used and I'll explain to you why we use that in a minute but I've got it at the very end because it only came out in 2018 and we had it in the lab in 2017 we're looking at the prototype and there's benefits to why we use that which I'll explain in a second so what is the internet of things the internet of things is a system of integrated computing devices blah blah blah that's a wikipedia it's sticking in there so the slides are online you can read them but effectively in a nutshell to me IOT is a thing that sits there and can communicate with other things via some network. It could be LoRa, Sigfox, cellular, Wi-Fi. It doesn't just have to be one particular technology. So it's technology agnostic. Problems that happened with the first set of IoT devices, they're all basically C-based. They weren't using Python and high-level languages. So they're quite hard to use. so if you have picked one of these things up you might work on it and think yeah this is great I'm using it and then you take it somewhere else use different libraries, everything doesn't work C was designed to be portable but it's not really and it's difficult so if you get web developers or people who are used to using high level languages they don't really want to work in this low level construct and it's difficult, you've got to manage everything on these embedded systems you don't have or sometimes or most times you don't have an operating system handling everything underneath the hood yep so that's when these things started using high level languages and i've got i've even got a board that uses javascript bare metal so why micropython so micropython isn't just uh python and uh micropython you know binary that It just sits there. It's the whole ecosystem. So you have the main web portal, the web store you can buy stuff on, and then your GitHub repo. So there's kind of three parts to it. And we looked at that and we thought, well, there's a lot of action in the forms. There's people using it. And then the new stuff came out and we thought that fits the bill for what we wanted. We wanted a modular system that we could build these sensors around the campus where we could plug in our own sensors and get data. We didn't want to have to reinvent the wheel. didn't have to develop our own hardware we just wanted something we could put together quickly right so this is effectively the thing we're going to use there's three components to the thing we're going to use there's the brains part number one which is the microprocessor flash and so on uh and that plugs into number two which is really just a little board with pins on it and i'll explain pins in a minute and then number threes are tiles or little sensors we can use so this is the part we bought into we can plug in our sensors and we can change the sensors so they may be in the kitchen we may want sensors that will do rate of rise temperature or it may be in a hallway and you want a passive infrared or it may be as an alarm system and you'll have a Doppler radar on there and I can show you all this stuff as well I've got it with me so in fact I'm going to switch across to there can everyone see that I apologize for the So that's the actual real board. So here's the pins in the back. And here is the microprocessor. And here's a tile with three sensors on it. But if you want to see what the real prototype looked like... So this is actually from the lab. This one's got passive infrared here. you can see a little dome and then over here you can see the doppler radar you see it yeah that's a that's a doppler radar uh and we've even got a little led screen here which we use to put stuff on and we connected that to uh the micro python mode so the the idea would be once we're happy with this we would take the sensor and make it into a tile yeah so we'll just quickly show you a tail so here is another one they're really tiny so I am going to try and get a web server running on that there's only a hundred 20k of ram uh two mega flash and here is a little tell this one doesn't have anything on it so this one has got pads where you'd solder your your uh your test sensor on it and here's your down rail at the back where you plug it in and the down rail stuff's come from mobile phones actually so that this technology is promiscuous. Does that make sense? Oh God. Right. Okay, so wouldn't it be nice if we could hold up the phone, and here's a demo in the lab, I took a picture just in case the demo doesn't work. So we hold up the phone here, and then And up pops, you can just see this is just an object, a 3D object that the phone's rendering. So it spotted the marker and then it's exposing this augmented reality display. There's no data on here, it's just showing that this is the first time we got it all working. So that's actually coming from the little board you can see in the top right. And over here you can see the board there as well, so that was the first time we actually got it working. was a lot of problems we had and trying to get a web server running on that does that make sense no if everyone just doesn't see anything i'm going to assume that's yes okay right what is ar so this is a wikipedia as well augmented reality is an interactive blah blah blah right so it goes down and then that's quite interesting it says including visual auditory haptic somatosensory and olfactory does anybody know what somatosensory means do you well i didn't so i had to google it so yeah so effectively it's uh you you can say you can have something on your body when you press it and it would cause some other sensation or if something was hot you'd have a pressure sensor or actually on your body so it's relating different things to different senses that's my understanding but we're not going to do that if it's too too complicated we're going to just try and render some data from this little tool so the frameworks we're using uh there's kind of four frameworks that they go to frameworks for augmented reality there's a frame here.js 3ds and aw.js okay so 3ds is really for rendering really complex 3d models uh we're not using that or can i really make it as simple as possible and the frame in the r.js is really to use uh javascript and html to build your your scene and AWE.js is really to get to because it's the kind of GQ area of augmented reality but again, we're only going to use the red ones A-Frame and AR.js in our little demo Is it so far so good? Right, sensors and microcontrollers now I'm this may be where I lose some people right so the sensors and these microcontrollers they all follow a particular protocols so then there's the standard types of interfaces you can google these and find out more about them I'm not going to go into detail I'm going to try and give you a high level overview of one of the protocols that we're using we're using the I squared C or i2c some people call it you can have use SPI and can and there's a whole raft of others these are the main ones and I swear to see is very very standard in all these microcontrollers it's the one that's quite easy to use there's lots of libraries that support this stuff you know how does it work so you don't need to know this so when if anyone looks at the get a repo that we've got and I'll I'll show you that in a minute. We've written some drivers for all these sensors. You don't need to know it, but it's just here, so you've got a basic understanding of how these things are working. So it's a two-wire bus protocol for I2C. You've got a clock and a data line. And your slaves, your sensors, hang off that bus. But how do you get all this kind of working on one wire? So you've got one data line for all those things. so effectively you have a start bit when you pull down the data line and then you have a clock pulse that's recognised as a start and all of the things on the bus will then listen and then the next thing is to explain is once you're in a listening mode every time the clock pulses Because if your data line is low it's a zero and if it's high it's a one. So I've highlighted the second command here, they're all zeros. So every time the clock pulses you're going to register a zero every time. But how do you tell it where it's going and what to do? There's a bit in here called write and read. google it you don't need to know it but the second thing to point out is there's actually two frames the first frame is actually the address where you're going to send it to and the second frame is the data so effectively what you're doing is just you're giving it an address and then you're giving it a register and what to do with that bit of information so you might be saying what's the temperature and it'll send one back or you might be saying set set this motor to on or a light on or it could be anything so that's in a nutshell how it works and if you want to try it out and you want to look at it there's a github repo it's samsung research UK IOT meetup it's quite a big word projects wiki and sensors and you can go there and you can actually see the Python code so let me just quickly so here's the sensor page here for real and you can do this at your leisure if you want you if you want to try this out and buy this kit and try it it's all free it's open source and so there's just an example of using the the driver we wrote for for that particular sensor in actual fact it's based on the code that's in the micro python ready we've written a little wrap around it so we can do things like ask at the maximum temperature that's been it's that has recorded or ask at the minimum temperature and so on so these are drivers and they're all using i2c and if you look inside the code you'll see all this stuff about registers and things and now you know what it's actually doing because you've just learned up so if you go down here there's actually a whole bunch of sensors in here there's the the sense of type they'll tell you it's connected you show you a lot of stuff about controlling the sensors errors you'll get back there's the PIR sensor that I just showed you a second ago how it's connected the different modes it can be set in and then there's you know you can creating a micropython sorry a PIR sensor giving it a pin starting it and then you can give it a call back and it will call the call back when it detects the movement I can't do that now because it's just too difficult to set up but on Saturday we could set one of these up and you can trigger it with your hand and see it working okay IOT Basics, so we're nearly at the time where we can do the demo, but anyway, so IOT Basics on these microcontrollers is all about controlling the pins. So you're basically going to stick a protocol or you're going to control one of these pins by setting it high or low or a service you can connect handlers to it and so on and these things are all done for you there's you don't have to reinvent the wheel and there's lots of tutorials around it so to show you the pins where's my screen so at the back of this board here there's a number of pins and you would connect your sensor with these jumper wires like so so you'd have a ground positive data line and so on so some sensors they're not that complex, some sensors are actually just high and low and I don't have one with me unfortunately but I'll have one tomorrow and you could put on for instance you may use use one to detect a magnet mean set on the door and you just use a pin directly you configure the pen and say this is going to be digital in and out and that would be it so effectively you're writing a Python code and you're actually interacting with the real world so this time you write code and you're driving or you're pulling data and from the real world it's not virtual anymore even though we're going to use a virtual reality display so that's the cool thing about it you're actually doing things for real and again I've just showed you a board here so this is the they don't know that this is the other world of working with IOT so you have your pins connected to a breadboard and you have your cables and these are just connected led lights and effectively you're just putting a pin high to set the light on and off okay so let's just try and get this working so what I'm going to try and do is spin up the micro Python board okay so here's my my repo can everyone see my terminal screens that too small? Shall I make it bigger? Too big. Okay, so I'm using our shell, which is a way to get a serial prompt and look like an X terminal onto the board, right? So this is me talking to the board, and I'm going to get a rebel prompt up, so that is actually on the board. So I'm on the board right now. So, you know, if I do. Yeah, it's just a Python prompt. Right. So I'm going to do an import on the start script. And that's just really dark, isn't it? I really apologize, guys. So here's my Wi-Fi connect. And I'm using my phone as a hotspot. so I'm going to do an import start and then try and connect to my phone's Wi-Fi and then the web server is going to start up that's my mouse broken so it's going to try and connect we had to write a Wi-Fi driver so now the web server we're using is run by a guy called John Claude Boos and but we've added some debugging and there's also a pull request from Damien George to to go as TLS which we're not going to use that today by the way so now it's connected to the IP address and in theory if everything's working we should be able to see that so let's use Chrome So I did this earlier to make sure it was working, so lucky enough we've got the same IP address, so if I refresh that, we should see, so you can see the debug coming out here, we've got a HTTP request come into the web server running on this tiny little board. And then, yeah, so there's the web page. This is just a hello, the thing works. So this tiny little web page is actually running on that tiny little board, 40 millimeters by 30 millimeters. And now we're going to just try and look at the status page. And we can see I refresh that we're getting data from sorry that's the status page we want to go to the sensor page so we're getting data from the sensors right now so it's quite dark in here and the temperature sensor is 24 degrees we don't have the radar and the PIR connected you saw that and we got current time where's the humidity there's a problem there shouldn't be zero humidity should be about 43 so that's so that's just the pages we've written to show the sensor data so you can see the pages so open up urls.py so this is actually, oh it's too dark isn't it? You can't see it. So this looks very much like flask and you've got a decorator here for the page. Can you see that? Oh God. Pardon? Right, anyway, so you can get the idea. It's actually, you've just got a simple decorator at the start to handle that HTTP request, and then we'll classify that as sensor. You're just pulling the data with the drivers we spoke about earlier from those sensors, and I've got print lines coming out there as well, which you would see in the back if I hit the web page. And then there's really ugly code down here. We've actually got the content directly in here. Just for this demo, we're making it really simple so you can see effectively oh that's too dark I apologise, you can see it's just HTML there right and then down here we're just populating the HTML with the sensor data so you can see your current time dot blah blah blah, your PIR dot PR total and so on and that's us populating the web page with the sensor data, but the cool thing is you're on a web server on this tiny little board, that's the cool thing right where do we get to so yeah so the next thing to show you this is the part where well first of all i'm going to show you it not working so if we go back one page and we try and get to the augmented reality page this won't work yeah so what's happening here right so it's not working because and it's a really spurious error message it's not working because WebRTC won't allow it to work. WebRTC won't allow it to work because we're not over HTTPS. So one of the safety things for WebRTC and WebNow is you've got to stop man-in-the-middle attacks and we're about to launch a camera and control something, so we have to make it secure. We can't put this over a normal HTTP session. for this purpose demo I've got nginx running on the local machine which is going to forward everything to the little micropython board right and the only reason I've done that is because if I run the TLS stuff on the micropython board it might you might see some error messages coming up because it's not quite ready yet as I said it's really cutting edge so what I'm going to do is so this is my IP address dot 90 and I want to prove that to you so if I run ifconfig you can see dot 90 is the IP address of my machine and nginx is going to forward all that stuff over to the little microphone board and we're just using NGINX, the HTTPS wrapper. Does that make sense? Has anyone used NGINX before? Yeah, loads of you, right. So we're using NGINX just as the HTTPS for now and again if anyone comes Saturday we can use the TLS portion of this tiny Mac Python board. Okay so if I hit refresh on there you should see data data spew up, or you should have seen data spew up if I had it. No, let's do it again. Yeah, it's too quick. Okay, it did work, honest. Yeah, it worked. Okay, now if I go to the augmented reality page right yeah so what it's done now is it's activated the camera on my my computer now I'm going to try and do this on a phone as well yeah so which will take a few more seconds and dive what five minutes left minutes right so there's me here's my marker and here is the rendered display the reason why it's not working very well is because the the shine from the lights please work can you know God can you see it unfortunately if I go back a bit more maybe and yeah it's not very good is it but if I turn the whole laptop run and put it on the ground you could see it better, there we go so you can, there's the data there it's really unclear, so you've got data coming from the sensor values now, I'm open to suggestions on how we can improve this, maybe we could have icons we could have images yeah, we could do anything now who wants to try this on their phone go on, come on up and I've got 6 minutes left right what's your name? Daniel is it a modern phone? it's an iPhone 8 should I leave? I can go Well, I've now tried an iPhone. Should we try an iPhone? Right, listen, don't worry because I've got a backup phone. Right, so can you come over here? Right, first of all, you're going to have to go into my Wi-Fi address. So, can you use Samsung Test? Is there a Wi-Fi Samsung test you're on? Yeah. Let's test. 1, 2, 3, 4. Yeah. Don't tell anybody. Right, so... Sorry, I need to turn my VPN on. Oh, he's got a VPN on. Okay. right okay now you're connected great so go open up your browser and then go to that IP address and I will try and hold up the camera on this if it works if it doesn't know use the other phone yeah it's come up with a connection this curious okay because we're using our own certs so yeah yeah it works right now go slash AR on the end, capital A, capital R. Allow the camera. Right, so hold it up to the marker, which is over here. Yeah, it works. Right, now I can show this off. Give me two seconds to, this is going to be really difficult, but let's give it a go. So that's, God, Where's your phone? It's backwards, isn't it? So there you go. It's on an iPhone, so you can see the Samsung logo. And you can see the data there as well. Right, so that's how you do it. Thank you very much, Daniel. Thank you. Right. So, we've done our demo, we've got three minutes left. So you can find out more about the web server. The fork I've got is the one with the TLS stuff and you see extensive debugging in there. This one has no debugging. I would really advise you to use the debugging because if something doesn't work, you're just lost. So we did that. We did start the web server. we showed the index page we went to the browser and we got to work on an iphone it's first time ever and you're the first people except for five people in samsung has actually seen this working so so only we only got working last week so so you can put yourself on the back so you're the first people to see this stuff working okay so we showed that and we've kind of looked at the live code already and we showed you the urls.py and showed you it was very much like flask uh and where you have a decorator and your endpoint and there's more there's much more you can do and i didn't go into the augmented reality piece because i'm running out of time and basically this is this is the an augmented reality uh html page and you can see you're just creating uh a torus shape here an animation frame a plane so we had two planes on us we had a plane in the bottom with an image and a plane at the top and we made the top plane perpendicular to the bottom plane and then we gave it a transparency level and we put text on it so the code is in github and it's really simple it's just one page uh there's two bits at the top with a camera static um anchor frame there and the there's a javascript library at the top as well you need so the So you need the AR.js and you need these tags at the top. If you don't have that, and there's pages on there as well, you can see you just get an object rendered in display. You don't get the camera stuff working. So there's only two pieces you need to get the camera working, the AR.js and your tags at the top and the bottom. And you can read all of that on A-Frame. And tutorials on that are really good. but this is a static page the stuff we've got is in urls.py it's the same as that there but it's hideous looking because it's inside triple doc strings you know and the sensor objects we showed you that as well we showed you how you connect the sensor objects you're just calling in, you're importing the sensor drivers at the top and then you're creating a sensor object and then you can call methods on the object and they're all documented in the JavaScript repo as well and I'm running out of time okay so what can you do so this stuff's good for static web pages it's a bit like flask you can you can parse and form data which I didn't do this forks or logging bill and you can use web sockets you can do dynamic substitution with a PI HTML file I didn't do that either and you can use HTTPS but it's only for your own sets at the minute, and Damien George has got to fix that if he wants people to use HTTPS or someone else has. What it's not good for, again, HTTPS isn't quite ready yet. Don't think you're going to hit this with thousands of requests. It's not, and that's not what you do anyway. In real life, if you had an embedded system, you're not expecting thousands of people to hit it. It's going to be just for somebody coming up and, you know, put their phone up. You can imagine this being a fireman might come up and offer up and he's got control to shut down power to a building for instance and that's actually mandated that all of your shops will have a power switch at the top, you'll see whether the fireman can pull it down, wouldn't it be nice if it was an augmented reality one where you could just do that and switch it off so there's a use case you could use for it it's like Django Flask but it's really really, it doesn't have CMS or ORMs or user authentication or plugins and so on, it's very very basic and it's not fully stress tested as you've seen, but it's free and if you want to try this stuff out you can and put it on these little embedded systems and to summarise I'm two minutes over we went through what's IoT, microprocessors micropython, we went through the bus architecture and sensors and a real demo a basic augmented reality page we dived into the hardware and looked at some of the code and again just come on Saturday if you want to get your teeth into this and I'm done, thank you very much for listening Thank you, Nicholas, for a lovely live demo presentation and a live testing. Now you know that it works on iPhone web browsers. Congrats. So does anyone have questions? First of all, thank you for the nice presentation and the demo. And I would like to ask, considering the small size of the board, Do you think this whole thing would be fit for wearable sensors and wearable electronics like a heart sensor I could just put on myself or something like that? Depends. If you're talking about stuff to go on to your body, then that goes through a very rigorous testing regime. So it's got to be in the field for a long time. This isn't. So I guess no for that, but for the form factor on the side, yes. Yeah, I could see that's being used. I mean, one of the projects we're going to look at now is to use automatic speech recognition, but do that on the edge of a network. So the board would then record speech, pass it up into the clouds or the edge, and then get the response from a JSON object back with context, what the board has to do. So if it was an air conditioning unit, for instance, you could put a really low power board in there with a microphone on it and then you'd say switch on the air conditioner to 23 degrees that would go up to the cloud, come back and then the board would do its stuff so it depends again on the thing you're selling, if it's a white goods thing like a washing machine, people aren't going to pay a lot of money for a washing machine or a microwave, so if you're going to put something really meaty on there like a rash of pie or higher then there's going to be a cost attached to that and then you've also got the cost of stuff in the cloud. So if you're pushing things into the cloud who pays for that as well? So it's really complex but I think there's a space for it and I think yeah now the main thing that's different about this is it's been certified to be used as a product. It's not a toy it's not a prototype tool anymore it's you can imagine you build your thing clip that on and it works so an example would be you don't want to build your whole thing because you ship it off and get it mass produced in China then they copy it so you can have the the board's separate. The other board's got all your sensors on it and the brain thing gets plugged in so there's separate pieces. You join them together and then you're protecting your IP that way as well. Does that answer the question? Just one quick question. Is this technology, this server, is it very specific to this MicroPython board or could you also run it on a microbit or some other microcontroller that runs MicroPython? Well I took it from, it was using an ESP board at the start so we had to port it a little bit, change it a bit to get it to run on this board so it did run on ESPs already and you can see on the website, if you go there you'll see the boards it supports so yes it can run on other boards. You'd probably have to make sure sockets is compiled into your binary on the actual board your interpreter. So we had to get sockets compiled in for the MicroPython board and TLS was an issue as well. Logging was an issue. What else? And that was it. Yeah. We have time for one final question. Does anyone have any other questions? Good. Thanks for the lovely presentation. Thank you.