Quiet on Set: Building an On-Air Sign with Open Source Technologies

Speaker 1 [00:08]

Yes, so my name is Danica Fine. I am a developer advocate with Snowflake, but I'm exclusively focusing on open source technologies in my role. So for Snowflake, I'm mostly focusing on Apache Iceberg and Apache Polaris, two things that we will hear about today. So if you're curious to learn more there. But in a past life, I was a developer advocate around Apache Kafka and Apache Flink. So those technologies are also near and dear to my heart. Also, as part of my past roles, I really got into IoT projects. I'd never really done them before a couple years ago, and now I am obsessed with bringing as many different hopefully useful projects into my day-to-day life. And so if you've seen any of my other talks, you've probably seen some of my IoT projects. And so, yeah, you might be asking yourself, wow, man, another IoT project. Do we really need another talk on this? Yes. Okay. Why not? And I also have some pretty good reasons. So when I actually go out and build a project for myself, I know a lot of folks who actually build projects for themselves have different ideologies around this, but I want to make sure that the project that I am building actually solves a real problem in my life instead of just inventing a problem to then solve, right? And so the problem that I had was a very real one over the last couple of years, as we're all sort of working from home and we have hybrid roles now and whatever, I take a lot of calls from home. Even though I have an office in my house, my partner, without fail, will find the most inopportune moment and open my office door. I'm either recording something or I'm on an important call, it doesn't matter, somehow he finds a way to come into my office at the worst possible time. And so basically this project was born out of that issue. How do I indicate to him, please don't come into my room at this moment, I know the door is shut, but also, really, pretty please, do not come into the room at this moment. And so, yeah, I figured I could do this with a fun project at home, and because I work with so many different open source projects, I figured this was a good way for me to combine as many of them into one workflow that also mimics what we would see in sort of a production environment as well. Because why not? Let's learn something out of it, too. And so what I envisioned, because there's always a plan, right? What I wanted was a way to, and hopefully you can see this, to connect what was going on in my computer. And in this case, this meant camera activity, right? When I'm actually recording something or Zoom meeting activity, I wanted to connect that with some sort of indicator light on my Raspberry Pi. So I wanted to build an on-air sign. And ideally, I'm capturing this information from my computer in as close to real time as possible so that we can then feed that light in as real time as possible. And so there's a lot of different technologies that you can use to put in between here to actually serve as that messaging layer. I'm going to choose Apache Kafka because I like using it and I feel like it works well with a lot of production environments. And so I'm going to feed the activity from my computer into some Kafka topics and then we're going to use that to power the on-air sign. And so I mentioned that there are a couple of different technologies that we can toss into this piece of the project. The reason that I really wanted to choose Kafka here, even though there's some other friendlier ones in the IoT space like MQTT or other messaging queues, but the reason I chose Kafka is really for the persistence of the data. Because I don't want to just build this for one specific use case, I want to challenge myself to build out as many other components to this project as possible. And having that additional layer of persistence, even if it's only for a little bit, is really going to help me do that to unlock some additional use cases. So knowing that, I can also set up some Kafka consumers to then feed some stream processing. And I can do as many other things as I want with this. But to take it even further for longer-term analytics, I'm going to off-board the data from Kafka through Flink to Apache Iceberg so that I can then keep this data as long as I actually want to, and then conduct historical analysis on some of my usage data, all right? So this is the overall goal. We'll see how much of it we actually stick to, because that's the reality. Oh, and yeah, we'll drive this analytics. So first things first, and what you probably came here for, actually, is to see what I built and what I ruined in the meantime. So let's build the actual physical system here. And so I've only done a handful of IoT projects, but I think given what I've worked on, this was probably the simplest one that I've put together, all right? And I'm going to pretend that all of you are here so that you can build this exact thing in your own home. So I'm going to try to give you as many details as possible, but if I don't cover something that you have questions about, please ask them later. So first up, most important thing, Raspberry Pi. If you read the docs on a lot of these components that we'll get into, you can technically do this with an Arduino, I'm not the right person to ask about that, so you will have to deal with that on your own. And this is what's going to serve as our actual indicator light. I bought a 64 by 64 pixel LED matrix. This is just the vibe that I wanted for the look and feel of this. This might feel a little rudimentary, but I also have plans to 3D print some interesting components to mount this later on. So this aligned with the aesthetic that I wanted for the project. And so finally, it might seem like, all right, that's enough, but a 64 by 64 matrix is actually a pretty big component to power and deal with on your own, so to interface with that, we need a matrix hat to sit between the Raspberry Pi and the LED matrix itself, and this is just going to make it a lot easier for us to power the individual lights. This list, not exhaustive. If any of you have done projects on your own, you know that. You always forget something, and then you have to wait two more weeks to order it. There's definitely a bunch of extra bits in the way of, you know, extra cables and connectors. Many of them are pictured in the center with the LED matrix itself. And I feel like when I was looking into this project and buying the individual components, this one was a lot easier in that if you looked at the LED matrix itself, it had all of the additional components linked to it on that website, on like the three sites that I checked. So if you're using a retailer like Adafruit or DigiKey, I'm not really sure what you have available to you here, it should be pretty simple to get all the pieces you need, okay? And so what this means is that as far as IoT projects go, this one is pretty simple to put together if you ignore one fact, and that is that you need a soldering iron. What is that, like 20 euro you can get one, I guess, probably? You should have one. to build some things, you should have one. Just don't lick your fingers afterwards, like lead, that's kind of a bad thing. But you do need to solder a couple of things on this. There's actually just one connector that you actually have to solder. I tried to get around it, but you do have to solder it. And then while I had the soldering iron out, I soldered all the other connections just to practice, right? Because who doesn't want to practice their soldering? And so, yeah. So besides that, all very simple to put together, here's proof that I actually did the thing, so this is my very, very messy workstation after I soldered everything together. And so everything's plugged in, everything's connected, and this is the very stressful part of the project where you put all the physical pieces together and you try to see if it all works, and that's, for me, that's the most stressful part, because when you enter, when you exit the software stage of things and you enter hardware, then suddenly debugging isn't just looking through your code. Debugging is, did I solder that thing correctly? Did I plug that in all the way? And usually the answer is no, and then you have to go over with a microscope to figure things out. So thankfully with this, the actual LED matrix came with an associated repository where it has a driver that you can install, and in there, there's some really handy demos that you can run on the command line. And so I didn't actually make this beautiful gradient square, but this was just a demo that I could run to make sure that, hey, it's not a hardware problem, it's not a software problem, we are good, everything looks fine. And so I was very thankful for that. And so from there, then I could actually install this on AirSign. We haven't done anything interesting with it yet, but I wanted to put it where I wanted it to be. And sorry to disappoint you, but I just renovated my house, and so I haven't yet decided which wall I want to ruin to hang this up. So for right now, this lives on a bookshelf inside of my office, which you'll probably say defeats the purpose of an on-air sign. You would be right, but I have some other cool things that I want to build out to make it useful while I decide exactly where to mount this. So we can still make use of this data, I can still solve the problem that I have, and it looks really cool on my calls. So I think that's what we're all after here when we build a project, really, is less for it to be functional and more to impress our friends. So yeah, that's where it is for now, full transparency, still working on it. Okay. So the physical sign is there, it should work, right? We ran the demo, you know, I just have to actually put what I want on that sign. So let's actually power it by writing some data to Kafka. And so first things first, I did mention that we're going to be using Kafka for the real-time component here, but before I get too far into it, how many of you have used Kafka before? How many of you have never heard of it? Besides Franz. Okay. Cool. This is for you. So the easiest description of Kafka is that it is a distributed event streaming platform. And so what this means is that we're going to take events, in this case, just things that happen, so our camera data, our Zoom data, and we're going to produce that data into specific partitions of a data set, which is a Kafka topic. This data is then going to be ordered by timestamps. And so while the data is stored in these Kafka topics, any number of downstream components and consumers can come in and consume that data without affecting any other of those consumers. So rather than, you might be thinking of a messaging queue or some other queuing technology, rather than this data just disappearing after it's consumed, it's there. It's persisted there for some amount of time so that we can build as many downstream applications on it that we want. So essentially Kafka lets us build out a decoupled pub-sub system, but it also lets us build out systems that can take advantage of the fact that, more likely than not, you're going to have multiple partitions in a single Kafka topic, so we can parallelize the processing of those partitions with our downstream consumers. And then that's how we can scale these out to a production-level system, if we'd like to. So Kafka, it's a distributed system. You have to run a cluster somewhere. This is going to feel like a little bit of a hand-wavy bit, because I'm doing it at home. Where do I run a Kafka cluster? You can run it on Docker containers on your laptop if you want. You don't really need so much of the redundancy in your system at that point. It's just an at-home thing. You can also run it on a couple of Raspberry Pis. I have not done that. It's a wish list of mine, but I've seen other people who have done it. But you can also use sort of a trial account on either Confluent or build it out on Amazon if you'd like. You have options. But once you have your Kafka cluster running, then I could start thinking about, okay, what is the data that I actually want to produce as part of this system? So the first thing is that we have our camera activity. So camera activity is pretty cool because not only do I want to know if my camera is on or not, like, hey, am I recording something? But it also could be interesting from an analytic standpoint to know what application is actually using my camera. Is it Zoom that's currently using that camera? Is it some other recording software? Is it Google Meet? What application am I using to launch the camera in that instance? So we'll accommodate that here. And then from there, we also have our Zoom usage. I know I just mentioned a bunch of different applications here, like Teams and Google Meet and what have you. For me personally, 99% of my work calls happen over Zoom. And so if I was going to specifically try to capture information, I figured, all right, If I focus on Zoom, that's probably good enough for an analytics standpoint. So we're going to do that. And so this status is whether or not I'm currently actively in a Zoom meeting or not. So for each of these data sets, these are both going to have their own corresponding Kafka topic that I will create for the time being just with one partition each. And I'm going to set the retention time here to be one week. As soon as Friday 5 p.m. hits, I don't need this data anymore. We can move on with our life. But I do have the flexibility as far as configurations go to change this later on if I want the data for longer or shorter. And usually that's going to come down to, you know, how much do I want to pay to have this data in Kafka. So with these topics created then, now we can start to capture the data and actually write to those topics. So for the camera events, I had a lot of fun trying to figure out how to capture camera activity on my laptop, especially since not only do I have the built-in camera, I also I have a couple other cameras that I use for different recording use cases at home. And so it wasn't enough to just track, okay, the FaceTime HD camera or whatever it's called, the built-in one, but I wanted the solution to be able to pick up usage on any camera that I have plugged into my computer. And so in the end, I found that we could leverage the built-in macOS log stream. How many of you have heard of that? A handful of you. Not enough. It's really cool, okay? So if you haven't used this before, Mac has a built-in unified logging system. And so with log stream, this command, you can actually feed in a predicate to filter all of the logs that are running on your Mac that are being collected at any given time. And in this case, I found out that the phrase AV capture session, blah, blah, blah, blah, comes up in the log every time we change a camera. And you only change a camera when you're turning on a camera or turning off a camera. So basically I could take this command and then I could easily write a Python script to execute this using subprocess and have it be a long running subprocess and then capture that output and then I could extract the relevant information from those log lines, use that to build up the object according to that schema that we wrote and then produce that into Kafka. So with this running, basically, I'm getting everything. Every time I turn on a camera, turn off a camera, I get that information. And it also captures the application as well. I think this one was a Zoom capture. Now for Zoom events. And what's cool here is that because we're capturing this from our logs, this is a fun distinction I want to make, is that these are actual events. That log line is only written when something has actually occurred on my system, and the fact that I'm getting that immediately from the log stream, this is a true event-based system. We are capturing those as it happens. They're being pushed to me. I say that because with the Zoom data, I haven't found an elegant way to actually capture these events. They're not really events. I couldn't find a way to have Zoom push the status to me every time it changes, right? When I'm in a meeting, tell me that and let me move on with my life. And so instead, I had to be a little proactive and actually fetch the Zoom status, which was kind of annoying. This is a bit of a hack, but hacks work, so here we are. So I found out that as an analog, as a proxy, I could check how many ports Zoom currently had open. And that'll tell me if I'm gonna call or not. So similar than before, we have this command that I could run just checking how many ports Zoom has open. And so I could take this command and similarly run that as subprocess every second or so and extract the current status for Zoom. So if it's 0, it means that, yeah, Zoom could be launched. Doesn't matter, but we're not in a call. Anything other than 0 says that we are actively on a Zoom call. And so I could take that and similarly build up an object according to that schema and write that into Kafka. And so again, this is kind of annoying because it isn't truly event-based, but when we're checking this every second or so, we kind of get close enough, like it doesn't really matter. And also, I want to note that even though I am checking every one second if something has happened, I'm only producing an event to Kafka when the status has changed, right? So every second I'm checking, but if the status is the same, if we're not in a Zoom meeting like we were one second ago, I'm not going to send any data, I'm only going to send data if the Zoom meeting has ended at that point. So we're getting as close to event-based as we possibly can. If you can find an elegant solution, let me know, but that's what I figured out so far. So this next bit, it's not glamorous, but I want to let you know because this is how we make it real, guys. So we have the scripts that can actually capture these events for us, but now I need to make these long livings. So this involved actually scheduling these tasks on my MacBook so that every time I log in, I don't have to do anything manual to start them up. They're just there. They're running. Okay? So it wasn't that bad. You can take whatever script you have written. In this case, it is my camera monitor.py, my Zoom monitor.py. And I can create a plist file, properties list file, and we point to that script there. So a couple things to note. You will have to set the correct permissions on the file, right? It's not going to be you executing this script, so you have to make it executable across your computer. You'll also need to change any relative paths to absolute paths. So in my case, I had some config files that were relative paths and some log files as well. So they all have to be absolute. And so within the plist, you see about three-quarters of the way down, the last line, actually, keep alive. So that specifies that, okay, once this thing is up and running, we're going to keep it running forever, okay? So the plist file needs to be stored in the launch agent's directory, and then you'll have to load that explicitly using the launch control command, and then you also use launch control to start that for the first time, okay? And then from there, every time you log in, these scripts will just be running. It's great. So with this running, these are some of the events that I was actually able to capture, extract from Kafka. So yeah, it works. It does the thing. That's wonderful. You'll also notice that I've given a key to each of these events, which is just work Mac. I have a few different laptops floating around, so I figured this would also be very useful to have tied to, I don't know, if I'm taking a call on my personal laptop or what have you. So I figured I would just add in the key related to the specific computer that I'm referring to. And we can build up a bigger picture of analytics later on. So at this point, I've written the script that is for my computer to monitor the camera events, extract the Zoom activity, we can produce that into Kafka. Now all that remains is to actually consume it, drive the sign, and move on. So let's do that. So we're going to write yet another Python script, this time with two Kafka consumers. And each of these are going to consume from the Zoom activity and the camera activity topics. And within each iteration of a while loop, we're going to consume information from those topics and try to extract a new message. From there, I return a Boolean is on if, you know, based on that status, we should have the on error sign illuminated, okay? So we have two functions doing this, one for the camera consumer and one for the Zoom consumer. And then from there, I can combine those two Booleans to let me know if the light should be on or not, because if I'm just recording something, I want the light to be on. If I'm just on a Zoom call without my camera on, I still want the light to be on, yeah? And so missing from this also is a bit of boilerplate code for the LED matrix itself, so setting it up, creating a canvas, preparing it to display actual text. But once I check the Boolean, then I can display on error sign if I need to, otherwise we go ahead and display something else, like hello world, that seems like a nice thing to put up there. So something to keep in mind with this LED monitor is that there's definitely a learning curve, right? If you worked with GUIs before, I don't know, I feel like it has a really rich library of things to work with. Here you get three fonts, they are all the same, and you cannot create your own, otherwise you're working pixel by pixel, and it's very frustrating. So you take what you can get, you can draw some text, you can even draw lines if you want. It's really great. You can display images, but then you have to get the resolution right. Definitely a learning curve to get used to, but it's a pretty fun little tool to use. This is the thing. It did the thing. Okay? I know this isn't a live demo. Thank you so much. But wait, there's more. So yeah, so satisfying to actually have this work the first couple times. As soon as I showed it to my partner, he's like, well, people are only ever going to see the on-air sign because that's what's on when I'm on a Zoom call, so it doesn't even matter what's in there. But I'm like, still, it's fun. It looks cool. I could take a picture of it. So anyway. It works. Isn't that great? So we have a working model now. We're collecting the camera and meeting information. We're actually driving the sign. This is the basics of what I wanted to accomplish. But can we do more? The answer is always yes. Let's throw more open source at it, because why not? Okay. So here's a reminder of the plan. We've done these bits. That's great. But we can make it more interesting, right? So if you haven't dealt with Kafka before, Kafka is great for a lot of things, especially when we need to build event-driven reactive systems like we just did. But the reality is, if you're looking at the Zoom data and the camera data that I'm collecting to drive this sign, the TTL on that data is, like, really low, okay? It doesn't matter if I accidentally miss an event, because it only matters for the duration of my Zoom call, right? It really isn't that important otherwise. But just because that data isn't useful beyond powering this sign in that moment doesn't mean that it isn't useful at all, right? It's data. We're bothering to collect it, so why not find something else to do with it? And so I want to aggregate this data and push it into Iceberg tables so that I can look into it a little bit more. You might be asking yourselves, why? Why bother, okay? There's always a reason. I can always justify what I do. And so we're going to dive into the wonderful world of analyzing things that I do on a regular basis. How many of you use Google Calendar? Yeah, everybody. Now how many of you are familiar with the wonderful little feature, and I say wonderful, but it's a very stressful feature that they have, called Time Insights? Have you ever seen that? Some of you have looked at it. Some of you are frightened by what you do. It's a love-hate relationship for me because I started a new job in November, can you tell? Yeah, anyway. Mine is kind of sad, anyway. So the time insights is really, really cool. It looks at your calendar, it looks at the types of meetings that you're in, gives you this wonderful little summary of all the time that you're spending in meetings and a breakdown of the different types you're in. And so I figure, okay, I'm already spending time capturing the time I'm spending on these calls so let's recreate this feature for myself I know it's already been done but we're gonna do it better because as a bonus I have the camera data too so I'm not just getting details on the meetings itself now with the camera data I can look into well how many times am I lazy and I have my camera off or you know how long am i spending actually recording stuff rather than just being on a call So my goal was then to rebuild this tool, see how far I can get with it. So as I said before, for analytics viewpoint, I'm going to use Apache Iceberg for this. So we're going to take a quick step back and let you know, all right, what is Iceberg? Why should you care about it? And I am a developer advocate for Iceberg, so this is going to be a little longer section because you should leave knowing what Iceberg is. So Iceberg is an open table format that was purpose-built to handle large-scale data sets in a data lake. So Iceberg isn't a database in and of itself. It is instead a specification on how you should organize files in a data lake in your object storage to make it a little more performant, to give you some of the features that you want and like about data warehouses. So it does this by adding an extra layer of metadata on top of those Parquet files that you're storing anyway, and then you get some cool features out of it. So Iceberg, you can kind of think of it as the love child of a data lake and a data warehouse. It's a data lakehouse. Ooh, marketing. So you get the convenience and flexibility of data lakes with the structure of a data warehouse and the governance of a data warehouse. So Iceberg is also great because we're able to write our data once in this table format. And Iceberg as a technology, as an open data format, is really championing interoperability. We write this data once in one format, and then a number of other tools that are Iceberg compatible, you can then come in and use that for different use cases. You're not just tied to one vendor. And so really to drive this home and justify it here, I think Iceberg is a really great add-on to a real-time system, just because it gives you, really in most cases, a cheaper way to maintain that data long-term for actual analytics beyond where you need it for real time. Yeah, so once you're done with the real-time component, off-board it to a cheaper system, and then you can do some more interesting stuff with it, okay? So as a quick aside, Iceberg's architecture, because it really is metadata, so I just want to highlight that really quickly. So on the right here, we see the data files that we actually want to store from the get-go. These are all par K. And then from there those data files are tracked by individual manifest files. And the manifest files are tracking a subset of those data files for the data in each of the columns that tracks the min and max of that so that we can filter off irrelevant data files earlier on. We also have manifest lists from there that track a subset of manifest files. So all those column level statistics kind of roll up into the manifest list and we also have details on how we're partitioning that data, so we know the mins and max of our partitioning columns as well. And then those roll up to a metadata file. So the metadata file will keep track of all of the snapshots of this table as it's evolved over time, and also maintain details on, like, the schema as it's evolved, and more of our partitioning spec. And so each of these metadata files will have potentially multiple snapshots that it's tracking over time. And then the most important thing here to make it feel a little bit more like a database is the catalog. So the catalog is a way for us to track the individual tables and say for that table identifier, here is the latest metadata file with the latest snapshot so that we then can access it. So none of this is based on a true directory structure, which is good, because if you worked with data lakes, you can have some limitations on what's actually stored within an individual directory. And, yeah, so we're able to track all these data files with this metadata structure. And so there are a ton of different catalog options that you can use, but the catalog that we're going to choose today is Apache Polaris. It's an open source version of an Iceberg catalog. So we saw at a high level what Iceberg kind of is, the components, the metadata layer, and that catalog. But Iceberg is just a format for how to organize your data. That doesn't mean it stops there. It gets more interesting. As I said, the good thing about Iceberg is that it's interoperable, right? A number of tools are Iceberg compatible, so you have flexibility to choose what you actually want to use to get the right tool for the right job. So conveniently, the Iceberg project maintains a couple implementations of compute engines that you can use to interact with your Iceberg files, your Iceberg tables. They have built-in support for Apache Flink and Apache Spark. Both of those are really good options right out of the box to just get started. But the ecosystem of Iceberg tools is healthy, it's growing, so you'll see a number of vendors and other open source tools vying for Iceberg's attention and really championing interoperability here. So you can use one of these tools one day and use another one the other day. It doesn't matter. interact with the same Iceberg tables at the same time. So that's why Iceberg. We should all be convinced that that seems like a good next step. We should take our data from Kafka, throw it in Iceberg, because then I have flexibility to use whatever I want later on with that data. So you already saw a glimpse into what tool we're actually going to use for this. But in the process of moving data from Kafka to Iceberg, you have options on which tools you can actually use. We're going to go with Flink, but I wanted to give you a quick summary of what other things you could use within this space. Really, you can choose between Spark, Flink, and Kafka Connect. Those are the three main options to really get started with moving Kafka data in an event-based fashion from Kafka into Iceberg. Again, we're going to use Flink. I highlighted a couple of the pros and cons here. So if you were trying to do this at a production level, you can kind of ask yourself the right questions here. All right? But, again, we are going to use Flink. What is Flink? How many of you have used Flink before? Like two. Wow. This is great. Everybody is going to leave knowing so much about Flink and love it. So Flink, pretty straightforward, it is a distributed processing framework. What's cool is that it's really, really flexible in what you could use it for. It's flexible for both batch processing and stream processing. Flink is pretty much best friends with Kafka. I think the Kafka source and sync are the most popular ones for Flink, but it also has the flexibility to connect with a bunch of other data sources and syncs. It also has a bunch of different levels of APIs that you can interact with it, including a SQL wrapper. So we're going to use Flink SQL to read the Kafka data and write that data to Iceberg. So for reading from our camera activity topic, so first of all, I launch a Flink SQL shell, just something pretty easy, you just download Flink, and there's like two configurations, it's pretty simple. So we launch a Flink shell, and within that, I'm going to create a camera activity table. And it's very simple, because all I do is create it with a Kafka connector source. So that's happening on line eight. On line 9 and onward, I'm pointing to that specific topic within our Kafka cluster and doing a couple different configurations there to point to schema registry, tell which brokers we're connecting with, and what have you. But one thing I want to call out is on the first couple of lines, this is where we're actually creating the table as you would just generally within SQL, calling out those fields and their types. And one thing I wanted to call out is on line 6, I am actually taking one of the fields that we just brought in, the camera timestamp on line 3, and we're actually going to do a transformation on top of that as we bring this data into this table. So I do this because I want to create a different timestamp on which to partition, and I wanted to have a less granular value there. I wanted to just have the day of that timestamp rather than using it down to the millisecond like I have it stored in that camera timestamp as is. So just an interesting thing that you're allowed to do within Flink. And so we're going to do a similar thing here, again, with the zoom activity topic, creating it just with those same fields that we specified in the schema. And so we've loaded our Kafka data into tables in Flink SQL, but now we have to prepare Flink to connect with S3, which is where I want my Iceberg tables to live long term. And so that starts with defining and, in this case, connecting to our catalog, okay? Catalogs are really, really important to Iceberg because we need a way to programmatically update our pointers and specify, okay, which is the latest metadata file that we need for that Iceberg table, and they're also going to specify and maintain our asset guarantees so that we can have many people interacting with our Iceberg tables at once as well. So we can't function without them, okay? So outside of my FlinkSQL client, I have a Polaris catalog instance running. It's local for now. That will change soon. But I have that running. And so with Polaris, within Polaris, I have established a catalog, I have created a number of roles and privileges, and I have assigned those to myself as a user so that I can actually interact with my catalog and write data to my Iceberg tables. And along with that, I've created some external credentials that I can use outside of Polaris to connect with Polaris. So by running this statement, I'm creating a catalog within Flink, I'm giving Flink access to my Polaris catalog that I have running, and in doing so, by providing my credentials, I'm giving myself access to create those tables through Polaris as iceberg tables. And then from there, I go ahead and I can create the camera activity and zoom activity iceberg tables. So by actually creating these, I specify I'm using that catalog. By creating these tables, these are now appearing as metadata in our iceberg instance, in S3. And so also note here that on line 8 and 7, I'm also partitioning by the partition timestamp. And so that's the timestamp that I kind of trimmed down to have the day. And so when we actually go into S3, we'll see that we've partitioned by that. So any data that I write from those days are going to be within that partition for us. And then from there, once I create the Iceberg tables, I can go ahead and select the data from those incoming Kafka tables that I have in FlinkSQL and push that into the respective Iceberg table for us. And this is just one instance of running a FlinkSQL client, but you can imagine that this is something that I run at the end of every week just to off-board all of the data from Kafka into Iceberg for me. Okay? There we go. This is an S3. Okay? Did the thing. Okay? And so... Yeah. So they're being populated into these partition... I know I said there aren't directories. There are directories. It just doesn't depend on the directory structure. But yeah, all of the data for each of these days are going to be pushed into this partition timestamp for us. Okay? And so I know I only have a few minutes left. I know this isn't a very glamorous way to end because, like, we didn't actually do any interesting analytics. You may have noticed that when I wrote the first version of this talk, this is only from like two weeks ago, three weeks ago, I don't know, how long has it been, whatever, not really long enough for me to get a good look at, like, okay, are there any trends in the things that I've done? Honestly, no, because I took a week and a half off. So not really any interesting analytics to do on this data at this point. So instead of focusing on that, I wanted to show you a glimpse into my backlog. Hooray. What are all the things, because it's not enough to have a backlog at work. We also need one at home. So what else can we do with this data? And also looking for ideas, if anybody has ideas. But the first thing I want to do is make it real, like really real. So right now, outside of the Kafka component, the whole event-driven side, the rest of it is kind of running ad hoc on my laptop. So my first stage would be to productionize Polaris, have that running on S3 as well, or have it running on Amazon as well. And yeah, just productionize this whole thing and also decide what is the timeline over which I want to be running that Flink job to actually extract that data from Kafka and push it into Iceberg. That was that piece. And then from there, it's actually analyzing the data. This will mostly be sort of one-off analysis. Like I've looked into some of the data so far, but I'm really trying to nail down, okay, how sad do I want to be when I look at this data ultimately? What story do I want it to tell? Hopefully the happiest ones, really. But this analysis I think will involve two parts. The first is to really take advantage of Flink and Flink SQL and do a little bit more preprocessing of the data rather than just pushing it straight into Iceberg. So the first thing I want to do is align the two topics' worth of data. So I want to be able to see, okay, if I'm on a Zoom call, am I also, do I have my camera on? Like just kind of smooshing everything together and sort of looking at the windowing of the data and see when those things coincide. And then from there, over on the Iceberg side with whatever tool of my choice, Compute Engine of my choice, I could go in and compute the intervals and just kind of look into it and build up that time insights that we saw on Google Calendar. That's the easy stuff. Now for the stretch goals. So when my partner and I are both working at home, his work space is all the way in the garage, which is like this is a very American thing for me to say, but that's on the other side of the house, okay? That's very far away from where my office is. So rather than have him come in to have to check the on air sign, which currently lives in my room anyway, I want him to I want to create a chat bot that he can send a text to and it will let him know if he should bother me or not, right? Which is something that's pretty easy to do. So I'm very excited to do that so that he doesn't have to walk all the way over to check the status. And then finally, there are other use cases where I'd want to just say, hey, I'm busy. Don't come into the room. Right? I don't need to be on a call. I don't need to be recording something. Sometimes you just need, like, focus time, yeah? Or maybe I'm on a call on my phone. So I want to be able to extend this to another sort of physical system, like some buttons that are on my desktop that I can then push and say, okay, I'm on a call, I'm doing focus time, or, you know, I'm using, I don't know, something else. So yeah, just be able to have that to feed data into the same system and drive the light in the same way. And I'll also be able to capture the same data for the analytics as well. So we have something more exciting to, more to throw in there. So summary, because you need to know what the takeaways are. Just to remind you, I know that this was sort of a fun thing, and I know you're all here for fun and to see what I built and what chaos we got into, but I want to remind everybody that the reason that I really built this was to see what I could do with a bunch of open source technologies and also choose the ones that I felt were most aligned with production-level use cases that I've seen out in the wild. So I know this was a fun thing, and yes, you can recreate it at home to the degree that you want to. But also this is a very real sort of flow that you would actually put together in a lot of different companies for a lot of different use cases. So being aware of how Kafka and stream processing and Iceberg fit together is pretty useful. And so, yeah, so hopefully whether you're taking this home and using it or you're taking it to work and pushing to use some of these technologies, I think they'll come back. And I'm glad that you came along for this journey so you know more about them. And with that, thank you for joining me today. I think we have time for questions. Here is a link to my link tree. If you want to actually see the repository, it's on there. Add things as GitHub issues, that would be great. And yeah. Thanks. Thanks, Annika. So just a quick reminder that we have Slido if you want to ask any questions. And there already are some. So yeah. When do you recommend Lakehouse, e.g., Apache Iceberg, versus Warehouse, e.g., Snowflake table for storing data? Yeah, so I think that the reason that we kind of got into, first of all, the data lake architecture is really for the flexibility and kind of just to get out of some of the limitations of the data warehouse. And it's really around the formatting of your data, I feel. But then we kind of took it a little too far with just a straight data lake. And so bringing it back to the actual lake house itself would be kind of a little bit of a correction from going too far with the lake house itself. So when would you choose Iceberg over just a strict data warehouse? It's really when you ask yourself, like, what sort of formatting did you actually want? And a lot of data warehouses, you're kind of getting into vendor lock-in. So I think that it's more of taking a holistic approach of what you actually need for your system, and a lot of things nowadays, I think, point to a data lake house, because you kind of get the best of both worlds there. Cool. Is there a special reason for using a Raspberry Pi over an ESP826632? God, everybody asks that. It's because I bought the Raspberry Pi already. No, I bought it years ago. And the Raspberry Pi that I got is absolute overkill. It's like 16 gigs. It's like ridiculous. And so I just use it for everything at this point. I have not actually bought that chip yet. But I do have a couple Pi Zeros that I could re-architect things for. On my wish list, I'm just using what I had. All right. And how does Iceberg manage high frequency and high volume data ingestion in regard to data and metadata compaction? Wow, what a great question. So how does Iceberg... Iceberg is pretty performant. When you're asking about high data frequency, like are you asking for the frequency of individual writes to Iceberg or are you asking about the frequency of ingestion? And I know you're not going to answer because the person asked the question already. but Iceberg handles both pretty well okay and so we're gonna get into that metadata in a second but so when you're thinking about streaming data and pushing it into Iceberg which is effectively what's happening here all the tools that I mentioned Spark and Flink and and Kafka Connect have different mechanisms for handling the fact that we're gonna have you know very small amounts of data sent very frequently to Iceberg right we have individual events that are being written you know every couple seconds potentially and so they each in those tools have ways to mitigate the fact that okay maybe we would want to group some of that data together before we actually write it to Iceberg so that's something and of course there's gonna be trade-offs on latency there like and you have to ask yourself how quickly do you want to be able to access it in Iceberg but on that note if you wanted to write a bunch of data you know very frequently to Iceberg basically Iceberg use optimistic concurrency to make sure that we don't have you know overriding other data and so really it's just gonna be a bunch of retries if you have too many kind of in there yeah it's built into the technology but that's you know still can be a problem so we have two small last questions one is when integrating dbt with iceberg are there materialized views and how do they materialize when And integrating what with Iceberg? DBT, data build tool, I guess. Are there materialized views? Not officially. So Iceberg, and there's actually just a talk on this. Where was it? Anyway, there's a talk on this. I forget where I listened to it. But Iceberg effectively uses what it has when it builds additional features into it. And so there is a view feature in Iceberg, and there's a table construct and there's a view construct. And the view itself is really just here is the SQL that we would execute on top of tables. So it's not a true materialized view, but it does kind of have a notion of it. As far as how it particularly integrates with dbt, I don't really have the right answer. Okay. Thanks a lot. We at Flexa also use it quite a lot, Kafka and so on. So I really like this introduction with the practical example. And thanks again for Danika. Yeah. Thank you.