The future of AI training is federated

Speaker 1 [00:07]

It will be a, you guys know about this, it will be a 90 minute tutorial, of course it shouldn't take so long, there are three parts to the tutorial, but part zero is just about getting everyone on the same page about federated learning, what an architecture, a prototypical architecture of federated learning would look like, and then what the components of Flower would be when you deploy it, and then we'll hop into the tutorial, okay? So, yeah, what we believe in is that the future of AI training is federated. And this is a gentle beginner's guide using Flower. My name is Chong Shen. My surname is Ng, if you're curious about it. And, yeah, just to quickly start, can I get a show of hands on how many of you have heard about federated learning? Very nice. And how many of you have actually used federated learning for your work? oh super that's good um i think it's super informative as well uh you know the concept of federated learning has been around for a couple of years um and i think uh we're trying to see how we can encourage the adoption of it as well all right and then just the last thing how many people would want to follow with the tutorial as well to follow to do these components perfect thank you so much cool so i initially listed out the three steps that you uh can uh do to follow along the tutorial. The first one is most important is to check out this GitHub link. Here's the repo, and I'll be working directly out of that GitHub repo as well, so you can see how I do it live. The second and third points is to create an account, the Flower AI. This is for part two of the tutorial, and the reason is because you want to run the Flower Federation on a server that we are hosting at the moment, so that requires a bit of credentials, so it would be good if you can do that, but it's entirely optional. It's not from a federation, I think. And the last part is to install Docker. The last part is I'm going to do a live deployment on each devices, and in fact, if you have Docker installed on your machines, you can actually deploy a federated learning client on your machine, and I will try and demo it live. I also have two Raspberry Pis over here, and fingers crossed, if the stars align, I can get these two Raspberry Pis to also join in the federation, and you can see how it works in a real-life deployment. Okay? Let's get started. All right. So we did the setup. So again, I'm Chong Shen. I'm a research engineer at Flower Labs. I'm one of the core devs in the team, so anything related to the framework itself, feel free to feedback directly to me, and we can also work on that. Feel free to connect as well on LinkedIn, and the GitHub link is for the flower repo. It's completely open source. The merch is at the last table over there. T-shirts, tote bags, socks, and stickers. Feel free to help yourself. Knock yourselves out. Cool. I will start with just a bit of premise. Why federated AI? The data that we know is naturally decentralized, many, many data sources, and AI can't use most of it. The reason is if you want to collect all the data, there's a lot of restrictions, privacy regulations and whatnot, it doesn't solve the problem. The concept of federated AI is simply to push the computations of the learning to the data sources itself, and thereby we work around the requirement to move the data to a single location. So the summary here is that centralized AI can't access most of the data, right, because we need to move the data to the competition. So if we employ federated AI, we're able to move the competition to the data source and it just stays where it originates. We can then train an aggregated model from all the distributed data sources. That's the premise. Okay, that's a quick rundown. So how does federated learning work? This is a very visual diagram just to get everyone on the same page. There are, let's say, five steps involved. Step zero, let's just say you have a central server, this is a prototypical architecture, one server, multiple data sources. You initialize the global model, either with the random weights or a pre-trained model, neural network architecture. You have all your data sources from all the edge devices. Then you have the model weights, W. You send, in step one, you send the model to all the connected organizations or devices. Then, in step two, you train the model on the local data, on each of the organization or device. So there's some fancy equations over here. That simply means you update these model weights based on the local data. In step three, you then send these updated models from all of these devices back to the server. And finally, you aggregate them. The simplest, vanilla way to aggregate the models is to take an average of all the model widths. And that is your classical federated averaging algorithm, that's the core everyone benchmarks against. Then you simply repeat. So that is one round of federated learning. Repeat that as many times as you need to get the converged solutions, and over time, you'll see your loss reduce, your performance increase, et cetera, et cetera. The same thing can be done for federated evaluation. Let's just say you don't want to do any training, but you just want to see how the model performs. From the user perspective, you would send this globally trained model to a server. The server will send the weights to all the participating organizations or clients, just get evaluated. You get some metrics back. The metrics gets aggregated, and then the user can see how the model performs. Federated evaluation is also part of the family. There's also federated analytics. Say you want to do a query against all your distributed data sources, you can send a query against your server. Same thing, it goes to your clients or organizations to get back the statistics, and then you get an aggregated statistics. So all under the same umbrella of federated AI, so to speak. Okay, let's see how we federate with Flour. So what is Flour? So Flour, we call it a platform, right? It's a platform for a unified approach to apply federated learning, analytics, or evaluation. At our core, we are an open-source federated learning framework that you can use to build any of your FL workflows. We have used it for many use cases, large language models, different models, reinforcement learning, also fine-tuning these kind of foundational models as well. All of these are possible. We have documented use cases and examples on our website. can refer to them as well. We have a lot of run times. Now, this is a list, it's a laundry list of things that we have right now, but most importantly, the takeaway is that you can deploy the components of Flour into your use cases in a real distributed setting, and then you can actually train models in a federated way, right? You have the Flour CLI, which you will use today. You have your user authentication, which you'll also use in part two. And then the part about Docker is when you actually deploy it in your on your local machine later on right so this is a component of deployment in addition to this which i won't cover in today's tutorial there are privacy enhancing technologies that you can build or add on top of flower so differential privacy the calculations for circular aggregation homomorphic encryption for those of you are familiar secure multi-party computations and any modifiers or some something like a middleware that can work with the messages around it. All of these are available in the Opus Ons framework and you can actually apply them for your use cases, whatever you need to do with. And right now, Flower has been used by many industries. In finance, we have banking circles. In health, there are blood counts. And in the consumer applications, Thunderbird is also using one of our solutions as well. So what the Flower ecosystem looks like currently is that it's pretty massive. Of course, it works with Python. It works with many of the machine learning frameworks they're familiar with. You have PyTorch, you have TensorFlow, TensorFlow Lite. And then in the broader ecosystem as well, we work with a couple of other large companies, like NVR Intel, and they build specialized apps that integrate with Flower. If you want to just curious to know about this as well, just come and speak to me as well. All right, let's talk about runtimes. I'm putting this slide here just so that you understand the terms that I use during the tutorial itself. So I'll always refer to the Flower code itself as a Flower app. And when you want to test it locally, without deploying it into many edge devices or actual servers, you try it in what we call a simulation runtime. So in the part one of the tutorial, You actually can scale up 10 clients that you train on your local machine. You can scale up to 100, and then you can see how it works. You can also tune the hyperparameters of the federated learning itself and configure it to suit your needs. So once you're happy with that, you can then move to deployment runtime. The deployment runtime also uses the same code that you wrote for the client and the server, and you also use the same set of CLIs to actually execute it. So these, between the Flour Simulation Runtime and Deployment Runtime, the user experience is seamless, and we intentionally made it so. Additionally, there are the third-party runtimes that wrap around this, for example, and their Flare, they also integrate with Flour. They would also use the same Flour app to execute a federated learning workload in their Flare systems. So the key point here is to think about the Simulation Runtime and the Deployment Runtime, and we'll look at that in our tutorial. Right, so I think after this open source development with Flower, I'll jump in more into the technical stuff and then we can do hands-on, right? Open source dev with Flower, I think it's safe to say that AI devs love Flower. We have 5,700 developers. As of yesterday, we have 5,700 stars as well, now GitHub repo, and 2,000 projects that depend on the Flower framework itself. We have, we run annually the Flower AI Summit that's based in London. This was a picture taken from just a couple of weeks back. We had a very nice invitation from an attendee from Thomas Wolfe. So he came and gave a talk as well. So people from research and also industries will just come and give a talk at the Flower Summit and also just to share ideas. And it's really a cool summit to just be at, just to share the applications of related learning as well, what your experiences are. We have Disqus, so you can go to disqus.flower.ai, that's where you can ask questions and we will respond and it's nicely documented. I think it's a pretty cool forum to be in to just discuss things in general about using Flour. We also have a very active Flour Slack, if you have a very quick question, oh, I'm running into this specific issue, just hop on to Slack and me or any one of my team will jump in and try to help you out as well. And finally, we have this Flower Baselines, which is a self-contained project in itself. What we aim to do is to reproduce some of these strategies as published in these papers, and any researcher who wants to try and test out this published mechanism, they can just go to Flower Baselines, clone that repo, and then run it. So it's entirely reproducible. The aim is to make reproducible research and allow people to work and build on it. We are active in the conferences as well. We have our Chief Science Officer and Javier, the Lead Research Scientist, who are in ICLEI right now in Singapore. So if you are there at any of these conferences, feel free to speak to us as well. Okay? Cool. Now we go to a bit more technical part. So breeze through that. We want to build FL apps with Flour. We'll introduce you to the Flour CLI. are the five core Flower CLIs, starting from FSWR new. And that creates a new Flower app for you from a template. And all of this reduces all the boilerplate code you need to write to build a Flower app, and just helps you iterate much quickly from that. So it helps you build with template. The next four commands, run, log, list, and stop, that helps you, exactly as it says, to actually execute your app, view the status of your app itself, and to actually stop the execution. Because once you actually run it, you want to be able to kill it. So this is where FlowerStop is. The beauty of this is that we intentionally made this FlowerCLI to work with both the simulation runtime and the deployment runtime. So you can use the same set of CLIs, you work with it, and then when you want to deploy it into production, then you just use the same set as well. So I think it's pretty cool. So just navigate between simulate and deploy. OK, the first step right now to get everyone warmed up is to run flower simulations on a single machine. So the takeaway here is you should be able to just do flower run, and then you'll see how it works. So this will be tutorial part number one. All right, let's see. In the repo, it's pretty verbose. You will see the four parts here, part one, two, and three. Let me know if you can't see it, OK? Parts one, two, and three. We'll go to part one soon. But the easiest way for you to get started without installing a flower is to just launch it in GitHub Codespaces. You can do that. There's a link on that. I will, for the sake of simplicity, select four cores. here, and then create the code spaces, code space. It should spin up pretty quickly. Yes? I just want to find the name of the repo. Oh, the name of the repo. It is. Yeah, I tried to upload the description to pre-talks and then I put it in the wrong field so it didn't show up in the description, so all good? Yes? Who plays the code space in the resources menu? Pardon me? Who plays the code space in the resources menu? GitHub. Yeah. All right. Did anyone manage to successfully spin this up? Yeah. That's good. Nice. Anyone running into issues with this step? All good. All right, I'll just give you one more minute while I check some things. Is it big enough? Cool. Yes? I have the environment set up. Perfect. Thank you. Okay, everyone's happy? All right, let's move on. Perfect. So step one complete. You have an environment now. So these are the steps to install. Part one is the quick start with the PyTorch template. So the commands that you need is fwr-new, so in here, if I type it here like that, it should be fine, right? Flower new, and then you can provide the app name, let's call it AwesomeApp. Flower username can be anything right now, it doesn't really matter, I will just call it. CSNG, CS Ng, that's me. And then you're given with nine frameworks to work with. So this one, for sake of simplicity, you can install it in your, well, yeah, you can install it in your local machine or in the Codespaces itself. You can select PyTorch, for example. Let's see if that works. And then you say that, okay, this is great the app is created successfully and you're given with some options now you if you have not installed the dependencies oh yeah for those of you who have been paying attention the there is a new folder created here called awesome app and in it inside this folder there are at this directory structure right firstly you have so getting all pretty standard you We have this pyproject.toml that not only contains the project dependencies, but it also has the additional table with tool.flower, and that has some configurations that you can change over here. All right? There's a publisher. That's my username, which I created earlier. There is these two components here for server app and client app that just tells Flower where to look for the server app and the client app modules in the sub-directory. And then there is a table here which contains the configuration of the federal learning itself. So how many server rounds are you going to train, the fraction of the clients that actually train, and also how many local epochs that you want to train per round. Yes? I'm sorry, could you say again which framework you were supposed to choose? I selected PyTorch Feel free to select any others as well, I think, hopefully it should be pretty straightforward. We have the last two tables is about the federations, it's about the name of the federation. The default name of the federation that you built here is called local simulation, and it points to this table over here, sorry, at the bottom. And by default, we specify 10 nodes or 10 clients. So now that's the structure of PyProject Tomo. And then there is some additional code in the subdirectory or the module awesome app. I will just do a one-minute run-through of this, but you can feel free to look at it in your own free time. The first is the task itself. So for those of you who have done deep learning, this would look very familiar, especially for PyTorch. You have your net class. This is very standard. Nothing is different over there. You have a load data function which just loads the data for each of the super nodes based on the number itself. So if there's 10 super nodes, you want to partition, group it into 10 so that each client would train on its own data set. This is just a utility function. Then you have your train function for your PyTorch model, that's pretty standard, your test as well, and then you have two additional functions here called get weights and set weights. And the reason for these two functions here is that under the hood of Flower Framework, all of the model weights are converted to NumPy weights, or NumPy arrays, and then these NumPy arrays will be moved between the server and the client to do aggregation. The reason for this is that we want to make the framework agnostic to any of the tools. So if you have PyTorch, TensorFlow, or any other exotic frameworks, all of them will be converted to NumPy arrays, and then these can be moved around in a very standardized way. That's just the task.py. Now, let's start with the client app. The client app is very short, it defines a server function, and this server function is what's going to execute the aggregation itself. It reads in the number of rounds and fraction fit that is defined in pyproject.tomog. So if you see here, the number of rounds and fraction fit, these are the two variables that you set. With this, the server app will be able to know what the values of the number of rounds and fraction is. And then these values, fraction fit, is passed to the strategy. So how do you combine the model weights from the server side? Then there are other variables over here that retrieves the weights from the neural network architecture, so it converts to numpy weights, and then it converts it into a parameter set that can then be taken in by the averaging strategy called fat average, and then we do the combination over there. I'm glossing over some details over here, but if you want to learn more, you can come chat to me after the tutorial, or check out our documentation as well. Now I'll jump over to the client app. The client app looks, I would say, fairly moderate as well. Just for the sake of simplicity, I will compare the client function with the server function. Here we have a very similar structure to the server app in that there is a function in the client app that will be executed on the client device itself. It has the network architecture. It knows what the ID is so that you can properly split the data set. And then it has a train loader and validation loader. These are just your data loaders so that the data set for each client can be loaded separately. Finally there's this flower client instance. Let's take a look at what is this. The flower client instance is, it wraps around a NumPy client, so this NumPy client has, or the structure of it has three parts, right, it has the, there's the init, and then the fit and evaluate, this is the bare essentials of a client. So what this does is, on the client itself, the, when the client receives the weights from the server, it will do a fit, basically trains the model, right. You get the weights of the trained model, and then that gets returned back to the server. So this is what this fit function does, the fit method does. Then there is also the method for evaluate. So after the model has been sent back from the clients to the server, and that gets aggregated combined, this model is sent back to each client, and then the client will evaluate the model, the aggregated model, on its own local data set as well. So this is the part for evaluation, okay? All right, cool. That's the explanation done. Of course, feel free to ask me later on if you are a bit unclear, all right? Now we have this, we'll go through the next steps. We'll CD to the model itself, or the project itself. We'll pip install it, let's see how long that takes. Cool. Pretty decent. And let's see. How are we doing the local install? Anyone doing some local installations? Hopefully it will be easier. Yeah, it's good, because the Codespace instance has a CUDA attached to it, so it's like, oh, we need CUDA to cost everything, yeah. But if you don't have it, then it just defaults to the PyTorch CPU, because it's much faster. there okay what to execute can I get it this one All right, let's see where we are at, okay, almost there. Well for those of you who have already installed it, you can actually go to the next step which is just do a flower run dot, so you can do this, fllr run dot. So that will actually run the simulation in the environment that you installed the app. So if you have a local installation, you will actually run it and you should see it complete quite fast. If I run it on Codespaces, I should also see it as well. All right. Let's clear this. Flower run dot. All right. So a couple of things happening. loads the project. Great, successful. You will then see a few things. It tells you the round of training. It tells you that the strategy has sampled five clients out of ten and this is consistent with the pyproject.toml because we have ten clients and we have the fraction fit of 0.5, so five out of ten, that's fine. It should also go for three rounds because we've configured the number of server rounds it goes to three. So that should run pretty steadily, oops, and there you have it. So pretty okay, pretty decent. For each round of training, we see that the distributed losses, which means the losses aggregated from each of the clients, are reducing over time. That's a good sign, that is a good indicator. So of course for this toy example, it will vary, so depending on the use case, so you have to tune your hyperparameters for it. So this is the first taste of executing a simulation completely in its own local environment. So under the hood, it uses Ray to spin up multiple processes for each of the client. Yes. Yes. Yes, they distributed back. Yes. Just propagate the gradient and approach the gradient. Excellent question. So the question was, instead of exchanging the weights from the clients to the server, why not exchange the gradients that you use to update the weights themselves? It's absolutely correct. The drawback of that is that you have many more computational steps, steps, which means each round of federated learning, you have to propagate the gradients. But if you propagate only the weights, you can do more than one epoch per round. So you can do ten weights update step for one round and then move the weights back. So hopefully that clarifies your questions. Yeah, I mean, the single line answer for that is for communication efficiency. Before federated learning as it stands right now was popularized, there is the concept of distributed stochastic gradient descent. And that's exactly what you described, to move the gradients back, correct? And that is just very expensive, right? Because you always have to move the gradients in order to get the next model weights passing around. Correct. Correct. Yeah. Instead, if you update the, show this one, this one over here. So what you're suggesting is to, instead of moving the W, you want to move like the gradients that you update, right? Correct. So that is more expensive because you have to do each round. Yes. RoboNet will have seen the startup from Internet. Basically what they do is they accumulate the gradients, then they propagate the accumulated gradients over 500 steps. Send it to, of course, they exchange the grids, and then if you want, that propagates the aberration What I want to say is I think there is no silver bullet and really depends on the use case that you are trying to solve. The vanilla federated learning moves the weights and of course you can have different combinations it depending on use case you just have to explore what works best for you yeah yes why not move the losses you said um in order to get the model updates right you need to either have the value of the gradients itself or the actual updated model yeah I don't think because you need the you also need to multiply it with you know your learning rate as well too yes the weights are aggregated so they're combined under the server side. Can you say again? The server only does aggregation in the vanilla sense. So it takes, if you have five clients, it takes the exact models from all the five clients right and combines them and take the weighted average and then sends it back out. That's the most vanilla way to do it. Because there's no data to train against from the server side yeah yes Does it happen after each epoch, so the models get better for each client during training? Or does it happen sequentially? The training is completed at one time and then goes to the next. Ah, I see, I see, I see. It sounds like what you're asking is sort of asynchronous federated learning, right? Do we have to wait for all of the participating clients to come in to join, to aggregate the weights before you send it out? So the answer is, it depends. So the classical federated learning, if you say, I want five clients, you wait for five clients to send the weights back, and then you aggregate them. So that's sort of a blocking. There are concepts of asynchronous federated learning where you relax that requirement, and then you allow subsets of clients to update them so that you can have concurrent model updates. Yes? Yeah, I would say there is no one answer for that. So it depends on a variety of factors. I think over a long period of time, the models should typically converge, and it really depends on the quality of the data that you have for each of your clients Correct Yeah, correct There are many hyperparameters to tune as well for federated learning How many local epochs per round You could do some variable learning rate type of strategy as well the typical deep learning hyperparameters as well. Yeah, correct, correct. Yeah, yeah, correct, correct. Yeah, there's always a Pareto balance over there, right? Optimized number of rounds. Yes? Yeah, so the question was, how do you tie in data pre-processing for federated learning? right so is it so I will say the answer is yes right and and because you don't always have to do federated learning if you want to do some federated computation such as you want to update the the mean for each of the client and there's a there is a algorithm for that as well how the mean would change you can also do the computations locally on a client so the first step instead of executing a training, you do a pre-processing step on the client first, update the mean, and then you have the updated normalization factors, and then only you start the training process itself. And for that, we have another set of APIs called the message APIs that you can have more flexibility to control what you want to do first. Pre-process first, and then do the normalization, and and then only do the training, so all of that can be chained up, chained together. Yes? When you speak about managing high-tech companies, what is the number of people? I think it depends on your use case. I think it really depends on the strategy that you employ as well. I don't have a number to give you right now. Now, I know that, at least for our baselines, it can go up to 15 type of hyperparameters as well if you want to tune, because you might apply a penalty to clients that have lesser data and things like that. So that is also a tunable parameter. Okay? Cool. Let's crack on. But feel free to also interrupt me as well. I'm happy to run through the tutorial later on with you as well. So that's the first step. There is a run configuration that you can change. So earlier we had, earlier our run is defined as three rounds with a fraction field of 0.5. You can also change that in the flower CLI, right? So you can, so this was, I added this line here, maybe that's obsolete now. You can copy this to change it. So let's just say I want to run one round for the six-word simplicity, and then the fraction fit, instead of 0.5, I want to say I'm going to do 0.2. It will go again. And hopefully now you see that it only samples two out of 10 clients, so that's perfectly fine. So the values that you pass during the run also updates the actual execution. Then you can configure it if you're in the run with many, many clients. For me to do so, I might skip over this for the sake of time. If you increase your number of supernodes to like 200, you can run it as well in a simulation environment. And you can actually see that it actually works pretty well. And it's quite fast because under the hood, all the processes that handle the CPUs optimize their workload. Cool. Let's jump to part two. So part two is about running the simulation runtime on a remote superlink. So before I dive into that, I have just one slide to show, so that's part one. So part two is about running simulations on a remote machine. So you had your local environment installed on your machine and then you do a flower run. But sometimes you don't have access to full resources, oh I need CUDA, I need to run massive simulations and maybe a large data set just to experiment with. So what you can do is actually host what we call a superlink. So a superlink is a name that will refer to our server. So this is architecture. You have the flower CLI that you use to interact with, and then you have this big server component. We have within the server the superlink. It is a persistent component that runs. It accepts the commands that you execute from the Flower CLI, and then it can trigger this fwr-simulation process or sub-process. So under the hood, it takes your Flower run command and then starts a process that computes the simulations. That way, you don't have to worry about what the dependencies are on your local system. You can then have someone host this superlink for you, and then you just point the address to that Superlink and run that. For this case, there is a flower simulation that you can run at pyconte25.flower.ai. It is not a website to navigate, but it is a URL of the server that we host right now for Superlink. So I'll show you how that works. Part two. So in part two, we will jump back into this. I will clear, and I'll make it a bit easier. I will create a new app, this time around I will call it a NumPy app, just to keep things a bit easy. We have a flower username, I'll just call it CSNGS before, and I'll create a NumPy app. It's just a bit easier and faster to run, so you don't have to worry about the speed. Number six, NumPy. Now, in the NumPy app, so I'll cd to NumPy app, clean up some things, close this. So, in the NumPy app, you have the same structure as before. You also have the client app, server app. It's a very familiar structure. We're going to modify some things, right? In the pyproject.toml, you add a new table. So, now this new table will look like this. You can copy this exactly as it is. Copy this. Paste it here, and now you have access to running a flower workload on this address. For those of you who have created a flower account, the first step you need to do is to do a flower login and then you need to oh wait let me clear this so it's easier do flower login and say that i want to run the app in this folder called numpy app and you want to point it to the federation called pi con d so pi con d e 25. so flower login run this app with the dot and then pi con d is to point to the federation called pi con d e 25 this name over here If you do that, you would get a URI that you can click, click, open the link, and then it tells you that you want to access it. You would get a prompt to say key in your flower credentials and log in. I have already done that, so it just says that I have already logged in. I say yes, and then I can close this window, and it says log in successful. all right now i'll clear this again you can then do flower run like before and say that i want to run this app numpy app and i want to run it at the pycon de25 superlink the server flower run boom it's run you don't see anything right now so all you can do is you can look at what the status of the run is. We copy this, we go flower list just to see what the status is. We call this the run ID of the flower run and then you paste it. It's a bit verbose. Say I want to list the run ID of this app with the dot at the validation PyCon DE25. So this was run here, so this is the run ID, the status is completed, it ran for 16 seconds and this is the timestamp that it ran just now. Sorry, can you move the terminal down a little bit? Down, okay, yeah. Okay. So in fact, if you want to see what the runs are on this superlink, you can just do flower ls of this app and say PyCon DE25 and it will list all the runs so I can see a few people here have tried to run something it's pretty cool actually is that is that some of you guys yeah so if you don't do flower login if you don't do flower login you can actually see the error that said you need to log in first Does that work? No? What error does it say? Okay. Ah, okay. Can you do this? flowerlogin. to say the app, and then you point it to the name of the federation, PyCon-D. So the name that you input here, you should do this. Oh, indentation. There we go. It works now? Oh, yes, actually, so the minimum Python that Flower supports is 3.9. Oh, okay. I can take a look later just to resolve that. I'm surprised that you can do Flower Run. Yes? Yeah, it's a very good question. So right now, in this simulation setup that we have, you cannot actually retrieve the weights from the server because everything sits in the superlink. Right now, this guy over here, all the model weights that were aggregated sits down here. So we're currently working on a way that you can actually retrieve the trained model from this. So it's a good observation. Cool. Yes? Yeah. Simulation runtime is you don't have to have many, many like physical servers or physical devices or spin up multiple virtual machines to actually do the federation itself so everything runs in a single self-contained process they can use multiple cpus yeah so it's like parallel competition or self-contained yeah okay uh yeah i see i get a sense that i think we are more or less That's okay, but I'll definitely help any of you who runs into any issue come to speak me afterwards. Hopefully it's something quick to debug or could be a bug on our side as well, just want to be sure. All right? Cool. So this is what you did earlier. You did a flower login.pycon.de25, and then you did a flower run.pycon.de25 as well. The stream flag is just so that you can stream the logs, okay? I can actually do this again so I go if I copy this entire code and I run this here I will see it stream to your console so earlier it was just hidden by default because the superlink will just run it you can always query the logs from it if you want so for example I can grab the run ID from here let's copy it you can control C safely you won't it runs it stops quite quickly I'll do this again I will grab the run ID I can control C you will still run it's perfectly fine I can check the flower log FLWR log followed by the run ID followed by the app say I want to look at the logs for this app that I execute at the Federation PyCon DE25 and then you show me everything so this is what exist, these are the logs for that run at the superlink, all right. Cool, so we did that, we did flower run, we did flower list, and then we did the flower log as well. So this is how you can interact and run it on a self-contained server that has all the resources that you need. Yes? I want to print a specific amount of logs, not all the logs, so can you control it? Yeah, yeah, yeah, you can absolutely control it. Like, what kind of control do you want to do? Can you show the base for the scope test? Yep. For example, if you scroll up, I don't want to print this PID, it will still take four bytes. Yeah, yeah, yeah. So this is a very specific issue because we installed the CUDA dependencies here, and And then the array will complain for all the process IDs if something is not installed properly. Usually, for example, if you run it on a local machine and you install the PyTorch CPU version and you run it, you should not get all these warnings, hopefully. But if you do, we're also trying to... You can also set the array log-level warnings if you want to. Yeah, we can do that. And then hopefully, a couple of months down the line, we also want to create another back-end for the simulation so it doesn't use Ray, something that is a bit more robust, it doesn't give so much more. Okay. How are we? Hopefully okay. Ray is a way of running any of your AI workloads in a distributed manner. So yeah, there's Ray on GitHub, you can check it out. If you do any parallel computation, this is one of the sort of go-tos as well. Currently, for the simulation runtime, it uses Ray. But we want to try to move things away from it. We are waiting for... There's a lot of things as well. Python 3.13 support is not supported by Ray yet. It would be cool if in the next release they do support it, and then we can also widen our Python support. All right. Let's move on to part three. I think we're okay for time. Yeah, I think we're okay for time. Cool. So where are we? All right. So we were here. We ran flower simulations on a remote machine. You ran it on a URL for pycon-d25.flower.ai. Feel free to try it out on your own time. We're going to disable that server in about 24 to 48 hours just because of resources, stuff like that. But right now, for the rest of the conference, it's a great chance to just give it a shot, an experiment. Also, let me know on Slack if there's any issues as well. Yeah. Yeah. . It is a, you want to say regular cloud resource? Yeah, sure. It's deployed using Kubernetes, and we have some Helm charts that help to orchestrate that as well. So. . Not yet. Not yet. It's something that we're working on. So trying to gauge the interest of the community to use Kubernetes for deployment. All right, that was tutorial part two. Let's go to tutorial part three. Now we go into the deployment systems. So to deploy Flour on distributed systems, the architecture changes slightly. You still have the Flour CLI. you still have the server components. Now, in the client components, there are some additional flower components to run. So, we have the superlink as before from the server. Now, on the client side, we have the supernodes. Supernodes are also a persistent process that runs on a client side and its function is to receive and send messages to and from the superlink. Okay? So what this means is, let's say you have a physical device, you have some data on it, you want to train it, you can either launch a SuperNode in on the device, and then that can handle the execution of your client code itself, or you can have the SuperNode run in a very slim container somewhere else, and have only the client app running on the edge device itself. Supernode and the Superlinks are these long-living components of Flour that can be deployed in a very slim way, in a very slim fashion on different resources. So just to recap, we have the server app process that runs on the server side that aggregates the model weights and we have these client apps that also sends and receives messages from the Supernode and the client app's purpose is to actually do the training on the client data itself, okay? This is what we call the subprocess mode. Each dashed green line indicates that the superlink will spin up a server app in its own subprocess and then it handles everything for you. Similarly for the client, the supernode will spin up the client app in the subprocess and do it for you. If you want complete isolation between the superlink and supernode with the app you would have the process mode i wouldn't actually show the demo for that but what happens is in a process mode you can deploy the superlink and supernodes in in using alpine based images it's hyper slim it doesn't do any computation apart from moving data to and from the server apps and the client apps okay so to deploy it we do in three steps you have the pip install flower which you would already done either on your local machine or in a code space you would run the super link in docker container in this case i've already started one or i'll start one soon i need to check and there's some additional arguments just to configure your ports and the volumes for your certificates etc and then finally you start the supernode containers containers at the data source itself. So on the clients. Alright so these three steps. So let's jump to tutorial part 3. So tutorial part 3, let's clean up some things. Let me first, I have a a virtual machine that I just want to run a superlink. Oop. Broken pipe. I have I don't need a NumPy app. Really, you can look at run.sh. What I did is to run the superlink with the latest image that we have, 1.18, which we just released yesterday. And there is some additional arguments for the certificates as well. So these lines are necessary, like that. Okay? So this is what I ran, run.sh. You don't have to run a superlink for the demo, so this is just to show you that what I'm actually running under the hood is Docker, and you can see that right now it's started, it's deployment engine, and then it's just waiting on some clients to connect to it. So let's try to connect some clients. All right. So now this part is where I will need a bit of luck from the networking gods, and I'm going to try. So I have a Raspberry Pi 5 pre-configured with the super node container. What I want to do is when I connect this to the power source, it should hopefully start up, connect to the Wi-Fi, spin up the container, and connect to the super link. So all of these are the sequence of steps I wanted to do. Let's give it a shot. That's bold. That is bold. Yeah. All right. I'm going to do some magic. Let me see. Oh, what would be cool is this, actually. I'll show you the terminal. Oh, someone has joined. Did someone run the Docker command? Did you? Yeah. Nice. Very nice. All right. I can't see the... You can't see this guy join them. I'm just kidding. Alright, I can't see the, you can't see this guy join them, I'm just kidding, alright let's give it a shot, boop, so this is cutting out, no hotspot, I'm going to get this to connect to my hotspot, so currently there is one super node connected, so someone's machine is connected right now, hopefully this will connect in about one minute, you will see a different node ID connect to it, and I have two Raspberry Pis so let's try connecting two. It's a bit of a theater. It's chaotic and it will be glorious. All right I know that it's connected to my Wi-Fi so it's a matter of the docker spinning up properly and then connecting to the superlink. Yes, hopefully it's somewhere you guys connect, no? If it's not then it's this guy over here. That's one, so now I have two supernodes connected, and then hopefully this one will start up soon as well in about one minute. So while this guy, do I see three numbers over here? 991320 nice three, very nice. It's a green light? I think it's a green light. So for you guys actually run it you can grab this command over here okay you need to download the certificates folder onto your local machine that you have they can run docker for me I have I have this folder over here I have the certificate folder which contains the certificates so this is the same set that I have above okay and then I'm going to do want to make sure dock is running yes good and now execute this so if I run this it started and if I go here I should see a number of nodes connected one two three four five maybe there's another one so So, any other two just joined? So, yes? Yeah, you basically just see two lines right now. Well, to actually test that, what I'm going to do is to run a federation. So, this is the world's first PyCon DE federation. Pretty cool. I'm going to run it against this. Okay, let's do it. So, I'm going to do a... On my local machine, I have Flower pre-installed. I'm going to do flower new like we have before, I'm going to try a PyConvEFL, I'm going to call it CSNG, I'm going to run it as a NumPy configuration, this is just what I'm doing and on my machine, I can see the PyCon DE clear, and then I am going to edit a couple of lines in this code, actually, why I can do it here, just so that you can see it. So, this is my local VS code that I just created, PyCon DE FL. This is the template PyProject Automatic that is generated for you, and I'm going to add a couple of lines here. So firstly, I'm going to add, where is it, oh no, here, sorry, over here. So this is the Superlink IP address, because I'm going to run it, you can run it as well actually if you like, but I'm going to run it bare bones coding, so let's go to flower federations, pycon.de.toot, address equals to this. Yeah, yeah, OK. I'm going to go, what do I need? Oh, yes, root certificates. It goes to the certificates folder over here, Certificates.ca.crt. All right, so now I have this guy over here, PyCon D-Etude. I'm gonna go PyCon D-EFL, clear, and I'm gonna do flower run for this app and I'm going to run it as PyCon D2. If all goes well, nope, what's wrong? One second, let me check. I have, what's this, address, root certificate, no, oops, I'll debug that, hang on, I will get it to work, one moment, as with live demos, stuff will get broken, let me see, is this Is this guy still running? Yes, it is. Address, root certificates. Let me see, did I do this? I got TLS. I got 91, 99, 49, 58. Do you see on those who have participated, do you see the, nice, I also see it on my side, there's a missing port, that's right, yes, cool, so yeah, so this is how you configure it in a deployment settings. So two Raspberry Pi is configured with a number of Docker containers running as well on your local machines. This is super awesome. Thank you for also joining as well. If it's only one round, hopefully there'll be more. I think it's just communications as well. Let's see. Receive nine results and one failure. Okay. I think it's also I think if you sort of kill the super node halfway while the training goes and then it's not like something is wrong as well this is because we are we specify a pretty strict requirement when we actually executed it that's perfectly fine right so if I do flower list PyCon D EFL sorry PyCon D tut tut you see that this guy is running so I can actually grab this ID, flower stop, I can stop the run, 77, I'll stop this app and I call it pycon.de etude and I'll list it again, it should be stopped, okay, you shouldn't see anything happening as well. If I go into the client app, I can say print hello PyCon DE, okay I can relax the, sorry I'm going a bit fast over here, I can relax, okay let's go three rounds, that's fine. If I do flower run for the current app and do pycon de sorry pycon de tute again you see that executes once more and in your in your app you see a hello pycon de appearing as well so this is how I can configure the client apps so the summary here is that as a flower as a user who wants to execute a flower workload you can change the code on your machine the client apps and the server apps and when you execute it right that code is packaged sent to the super link and then the client app code is distributed to the super nodes so everything that you see right now the changes of the code element over here and it runs on your in your system okay so the beauty of that is with the same set of flower CLI's you can interact with the simulation runtime and execute a deployment engine, deployment runtime, and you can also do your experiments in that way, do any changes over here, execute it, and then see that change is propagated. You don't have to copy your code to the client side and then start something manually, but everything is driven from the CLI via the superlink. Cool. Are we feeling okay so far? hopefully, it got a bit chaotic towards the end I do not understand, the other black screen is your local computer or what is it? Oh, this black screen here? Yeah, this is actually on my local machine so the one over here at the top I ran a docker container just to join the federation, the bottom here I'm not doing anything with it, in this additional tab, I'm connected to my superlink there's a virtual machine that runs superlink, so I'm just showing you what the Superlink logs are looking like right now. You simulated one node in the federation? Yeah, well, I ran an actual node of the federation on my machine. And then you have a couple over there and two from the Raspberry Pis. And the server is actually your GitHub of all this? The server is, no actually, the server is actually another virtual machine which I spun up and its entire purpose is just to start one docker container for the superlink that's it so what I did was in the architecture over here so this docker container for the superlink runs in this virtual machine that I have and then each green box for the client is like one one over here over here one on my machine and then a few from the attendees over here okay all right good I think that is for tutorial part number three pretty okay so far so what's next just to wrap everything up for you to ask me any questions we'll keep it a bit more open and flexible so what's next if you want to dive a bit more into federated learning and actually use the code as well. You can check out our deeplearning.ai short courses, we actually have a very short intro, I think it takes about four to six hours in one sitting, you can run through it, that's pretty cool, you can do it as well. There's also the Flower Monthly, so every month we invite users and researchers of Flower as well as their applications, actually doesn't have to be users of Flower, anyone who does federated learning jump on a call and then we actually record the call and just share it. So if you have some interesting use cases about privacy enhancing technologies or in a distributed learning mechanism, you want to share it with us as well, you can feel free to reach out to me, and then we can organize you into the flower month, please. Yeah, so the path from version 1.7 to yesterday's version 1.18 release is quite a lot. There's so many features to describe. We didn't run through everything, but what I mentioned in my talk today, in the talk today, is the FAB delivery, which is exactly what I described. You package the code that I tweaked on my machine and that gets delivered to the superlink and the supernodes and this gets executed. So any changes that you made over there can be reflected downstream in your process. We also talked about isolated execution. While I didn't show this, isolated execution means you can just split the process of the superlink and server app so that superlink is just a super minimal image you can make it like you know read access only you cannot write anything to it and similarly for the super node and the client app there is node authentication which I didn't touch but it's part 4 of the tutorial so if you want to deploy on your local machine you know feel free to just check out part 4 try it out so what this does is to ensure that each connected super node and a super link has a certified authentication so you cannot just simply join the federation like what you did earlier there is no node authentication so actually anyone who has access to the docker commands can actually participate but if you have no authentication you just fix and you have a handshake between the super nodes and the super link and then finally we there's a bit about custom message handlers so any information that is exchange between the superlink and the supernotes, we call it messages. You can control how you want to handle the message, so something like a middleware, you want to alter the behavior slightly or you want to measure something about the messages, the size of the message or the timing between the message deliveries, you can do that with custom message handlers. Yeah, that wraps set up okay so that's that's flower uh ecosystem is pretty big um i'm super open to any of your input as well um any suggestions i'm curious to know what kind of applications you want to build uh with the flower cli i show you the flower run and flower new um the flower apps of the superlink and supernotes and then you know check us out at flower.ai uh if you want to learn more Yeah, it's a very valid case, so I can talk about two projects. So the healthcare example that I described earlier, there is a project that wants to build computer vision models that does retina scans, so they classify it for early onset of diseases. So each hospital has their own image data set, it's huge. You can't share it for privacy reasons, you have patient records and whatnot. So this is where federated learning comes in. You can engage a number of hospitals, they say 20 hospitals, each of them have their own data set for this use case. And then with federated learning, the local models get trained on that data set. You can aggregate them and the global model eventually, hopefully, will give you a meaningful model that can run inference on this retinal scans, this one use case as well. Yes? In this use case, can you see the data of the hospitals? No, you cannot. So the data always remains on the site, and this is the beauty of federated learning. The caveat is if you want to do these model inversion attacks and all the other type of privacy attacks, it's never 100%. It's always possible if you know what to do. In addition to federated learning, you can add privacy-enhancing technologies like differential privacy, secure multi-party computations, and homomorphic encryptions, and this improves your level of privacy and reduces your attack vectors. So a lot of the work that Catherine Jammu was talking about as well, model inversion attacks, all of this comes into play. There's always this balance. Yes? Yes? Is it possible for a client to monitor which communication takes part between the split links or not? Yeah, you can actually do that. Right now, for each app, each Flow app that you do, there's a hash of that app. And that hash is, you can't change it, right? So if you change one line of code or you alter one character, that hash will be different. The supernode or the client itself will be able to know what app is running, so you can see it in the logs. So you can use it as a mechanism to authorize the execution of specific apps for training. Yeah, you have a question? Yeah. At this point, does the differential privacy apply Applied at the client side or later centrally? Both are possible. So depending on your differential privacy mechanisms, there's a client side and there's a server side. And then depending on the use case, you can combine both. And how would you track privacy growth budgets if 20 hospitals have their own differential privacy implemented and also maybe on the server side? How would you give out, like, this is our current epsilon? Yeah, absolutely. It's a very good question. The Kavitra is, firstly, I'm not a privacy expert. Yeah, yeah. But it's a very valid question. So I think one way to do it, it's not the, I guess there are many ways to do it. The one way to do it is when you actually do the, when you actually try the FL system itself, you know, you build one version of it, you need to have three things. Firstly, you need to try to apply the differential privacy mechanisms. Secondly, you need to compute your budget, and then thirdly, you need to make sure all the parties that are involved are accepting of that level of privacy. And it's an iterative process, that's all I can say. So while you do it once, you have to assess that, and as and when new data comes in, you might want to do some continual learning or change the models, and all of that needs to be reassessed. So there'll be a flywheel, there'll be a process that keeps running to get that to work, to get all the parties involved to agree on it yeah yeah yeah I guess yeah I guess that's what that's one way of how we could work yeah no worries yes yeah no Is there a way to monitor which clients are getting meaningful updates? Yeah, absolutely. There are various ways to do it. I think one of the ways is to employ a very specific strategy. There are strategies out there which look for outliers in the model weights. So either on the full picture of the models or from each of the nodes you will have different weights. You can clip the weights that are outliers and that helps you mitigate data poisoning attacks or any of these attacks effectively. So yeah, there are mechanisms for that. That's the bottom line. Yes? How to clean up, ah okay, in the bottom left corner, you should see this blue thing, you click it, you can just stop current code space and then give it 10 seconds or so, everything will shut down. yes and there's a question back yes yeah yeah yeah yeah absolutely yeah absolutely the first level of control is via, I guess it's in the awesome app over here, it's in the fraction fit, this is the first level of control and then you can further relax that depending on a use case and then there are also strategies which specifically just either, yeah you can configure it, I think this is the first hyper parameter that you want to tune basically for it yeah yes yeah yeah it's a it's a very very good question in fact the first time when federated learning is coined or the term is coined right was when if anybody remembers the Google keyboard or the G board is it called it was popularized in that era so the federated learning was coined by a researcher Brandon McMahon from Google for federated learning and it's specifically for the next word prediction problem so yeah Android devices will have that and then there are companies right now they actually do that as well Yeah. Correct. Yeah. Yeah. Yeah. Correct. Correct. So in this setting, right, it's a little bit different to the healthcare setting. So healthcare settings is like what you call cross-silo because you have different entities that participate and everyone needs to agree on the data standards. Whereas for the Google approach, it's more like cross-device. sort of own the software in some way right so they can control which device would participate when what this means is that the criteria for the device to participate in training would be depending on your time zone you'll be at night it has to be plugged in connected to wi-fi or has enough battery and then you would join the training itself yeah yeah That's basically it really, that wraps up my tutorial. Feel free to have a chat with me later on if you're curious to learn more, I can help you fix your flower login issues, and if you want to grab more merch, feel free to do so. Thank you very much.