Battle of Pipelines - who will win python orchestration in 2022?

Speaker 1 [00:02]

A warm welcome to my talk about Python orchestration. I will pretty much give a high-level overview, so not go too much into detail into everything very specifically, but hopefully you get out knowing what you will choose as your next orchestration tool, maybe. So, who am I? I'm Janus Grönberg, Machine Learning Scientist and Infrastructure Architect at Kineo AI. We are pretty much a small startup. and yeah I studied computational engineering at the TU Berlin and sort of after some time around in the car domain I found my way to the specific machine learning domain and if you want to connect to me or I want to tell me hey your talk sucked then feel free to do so here's my email and my LinkedIn yeah just happy to connect so I told you that I work at Kineo and AI and what we do is pretty much we provide customized AI solutions for our clients. So basically we work a lot in different projects, in different domains, and we always try to find some sort of value that the customers that we have sort of could potentially have with that data. And obviously a lot of that also focuses on the infrastructure. So we really heavily invest time to figure out what would be the right needs for our clients. And yeah, all in all, we just help as a strategic AI partner for different organizations, also in a cultural domain. And that brings me to my key introduction question, why do we need orchestration? And for that, I would love to take you on a journey, on the typical AI project journey here at Kineo. So imagine being me, and in a long time ago, in a galaxy far, far away, there comes a customer that says, hey, I want to have an algorithm that reduces CO2 emission worldwide by 90% and increases happiness. Oh, and don't forget, it should be sexy. So the natural solution, the obvious answer, nothing easier than that, right? And already in your mind, you're already like, I'm going to do so much work. You're thinking about all the different things. You think maybe this problem is too hard, this is too insane. But yeah, you start coding, you get data, you visualize it, You're already in your Python domain, Jupyter Notebook. You bring in all the heavy machine learning libraries sort of to help you. And then at some point, you're finally at the point where you say, hey, it's working. I can say something. I can go back to the customer and present them my solution. So you have the pitch. You're like, hey, 100% accuracy, no more CO2, baby. Happiness increased by 5,000%. Oh, and as a side bonus, we even solved world hunger. And you're like, bam, success, right? And then the customer's like, yeah, but what do I do now? And that's pretty much what this talk is about, figuring out how we can get from data science to pipeline orchestration. So basically automation in the background, having our algorithms that we deployed and the models to be run. And as you can see on the right-hand side, We have already a lot of frameworks, different things that we could use. And today, maybe you feel like, where do I even start? So that's why I'm here, and I will focus on these three tools, Kubeflow, Perfect, and Airflow. And when you think about the question of what to choose in an orchestration tool, there may be some considerations that you have to take. Obviously, these are not all the considerations that you have to take, but I just listed some of them here. So origin, popularity, deployment, costs, and the capabilities and limitations of these tools. So where do we come from? Basically, this should be a timeline, and at some point we had a lot of orchestration already made possible, but we had this in a very complicated way. YAML files, configuration, all over the place. And then in 2014, Apache Airflow came around and was most likely probably one of the first really big orchestration tools that fully supported Python integration. So really getting the data scientists, machine learning experts back then, into the flow of orchestration and data management. And as I said already, it was developed by Airbnb. It's focused on Python. And just as a small introduction, Airflow is based on so-called direct acycle graphs. and the way they do it is they use operators so operators are pretty much already predefined yeah you can imagine it as being a Python class and then you can sort of use them to do whatever you want with airflow so time progressed and then a 2018 perfect came around they said hey we see a lot of issues with airflow and we can do it better so they developed and launched tool that you can it also open source and yeah they focused on a lot of different aspects so they wanted to increase the performance of the scheduler i will come back to that later but also on yeah having more than just a running state and a finished state and yeah they tried to improve things and we'll see if they successfully did and then at the same time cubeflow came around or officially launched by Google. They in theory wanted to make their TensorFlow library more accessible, connect with Kubernetes and all these kinds of things. That's why they launched Kubeflow and since then I would say it is the main tool for machine learning. This is one thing that you have to keep in mind when considering Kubeflow. It's all about machine learning. It's not just data pipelines. So here we can see some comparison between the three frameworks. So all of them pretty much focus on Python, and Airflow and Perfect are pretty much only written in Python. But Kubeflow really took a different approach. They have a lot of JSON type script and, yeah, a very small portion of Python, which at least I thought was very interesting. And another key thing to look at here is that Airflow has like 2,000 plus contributors, which is obviously really nice if you want to improve your setting and help with problems. Here we can see the star history over time from GitHub. I copied this graph. I don't know if it's super accurate, but you can see that pretty much Airflow consistently increased in popularity. Then 2018, Kubeflow came around perfect, was launched a little bit earlier. Maybe they on this graph only have the Python official GitHub repository which they linked. You can see that Airflow pretty much outranked the other two but also had a lot more time gathering stars. And here, just for your information, I plotted the main Python pip packages from PyPyStats. And it's really, really insane if you see how much downloads actually Apache Airflow has. So it even has almost two times to three times the amount of downloads per month than Perfect and Kubeflow combined. And this is something that you can't really leave out when you think about an orchestration tool. So if you want to start with something that you have no clue on, then it's really, really good if it's widely used, if there are a lot of bugs already found, support is high, and stuff like this so coming to the deployment of Apache Airflow you have like a pyramid and at the top you have the basic installation that's like the installation on your computer that you could do you can do it on source from source you can do it via a pip package you can do it via docker or you can use help charts if you want to connect it to kubernetes but that is all nice obviously you don't want to have your laptop be the production solution for your for your company so you actually have to think about deploying that to the cloud. And for that, you need to have sort of some things in mind, like a database backend, you have to have local executors for single machines, Kubernetes and Celery or Celery for multi-node setup. And then again, you can come with the approach of using a production image or a Helm shot for Kubernetes. And the underlying background or the underlying foundation for that, and this is most likely with Prefect and Kubeflow the same, you need to have a back-end engineer. If you want to do it on your own, it's really hard to get into the mindset. It takes a full-time job, most likely two to three. You have to think about access management, configuration, logging, and scheduling. And it's just not so easy to deploy it in production. If you feel like, hey, that's too much to me, production deployment. I don't have a server. I don't have the capabilities. I just want to have a native cloud solution, you can fall back to either one of these three. I think there's a fourth one. I forgot the name. But, yeah, it's kind of interesting to see that Google and Amazon, for example, offer this service, but Microsoft with Azure doesn't. Yeah. And in the end, if you successfully deployed your Airflow environment, you see something like this. So it's basically a list of your DAGs. Hopefully you can see my mouse. Yeah. You can turn them on and off. And that means sort of that they will be scheduled eventually at some time period that you predefined. You have your runs here, like the successful ones, the running ones. There are no failed ones right now, which is super unrealistic, trust me. But yeah, just to keep it in mind. And basically, it's a very, I would say, not too primitive, but a very basic layout on what you have. So we move back from Airflow and come to Prefect. And with Prefect, it's even more simple. So you have the pip install. It's only focused on Python. It's insane. You can just pip install perfect and all the extras that you want to have and you're good to go. One of the workflows would look something like this. This is from their homepage. And if you run it, you pretty much have it done in three to four minutes. Super insane, super fast. And something that really caught my eyes when I first looked at perfect is if you look at it, it just looks like a Python script right so you have these imports task and flow and you just pretty much have a function that you say hey this is a task with a flow run this task and that's it and it's and it's very very basic and very out of the box so again you think about deployment how can we sort of run that in in production and you have two options so you can fall back onto the perfect cloud which already has everything set up you you create an account and you have actually 20,000 runs for free every month. And then you just get an API key and can connect to it. And if you want to do the server, your own hosting approach, then, again, there are Docker images, Helm charts. But then you also have to think about these four items. But they actually come in the Docker image. So it's not too complicated to set it up. And, yeah, you just say, hey, perfect backend server, and you're good to go. again we have cloud platforms so what perfect does it schedules your tasks and it orchestrates them but it doesn't actually run them right so you also want to have machines that you want to execute your tasks on for example if you have huge data loads task if you want to have machine learning models on kubernetes you need that in the cloud or on the server but just as an option you have these three main cloud offers, and there's a lot of integration already possible. So you can just go to their website, download whatever they want to give you, and you can connect it pretty much very easily to your backend. And then this is the architecture of what it would look like. So on the left-hand side, you have pretty much your laptop, so you can run something, you can locally develop, you can do everything that you want, and then it automatically connects to the API, in this case the perfect cloud and already shows in the dashboard but also if you have some execution environments in the cloud for example then you can these then you can connect these to the to the cloud as well and it's really really nice to have something locally on your computer that just runs you can develop on your local machine without having having to upload something to the cloud every time for it to be executed and then again the dashboard from perfect i think this This is the dashboard 1.0 version. They came out with a second one. We're planning to release it. But yeah, you can see a lot more details to it. So you have like some nicer graphs. I don't know if user experience is very important to you. But at least for me, it's something that just goes along. You have different information just overall over your workflows. Coming to Kubeflow. So, and Kubeflow, I think what they had in mind when they started the project is if you could do everything, that would be great. So, as I said before, Kubeflow is really about machine learning, and it's really about trying to have these machine learning data science algorithms run in production and orchestrated. So, it's not only about data flow and triggering small processes, but it's really about, yeah, your machine learning images. And the key question to Kubeflow that I want you to take is, if you want to implement Kubeflow, do we want to run it on-premise? And it's not just simply saying, hey, yeah, that would be easy, but you really have to think about, hey, do we even have the capabilities and resources to do that? Because it is a lot of implementation and it is a lot of maintainers. If you choose to go the routes or the left route to host it on premise. You have to have GPU support, Ubuntu, Kubernetes. You have to think of container native networks that give the possibility for the pods to communicate with each other. You have to have different types of storage choices. So you want to have different pods accessing different storages at the same time, read, write them, many, many, many different services communicating with each other. And it's not that easy. But again, we can fall back to a packaged Kubeflow distribution. In this case, I chose exemplary the Google Cloud, and then it is a little bit more basic. So you just have to create a Google Cloud project, authentication client. You have to deploy a management cluster and the Kubeflow cluster, and then you're good to go. If you're whole, like I told you that Kubeflow is about everything pretty much, If you only want to work on pipelines with Kubeflow, then maybe you can consider Vertex IA from the Google Cloud. So it's a managed service for Kubeflow, but the pipelines only. And then you have this nice looking dashboard as well. And you can already see on the side, there are a lot more options that are not just about orchestration. So you could have notebooks, you can have tensorboards, you can have like options that visualize actually your training accuracy, stuff like this. You can have hyperparameter search for your machine learning models and all of that out of the box. And, yeah, as I said, full, full workbench of machine learning tools. And if you felt like me when I created these slides, too much information, bro, let's try to narrow that down a little bit. So all tools have extensive guides online for implementation. I would say Prefect is the most easiest. Airflow is a little bit more complicated to implement on your own, and then comes Kubeflow. If you want to host any framework on-premise, you most likely need a dedicated infrastructure architect. So it's not something that you as a data scientist can take on at the same time while developing actually the algorithms or whatever you want to develop in your company. But you really have to have a dedicated person or, even better, a dedicated team to that. But again, out-of-the-box cloud solutions are available, and Perfect enables easy local setup that can be connected to the cloud. So that's like a really big bonus. You can just run your pipelines on your local computer without much effort, and you're already good to go. Kubeflow and Perfect come with native out-of-the-box scalability capabilities, meaning that it automatically sort of creates the need or orchestrates the need for more memory in that case. And then with Airflow, you have one instance that handles scheduling and computing. And that can lead to potential memory issues. And I want to talk about that also a little bit later. But in theory, Airflow provides you with a lot of options to connect to different services. So this is not something that is highly relevant. So, for example, we at Kineo, when we help a customer with Airflow, we most likely take a very small instance for Airflow that does the scheduling and all the heavy computing is then outsourced to different services, for example, on the cloud. So what about commercial integrations? Show me the money. Airflow, Amazon, and Google Cloud pretty much are the same, around about $80 to $100 for just the basic service. But there are additional costs. So if you want to monitor that, store the logs for these files, you have to connect them. This is not included in the official pricing guideline. and obviously if you want to have heavy data flow between services this is not covered in the clouds as well. For Perfect they offer you like the 20,000 free runs solution that I already spoke about. In this example I added 20,000 successful runs so that would amount to $50 and yeah the price goes down as increase as usage increases. And then, yeah, again, that's not all computing environments for your actual jobs, storage, data flow, all of that has to be taken into consideration. The cost for Kubeflow is the cost for Kubernetes. So if you run or host a Kubernetes cluster already, that would be great. So I gave you an example for AWS. You can have these EC2 instances, T3 Xlarge for basic operation, I would take roundabout two, and that would amount to roughly $90 per month with continuous uptime. And what you can do in these cases, you can combine other applications from your company to run on the same cluster, or even better, shut down when you don't need the cluster and recreate it and spawn it. Capabilities and limitations. I see that I don't have too much time, so I will go through it a lot quicker. So Airflow's open source community is unmatched. and this is something I really, really want to heavily focus on. It's just insane. You find a bug, you go online, you find a little forum called Stack Overflow, and someone somehow fixed it already, so it's really great. You have a lot of flexibility, a lot of different operators. You can run one big Airflow server or multiple small ones, and you have the option of moderate scalability, both up and down, so with Kubernetes executors in this case. What I don't like about Airflow, it doesn't have out-of-the-box data quality checks. You don't have versioning for your files, for your pipelines. Windows users can't use it locally unless they run Docker. Airflow scheduler is a huge problem. So basically you have one system that does the scheduling and computing and everything in its own and if something breaks, take time, make a coffee, you will need it. But advanced features like authentication, parallelization are not sort of very easily accessible. And again, the working states, you only have the failed running and success. That's very limited. If you choose to go to Airflow, make sure you host your code on Git, deploy it from there. Limit the access to the files that can be deployed at least to your production system. Test your code. I mean, this is fairly obvious. And then consider maybe having multiple smaller servers. So you have the support for multiple independent users and availability requirements. Coming to Prefect, very easy setup, versioning is there, workflow is Python at its core, so you write your functions, you put a task line over it, and you're pretty much good to go. You have a lot more different workflow states, you have a scheduler that only focuses on scheduling and then the task, the flows itself, handle their own tasks. So that's something to consider if you're very, very focused on minimal latency. You have data flow out of the box. You can test it locally. And you have a really, really nice user interface. What's not so nice, the support. So it hasn't been long enough around. You can still sometimes see bugs where there are no guides to it. Most of the time you can post that to the Slack channel. But yeah, all in all, it's very hard to debug sometimes. You get sometimes error messages that are a bit vague and for the cloud you have to pay but that's with pretty much everything. If you want to run Kubeflow, as I said, Kubeflow wants to be more, not just orchestration. So you have model deployment, notebook server, hyperparameter search, all these things. You modularize your steps most likely. So if you write one thing for a pipeline, you can use it in another pipeline very easily. You have upcoming open source services that are already integrated. for example the feature store feast and you have a tight integration with Kubernetes with which makes requesting more at CPU GPU as part of the pipelines very easy and then limitations no local executions sometimes debugging individual steps is tedious install can be tricky base running requirements are high and some of the integrated services complicate the setup and now to the final question what should I choose I mean most likely you already noticed it I'm not anymore the biggest advocate of Airflow. I would just say, hey, if you want to go for an easy setup, a beautiful user interface, and a very well-designed workflow, choose Perfect. It's very easy. You can go into it. You can get it done pretty quickly, be done with it in two to three days, and you're already good to go. If you have the resources for extensive integration, maintenance, and all these kinds of things, choose Kubeflow. It's really good if you are in the machine learning domain and want to go from there and yeah with that the choice is yours and thank you for

Speaker 2 [23:02]

Okay, thank you for your talk. We have a lot of questions. First one, do you have experience or thoughts on self-hosted versus managed airflow, for example, astronomer?

Speaker 1 [23:12]

actually no I have to pass on that so I'm not the super most detailed expert in all of these services and especially with astronomer I don't have

Speaker 2 [23:25]

Any experience? Okay. Next one. Have you worked with Dexa? How does it compare to a prefect or airflow? That's

Speaker 1 [23:32]

That's a that's a really good question. When I started this this presentation. I already thought of including it I at least again, I didn't work with it. So I kind of have to pass but what I can say is that online at least you find a lot of positive comments about Dexter and Yeah, I would say most likely you have to read up on your own but

Speaker 2 [23:56]

Okay, next one. We are currently considering prefact, but also Argo. How do you think they compare?

Speaker 1 [24:03]

Good question. Again, I have to pass. I'm sorry.

Speaker 2 [24:08]

Ah yeah, this one, you talked about the different pipelines, however, what exactly do these pipelines do? How is it different from running your own microservices?

Speaker 1 [24:21]

So having your own microservices is nice. I would say it's a different kind of feel that you want to get out of it. So if you have microservices, you have to really think about how they communicate. You have to set that up on your own. You have to implement them side by side and then make sure that you have the connection in place. And with an orchestration tool, you have that pretty much out of the box. You have that under one hood. I mean, microservices, you can already implement them on your Kubernetes cluster as well and then orchestrate them with a tool like that. I think any of them would be a good choice. I think for continuous uptime microservices, like if microservices require continuous uptime, then maybe it's something that you can't super efficiently handle with an orchestration tool, so you just deploy your microservice and then it runs. But if you're talking about pipelines and flow and machine learning model calculation, that should be triggered then orchestration tools.

Speaker 2 [25:19]

Okay, thank you. You said that you need a dedicated architect to set it up, but how much effort is it to maintain?

Speaker 1 [25:27]

That's a good question. So I would say all of these tools, at least they try to sound that they're very easy to maintain. But if you get into it and have your extensive, like pretty minor detailed needs, then it becomes very complicated very quickly. So, with Kubeflow, it's just, I would say you need at least two to three architects that work on it if you have, like, a moderate size company. With Airflow, it's a little bit easier, but the thing about Airflow is you have this sort of predefined setup. So, they try to get you into the Python world, but they still have these operators that I was talking about. And connecting them and making sure they work sometimes is a bit tedious. and for perfect I would say it's most likely the most easiest way so you have your yeah pretty much Python functions you can test them like you would do with any other Python function so that should be something that should not be too hard to maintain

Speaker 2 [26:28]

Okay, then next one, does Prefect support submitting jobs on Spark or Dask? Or does it only run Python functions?

Speaker 1 [26:35]

functions no actually you can run desk jobs on perfect for spark I don't know but most likely yes but desk absolutely

Speaker 2 [26:47]

Okay, can Prefect be used independently of the cloud? Yep. That's another one.

Speaker 1 [26:54]

I mean, so you can set up your own server, it's all open source, sorry if I just answered that too quickly, but yeah, you have the tools available online, you can download the Docker images for your server and you can set it up locally.

Speaker 2 [27:11]

If it's an easy answer, then it's completely fine. Okay, there's another one on Argo. This you already answered, I think. Yeah, and this one, I think you already said something about how does one run a non-Python task with Prefract? For example, spawn a Spark job in a managed cloud service.

Speaker 1 [27:30]

So I would just refer to their documentation to be honest. I don't know what the exact needs are that you specifically need, but you can run pretty much a lot of things already out of the box provided by them.

Speaker 2 [27:46]

Another one, can Kubeflow work with sensors and streaming?

Speaker 1 [27:53]

Yes, I would say yes, haven't done that, to be honest.

Speaker 2 [27:55]

Okay, great. Any other questions from the audience? Can you repeat the question?

Speaker 1 [28:22]

Okay, so what I understood, for example, with Airflow, you have a lot of possibilities to have different services, and they need to communicate with each other. And the question is if you can do that with Perfect. So coming back to the Airflow topic, one of the big issues with Airflow is that the operators itself, so the steps in your pipeline, it's actually really hard to share information between them. So you have a metadata database in the background, and what you do most, or a lot of developers do, is actually use this metadata database to actually let functions share different data types, so to speak. So they would pickle it and put it in the metadata database, which would be a security risk and can just shut your system down. So it's really hard to have something that communicates. And with Perfect, I don't know if this was the question, but with Perfect, you pretty much have the Python functions. So you give one function your input and the other one the output that is the input to the first one. And yeah, Perfect handles that for you.

Speaker 2 [29:29]

Okay, one last question how does prefix data quality check work or does it need something like great expectations?

Speaker 1 [29:38]

So great expectation, at least for my work, is a great tool. If you want to use it, especially, for example, in Kubeflow, we also use a lot of great expectations. I highly encourage it. I would say because we use that a lot, we didn't fall back to the solution of data quality checks imperfect yet, but at least they try to make it sound really, really easy and super fun.

Speaker 2 [30:02]

Okay, thank you for your talk and I'll see you around.