How to build a Python-based Research Cloud Platform from scratch

Speaker 1 [00:04]

Python sucks in the cloud? Isn't that what bullshit bingo consultants are talking about all the time? Well, that's the spirit that I encountered when I first joined Quonium three years ago with the mission to radically modernize our research infrastructure. Good morning, everyone. Today, I'd like to invite you all on a journey that's far from finished and might as well yet never really end. How to build a Python-based cloud research platform from scratch. And before I start, let me add, this is a practical approach with limited resources in the context of a company. We do neither claim scientific correctness nor completeness, and I like to say we leave that for the users of our platform. I am Andre and I'm head of research technology at Quonium, leading a team of eight internal and external data engineers. And I've been working in the financial industry for the last 18 years in various IT roles, but always striving to challenge the status quo. And Quonium is a quantitative asset manager. That means we transform data into forecasts for stocks and other financial instruments. And we focus on SRI, that means social responsible investing, providing sustainable and green products. So you surely can imagine that research platform is at the heart of our value generation process. Okay, so how to build a Python-based cloud research platform from scratch. In the coming roughly 30 minutes I would like to emphasize the following three questions. How to get started, how to deal with the ever feared regulators and users, and how to satisfy everyone's needs with the right tools and services. And to you, my fellow Pythonistas, Python is the obvious choice for doing data science. But three years ago at our company, we had the old guns who knew thus and not much else. And thus, by the way, for those who don't know it, that's an antique programming language that was en vogue in the 90s. You can Google it if you like. And also there were a lot of statisticians and financial mathematicians who had learned R in university. And they really liked it. I mean, R has certainly its use cases. But my first task was to convince everyone to start using Python. And by everyone, I pretty much mean everyone, the data scientists, the data engineers, and even the application developers, because it just makes things so much easier if everyone uses the same programming language. And of course, you can leave everyone to use his or her programming language of choice, but you probably miss out on a lot of synergetic potential and you need to translate scripts from one language into the other, so I would really recommend, if you can achieve it, to convince everyone to use a unified programming language and obviously my choice would be Python for that. So, yeah, there's a conflict there, of course, try to resolve it, maybe someone from upper management can support your claim. And trust me, in the end, and that's my, and I'm convinced of that, Python will get them all. Second task is to decide whether we want to migrate to the cloud or not. And modern data science, of course, needs a lot of storage capacity and computing power, but unless you are a really big team, the demand for that computing power can be quite volatile. So, in our case, it was pretty clear that we did not want to buy a supercomputer or two and put those into our local data centres. So, okay, makes a a lot of sense to rent that computing power from a cloud vendor. And also, cloud vendors or cloud providers do offer other benefits, like managed services. That helps keep down administration costs, of course. And also, some services are just only available in the cloud. The third task is, of course, once you've decided to migrate to the cloud to choose a cloud provider. And of course, you can use multiple cloud providers, but you have to keep in mind that that might increase costs and administration efforts. So I would really recommend to start with one cloud provider. Thing is how to choose one. And options, of course, include the known hyperscalers like AWS, Microsoft Azure, Google Cloud, and also lots of other smaller, often more specialized vendors. So how do we decide which one is best for us? Well, you should ask yourself questions like, are there already any cloud services being used? Do we already have a contract? Or do we have to negotiate a new contract? Because that's something that probably most of you guys and girls would not really enjoy. It can take lots of months to negotiate all those legal things. In my book it's not really fun. So if you have the option to use an existing contract, it might be a wise thing to use that. The other important thing, of course, is, as usual, that you really need to match your functional and non-functional requirements with what the cloud provides. Okay, how to deal with the ever-feared regulators and users. And there are typically two main sources of requirements, regulators and users. And regulatory requirements stem from laws, governments, institutions, and internal policies. And in a lot of cases, they do clash with your user's requirements, and that's pretty common. You need to expect that. For instance, your data scientist might prefer to use a tool that's only available in one cloud, say Amazon SageMaker, but your regulatory requirements might be better served by another cloud. happen to us will probably also happen to you if you start such a project. So let's start with the regulators. Cognam is subject to BaFin rules and BaFin that's the German banking authority, GDPR, general data protection rules of the European Union and also internal policies apply and that leads to a lot of regulatory requirements and I've just brought five of those here. It's an excerpt of a very much longer list, as you can imagine. The first one, no access from cloud to our on-prem network. That's not strictly mandatory by the rules, but in the project, after long discussions, we decided that it's safer to do that. Our government thought that the cloud is inherently more unsafe than our on-prem network and they wanted to have assurance that when the cloud is compromised, the on-prem network is not. Second one, it's also quite annoying, actually. We can only use services that are hosted inside the European Union and that excludes some managed services for us. For example, if we wanted to use GitHub, not possible because GitHub is only hosted in the United States and you can't self-host that. Third rule, establish separation of development, research, testing and production environments. That means we have to set up a staged platform, and in the development stage, we do implement our research platform itself, so adding new services, modifying existing ones and so on, and while doing that, we don't want to disturb the actual research that we are doing in our platform, so that of course happens in the next stage, the research stage, and in that we also develop and modify our data pipelines, which then are tested in the testing stage and of course they run in production where the production pipelines run in the production stage. So, the thing is how do we keep all those stages in sync? I mean, if you set up cloud services, then you typically have two options. You can either use the web portal and click your VM or whatever service you want to have. And then there's of course in most cases you have a CLI, a command line interface, and you can use to set up that for setting up services as well. Thing is if you do all that stuff manually, then it's virtually impossible to keep everything in sync, and there comes a concept that's called infrastructure as code, so you describe all your infrastructure components within a script or a configuration file. I brought here there's ARM, of course, that's a possibility, that's a Microsoft-based solution, so it just works in Azure. Or you can use some cloud agnostic technology like Terraform that at least in theory works in all the major clouds. Now we have the ability to create reproducible environments. We also need a mechanism to promote changes from one stage to the other, and I would recommend that you use CI CD pipelines for doing that, so a DevOps approach if you will. The last requirement I want to talk about today is the need to define an exit strategy, and that's actually something that's mandated by BaFin rules, so we need to be able or we need to have a plan to switch cloud providers in case our cloud provider moves out of business or sharply increases prices or whatever reason there might be that we want to switch. And in this case also, it really helps if you have infrastructure as code in place. So, now that we've gathered all those requirements, how do we decide which cloud provider would be the best fit for us. And that's a tough one, actually, because as a small project, we can't evaluate everything. We can't set up each cloud provider, we can't negotiate contracts with them, even if it's a trial contract, simply just not possible. So I would recommend at this stage to get external help. There are consultant companies out there that are surely happy to help you, yeah, and who also know all the major cloud providers from a regulator's perspective. So we did exactly that, and the recommendation was that for us, Azure would be the best fit. And the reason is that I think at least at that time, it's a few years ago, Azure actually was kind of a first mover in terms of the German or maybe the European banking industry, so they knew those requirements best, might have changed by now, so I think AWS and GCP also have procedures in place right now. But But at the time, we thought Azure is the best fit for us. And then in addition to that, we also were able to use a contract of our parent company that's a new investment. They already were using Azure services and they had a contract in place that we can also utilize. So basically, at this moment, we decided to start off with Azure, even though we didn't have an in-depth look at our users requirements, to be fair, but we felt that in terms of functionality, it was at least roughly on par with AWS and GCP. So how to deal with the ever feared regulators and users? Coming to the users. And there are a lot of use cases, I'm sure you all are aware of that. And I believe the most important features defining a great research platform is the ability to store virtually unlimited amounts of data and of course to provide scalable computing power. And I brought four other use cases that I think are also very important. You need a place to store your source code obviously. We want to provide a cloud development environment. We need a place to host our Python packages that we develop and we want a tool to execute data pipelines. And in the following slides I will go through each of these use cases and outline our thoughts on the topic also explain what and why we did it. So let's see how to satisfy everyone's needs with the right tools and services. All right. Starting with storing the data. First of all, please do not store large amounts of data in your source code repository unless you really want to break it. So where should we store our data then? Well, storage technologies can be roughly sorted into three categories, at least those that are relevant here. First of all, data lakes, and data lakes are cheap and fast, and they are best suited to store unstructured data, for example, text documents, but they can also store tabular data in form of CSV or maybe files, the latter one is better, I think, it's compression and is also most of the time faster. There are other file formats that you might also want to look into. Then there's SQL databases and data warehouses. And obviously, they are great at storing relational tabular data. And they usually add functionality like, for instance, transaction security. And there's no SQL databases that are best to store JSON-like data. All right, what did we do? At Quonium, we like our storage cheap and fast. And since we don't have JSON-like data and transaction security is not of a main concern at the moment, we decided to start off with Azure data lake storage gen 2 and use files for our tabular data heavily. That's not the end of it. We have future plans to evaluate data warehouses like Snowflake and Synapse to see how we can profit from using those. Okay. Providing computing power. Providing computing power for machine learning is probably the most important reason to have a research platform in the first place. And you can use big VMs and for relatively small use cases, they are often the best solution, because it is just low complexity and overhead there and you don't have to transfer data between cluster nodes and so on, so if you have a use case that fits into a VM, I suppose you should use such technology. But what about our big use cases if we want to process, I don't know, hundreds of gigabytes or terabytes or whatever of data. So obviously we need some kind of computing cluster. And yeah, what are our options there? So there's the big all-in-one machine learning solutions that usually come with the computing cluster, something like Azure Machine Learning, Amazon SageMaker, and Google's Kubeflow. And you you can really use those and they come also with a lot of additional functionality like no code, feature stores and whatever you like. The thing is I believe they are sometimes a little bit inflexible. They are kind of a one-size-fits-all approach in my book. So maybe a more tailored solution might actually be a better fit for you. And also there's the danger of vendor lock-in, at least for Azure ML and Amazon SageMaker, those services just cannot be hosted somewhere else. So it's different with Kubeflow. Kubeflow is Kubernetes-based that can be run anywhere you have access to a Kubernetes cluster. The next option would be to use a Spark solution, for instance, Azure Synapse or Databricks or there are other options as well. So that's the de facto standard for distributed computing, obviously, and it's been in place for a lot of years, I think 10 plus years, I guess. I haven't researched that. The thing is, what I don't really like about Spark is it's not really native to the Python ecosystem. Even if we use PySpark, we have to cope with different APIs compared to Pandas, for instance, and we can't just easily use our Python packages like scikit-learn or XGBoost or whatever. Spark comes with its own machine learning library like MLlib. If you like that, you can use that, of course, but you are a little bit inflexible there. That's where Dask enters the stage, and I'm sure some people of you have heard great talks about Dask on this conference, and there was a tutorial yesterday, I think. Dask has great integrations with the most common Python machine learning packages and its API, its DataFrame API is also compatible with Pandas to, I don't know, nearly 100%, I think. So, what did we do? Well, obviously SageMaker is not an option at the moment because we're just not using AWS, and then we evaluated Azure machine learning and Databricks, and they, as I already mentioned, they bring a lot of features, and most of them were completely ignored by our data scientists. So they had really low acceptance, actually, which was surprising to me, but that's just what the case, it was the case, or what it was like. So for example, in Azure ML, you can draw your models onto a canvas, and that's great if you're a beginner and don't know how to code, but our colleagues felt rather limited by that approach, and they told me they'd prefer to code instead of painting. In contrast to that, DAST was very well received because of the mentioned integration, the The data scientists didn't have to change their workflows, they really like it. So yeah, that's the solutions that we use for the moment. We do have future plans once again, evaluating Synapse, which would add Spark capabilities for those who like it to our research platform, and of course also I really think that we We need to have a look into cube flow because that really looks great at least on paper but we haven't tried it out yet. Then storing the source code, obviously we need to store our source code somewhere and while using Git seems to be a no-brainer today, we certainly need a place where we can host our repositories. So there are lots of options out there, you can use GitLab, Bitbucket, GitHub, Azure DevOps And of course also GCP and AWS come with their managed solutions as well. So the question once again is how to choose one. But first of all, you might be able to strike some from your list if you're not using GCP and AWS, those solutions obviously don't make much sense for you. But what about the other tools on the list? Well, you should have a look on tool integrations. If you are, for instance, using Atlassian tools like Confluence or Jira, Bitbucket might look promising to you. Another interesting question is whether you want to self-host or use a managed service. So Bitbucket can be self-hosted, also there's a managed service from Atlassian, but it can can be self-hosted, and that's just not possible with, for example, GitHub. You can't self-host GitHub. On the other hand, those managed services that are provided by your cloud vendor, they might have great integrations with your cloud services. And then, last but not least, you should also have a look on edit functionality. So if your platform brings the ICD pipelines that you might want to use anyway, also task management, artifact, repository, all those might spare you the effort of setting up additional tools. So, what did we do? First off, at Quonium, we started with an on-prem hosted Bitbucket and that was pretty great because it had really nice integrations with Jira and Confluence and we were using that. So, why even considering doing something else? Well, as already mentioned, we are prevented to access on-prem resources from our cloud network. So, we needed something that was hosted in the cloud. And we could have migrated Bitbucket, of course, but then we had a look at the options and we found that those Azure integrations of GitHub and Azure DevOps were very alluring to us, and maybe I need to mention that GitHub has been bought several years ago by Microsoft, so they have great Azure integrations as well. The only thing is we can't use GitHub anyway because it's hosted in the US only, so we are barred from using that. So, in the end, we ended up using Azure DevOps, which is fine by me, works great. In the future, I think we will probably revisit GitHub if they ever decide to add European hosting. Cloud development environments. We can develop our code on our on-prem machines, can we? Sure, we can. But developing directly in the cloud has a couple of advantages. For example, you do not need to transfer data out of the cloud and back into it all the time, which of course introduces time lags and actually also costs. And also, we have more control. That means we can quickly set up new dev environments and we can scale them liberally. And these environments are also comparable to each other, so no more issues of code running on one machine but not the other. of you might be familiar with all these works for me issues. Used to take a lot of my time to fix those things. So, how do we achieve this? First of all, the Python ecosystem, of course, provides something really cool, and you might have heard of that. It's called Jupyter Notebooks. If you want to add user management, then, of course, you should set up a Jupyter hub by yourself, or you could use a notebook experience that's inbuilt into other services like, for instance, Azure ML, Kubeflow, or Databricks. Notebooks are really great. They shine brightest, in my opinion, when doing data science or in presentation scenarios where you present your results. Engineering work, on the other hand, is often best served with full-blown IDEs. You can either connect your locally installed PyCharm professional or VS Code to a Cloud VM or a container, or you can, and that's becoming more and more common, you can use an IDE in a browser. There are some promising solutions out there, including GitHub Codespaces, Gitport, Eclipse J, or Coder. What did we do? We wanted to provide a notebook and an IDE experience, and in terms of notebooks, we wanted the real thing and not depend on another service, so we set up a Jupyter hub in an AKS, that's an Azure Kubernetes service in a Kubernetes cluster. And in terms of IDEs, we decided to start off with a slim solution, connecting our local PyCharm or VS Code to Cloud VMs, but in the near future, I'm really eager to explore the IDE in the browser solutions. Actually, a few days ago, we started a project doing exactly that. Posting our Python package, once we've created our own Python packages, we of course need a place to store them somewhere from where we can easily install them via pip or Conda or whatever means. And there are lots of options out there as well, including DevPy, JFrog Artifactory, Azure DevOps Artifacts, Nexus, and a Conda server. And to choose one, we once again should look on tool integrations, but also at the ability to host not just Python packages, but maybe we do need hosting for other software components as well. For instance, If you're using Docker images, you need a place to store them, or maybe you are using something like Java, then you can also host Java libraries there. And then there's the regulatory requirement that comes in here as well, so we probably need protection from vulnerabilities, so of course all those open source packages might have security issues or might even be malware. Those tools might also give you the ability to blacklist or whitelist packages. Cost, of course, is always an issue. So, what did we do? First of all, before we went to the cloud, we started off using an on-prem installed DevPy that has pretty limited functionality, but it actually really does what it's supposed to do. It doesn't have the nicest user interface, and I think there hasn't been much development in the last couple of years. When we went in the cloud, we wanted something that had a little bit more functionality, and originally we thought that Azure DevOps artifacts would be a nice fit for us because we were already using Azure DevOps and Azure DevOps Artifacts is a subcomponent of that. Yeah, it turned out it had a little bit of low usability because it added dependencies to additional packages that we didn't like. So we went to the next tool on the list and that was Nexus, set that up and really liked it. NEXUS also has the ability to optionally give you protection against CVEs, that's short for Common Vulnerabilities and Exposures, by the way, to be fair for additional costs. But for the moment we stick with NEXUS, might in the future also visit JFrog Artifactory and Anaconda server, but for now we are quite happy with NEXUS. and the last use case is to execute data pipelines and of course one of the biggest hurdles in order to profit from data science is getting it to production and for that we need to execute data pipelines regularly and there are a lot of options out there and I just listed five of those. Azure Data Factory is actually part of Synapse nowadays and it features a graphical designer that you may like or not. And it might be a good solution if you're using Azure. But once again, I have to warn you, there's also a vendor log in situation here because as I already mentioned, Azure data factory cannot be or Synapse cannot be migrated to other cloud providers. And then there's the other solutions, they have an explicit Python background which is great in my book, of course. Airflow, I think most of you know that it's the oldest and probably most widespread of those solutions. And actually, if you're using AWS, it comes as a managed service, which is great. Prefect and Dexter are newer contenders, and they promise to address some of Airflow's weaknesses, like, for instance, better transferring data from one task to another. And then there's also Kubeflow, again, So, what did we do at Kronium? As Stuart stated earlier, at Kronium we like to code, so no code solutions are met with skepticism in our house. So, for now we decided to use Airflow as the most widespread and maybe most major Python-based solution. That might change in the future though. Kubeflow is certainly interesting and it really covers a lot of our requirements And I am pretty sure that we will evaluate it soon and also the other contenders because their approaches seem nice as well. And even Synapse slash ADF might make a comeback because it just provides so many other nice functionalities like Spark and data warehousing. To wrap it all up, let's return to our initial questions, how to get started. But I think the single most important thing when you get started is you need to get buy-in from your users and also from your management, because if you don't, then you're doomed. How to deal with the ever-feared regulators and users? You really should carefully collect your requirements and you should get external help or support whenever necessary. You cannot know everything, but surely there are experts out there that are happy to help you. How to satisfy everyone's needs with the right tools and services? The space is vast and the options are numerous. You cannot explore everything. You have to make assumptions and if you are really in doubt, then I suppose it is a great idea to do the occasional proof of concept. And to summarize everything, I'd say better done than perfect, and even an 80% solution will help your data scientists tremendously. So you should just get started. Thank you for listening, and I'd be delighted if you contacted me through either of the listed channels. And finally, I guess we have a few minutes left for questions.

Speaker 2 [34:24]

Okay, so let's go to the questions. You said you cannot run GitHub on your own, but there is GitHub Enterprise, which many companies host themselves. Was there another regulatory constraint?

Speaker 1 [34:38]

Actually not, and if I am mistaken there, I will thank you for the hint, then I will check it. But we had a contact with Microsoft and they said it's not possible. I don't know, maybe there's a misunderstanding there.

Speaker 2 [35:03]

Thank you. How do you manage to convince our users to move to Python?

Speaker 1 [35:10]

Yeah, that actually that was that was a long process actually in our house there were long discussions and we tried to solve it democratically and we had a decision matrix which programming language is better and They're not really did resolve that in the end our CTO that had recently joined Made the call and then we went with Python. So I was happy with that That's, I don't know, guided democracy, I think.

Speaker 2 [35:46]

What features of Dask did your users find the most interesting?

Speaker 1 [35:53]

Yeah, actually, I think they are using quite a lot of those features. I think they use the delayed function to transfer your own written functions to the cluster, but they're also using the data frames and even the integration with scikit-learn, I think, a lot.

Speaker 2 [36:15]

Next question. What kind of use cases or bottlenecks that appeared causing you to consider a warehouse solution over the current data lake?

Speaker 1 [36:24]

Sorry, can you repeat?

Speaker 2 [36:26]

What kind of use cases or bottlenecks that appeared causing you to consider like in the future

Speaker 1 [36:26]

Yeah, sure.

Speaker 2 [36:33]

I guess a warehouse solution over the current solution of using data lakes. Yeah

Speaker 1 [36:40]

Actually, the thing is we are currently quite happy with the data lake. We've written a Python library to access that data easily. We have pretty standardized parquet files with standardized columns and so on. So that works really great. And we do not need things like, I don't know, foreign key restrictions and stuff like that. So for research, it's not really important. but if we want to develop real applications, in our research we are just doing data pipelines basically, we don't do anything with UIs usually and so on. So if you want to develop real applications, then I think something like data warehouses or SQL databases might come in handy.

Speaker 2 [37:26]

Thank you. How did you iterate through these decisions? Did you have a small group of users?

Speaker 1 [37:33]

Yes, we are in close cooperation with our data scientists, that's actually our sister team from research technology and yes, we're doing this all the time with our users. We have them within our project team and we ask them to use all the stuff that we set up all the time, we ask them of course what do you need and of course all the time what What do you like and what could we improve?

Speaker 2 [38:06]

Where does Kronion host JupyterHub, Nexus and Airflow? Are they all co-located in the same AKS instances? How does Kronion segregate deployment environments?

Speaker 1 [38:19]

Okay, first of all, yes, they are all hosted in AKS, Azure Kubernetes Service Clusters. And each of our stages runs in its own Azure subscription. And there's complete network isolation between the stages. We do have another subscription that we use that's called Shared, where we do store really few things. For instance, our raw data that we ingest that we do usually place in the shared. That's just a data lake, more or less. We do place it there because some of our data providers do not like if we ask them too many times a day. So we just download that once, put it in the shared space, and then to that, each of the stages has access.

Speaker 2 [39:16]

Thank you, do you have a plan something like model repository slash experiment tracking?

Speaker 1 [39:26]

Yeah, well, we did evaluate Azure machine learning, as I stated. There's that, of course, in there. Actually, we are currently doing that with a self-implemented solution in our data lake. I'm pretty sure that we could profit from using something like Kubeflow or maybe even Azure machine learning. I'm there yet.

Speaker 2 [39:54]

Thank you. How is the query performance and also the filtering capabilities when using Parquet with Azure Data Lake Generation 2? Great question. Also, how long have you been using the Data Lake?

Speaker 1 [40:06]

I don't have any numbers right here, but we did do extensive testing in terms of performance, of course. We felt that the performance was greater than the on-prem database that we were previously using. So we have still there, it's a SQL database from Microsoft. The performance is better than that. I don't have any numbers here with me, of course. Thank you.

Speaker 2 [40:37]

thank you have time for two more questions GitLab offers everything at one place such as CI CD along with Azure integrations why did you not choose GitLab any reasons

Speaker 1 [40:53]

Well, first of all, when I started off with this project, we were already using Bitbucket. I said that also in the talk. And that was really nice because of the integration with Confluence and Jira. And then we just had to make the decision to migrate to the cloud somehow. And of course, one idea would have been just to migrate the tool that we already knew, Bitbucket. And then we saw those integrations with Azure, so Azure DevOps was the decision. So we actually didn't really evaluate GitLab.

Speaker 2 [41:33]

Thank you. Last question. Have you ever thought about storing the data in a local data center and only use the cloud for the computing power?

Speaker 1 [41:46]

Before we went to the cloud, we, of course, had all the data or we still have all the data in our on-prem database. Yes, and you can do that, obviously. And actually, for some use cases, the users are still doing that. But I think if we go to the cloud, then we should do it the right way and the complete way. So, yes, you can do that. But I don't see very many advantages in doing that.

Speaker 2 [42:16]

Thank you very much. Please, a round of applause for Andrea.