Reproducible machine learning and science with python

Speaker 1 [00:03]

Hi, so my name is Prabhan Seng, and today it's really nice to see so many people interested in reproducibility and machine learning. I was not expecting this much audience, to be honest. So, something about me first. I'm a research engineer and machine learning researcher at OpenML Project and TU Eindhoven. I'm an open science advocate. As you can see, I work for OpenML. I have been active in the Python community for the last three to five years. And I was a PyData organizer for PyData and PyData and I like yoga. So what is reproducibility? Let's just talk about first I'll just give a small idea of what this tutorial is going to be. First we'll talk about reproducibility. what is it and how can we tackle it by using OpenML. Then I'll give you a small introduction on what is OpenML and what are the objects inside the OpenML through which you can reproduce, through which you can ensure reproducibility. And then we'll get to the hands-on tutorial. So the hands-on tutorial will be around one hour, and before that we'll do half an hour of just talking about what we're going to do. So I assume if you guys are here, if everyone is here, they must have faced reproducibility in some kind of machine learning workflow at some point in their life. And now the biggest problem we have with reproducibility is especially in scientific experimentation because we have these big models like GPT-3 or DALI, and they are not reproducible even by a big company. leave a small student but there are also a lot of papers or just smaller experiments by research team who want to ensure reproducibility of the experiments but not a lot of time they can be reproducible so what's the main reason we want to ensure reproducibility the first one is that students can learn from it we can ensure that our scientific experiment has more integrity because it's reproducible and that's how we verify if it's right or not and that this does allow more community community reachability so this does allow us to reach more people from different backgrounds and different countries who don't have the resources to or reach to access the experiments next one is about reproducibility in machine learning. So what are the elements we need for reproducing a machine learning workflow or machine learning experiment? The first one is data. So I think that's one of the biggest bottleneck that most of the experiments face. If we don't have the same data or even if we have the same data we don't have the same pre-processing workflow for that data to ensure the reproducibility in our workflow and to make them make the experiments again second one is task so tasks are a bit it's a little bit more how should I say meta term that task basically mean what do you want to do with the data so if I have a data let's say iris or credit G or any any other data set what what am i evaluating on that data and which metric I'm using for evaluating on that data. Sometimes even some papers use their custom metrics which makes them also not reproducible because a lot of people don't know how to compute those metrics. Third one is definitely models and I don't think I need to explain that part here because that's what most of us face. The fourth one is hyperparameters so if you have a model let's say I give a small pipeline about my machine learning workflow, but I don't give which kernel I'm using or I don't give which parameters I'm using. Then it doesn't give you the right model. Also just set my stopwatch here so I make sure I'm on time. And the third one is performance benchmarks. These I have added recently, and these we have been working on recently. But we think that these can be really beneficial for ensuring reproducibility in the machine learning lifecycle. So now let's talk about how can we make machine learning reproducible again. Now here we'll go and talk about OpenML. So how many of you already know about OpenML? Nobody? Oh, wow, surprise. That's exciting for me because now I can tell all about it, yeah. So OpenML, I can even tell the history now. So OpenML was this project. It was started by two people in the University of Leiden. So it got started in the Netherlands and some team from Munich and other countries together by a few researchers who wanted to ensure a global knowledge base and wanted to ensure reproducibility back in 2014. Right now I'm a core engineer for OpenML project and I've been working on it for the last three years now. And now we'll see how OpenML works. So let's talk about simple machine learning workflow. What do we need for machine learning workflow? The most important part that we already talked about is the raw data. which is being handled by data engineers, which then get pre-processed, we feed that data into a flow, so here we talk about flow basically means a model, but six to eight years ago there was not a right definition for that, so we just called it flow, and then we train that data on that flow, and we have a run, or a deployment generated out of it. So this is our current machine learning stack. I think everybody is familiar with one or two tools out of it. Panda, Spark are more of data pre-processing framework. You can include Dask also in that. Then you have these machine learning libraries, TensorFlow, MLR, PyTorch, Scikit-learn, Weka. I don't know how many of you are familiar with Weka here. Oh, wow, that's surprising. So we also ensure workflow for Weka, which is not very common. But that's also a good example, because if I make a really good model in Weka, how would I ensure the reproducibility for somebody who doesn't even know what is Weka? And when you go to deployment, you have MLflow, you have Dockerized containers, you have Kubeflow. So this is basically a small lifecycle of a machine learning model. And now let's talk about the objects. So these are all the objects which exist in OpenML. And now we'll talk about what these objects do and what are we trying to achieve by creating these objects. First one is data sets. So we have data sets and we have also metadata with them. So what is metadata? It can be information about classes, it can be information about feature distribution, It can be more than 100 qualities computed on a data set. So I'll give you a small example of how do we do that in OpenML. So if you have a simple data set like credit G, which most of us know, you have a description, you have features, and then you can compute a lot of qualities on top of that. So you can have number of instance, mutual information, random tree depth, max abdominal distinct values. So these meta features also do allow us to ensure that if a data set we are taking is a balanced data set or not a balanced data set and what kind of qualities we do expect from the data to compute workflows on top of that. So that's what we refer here by data sets and metadata. Right now we currently save open ML data sets in a very old format called ARF. You can download them in CSV and other formats, but we have started migration to Parquet framework this year and we will be done by the next few months to have all the open ML data sets in Parquet. Second is what is a task? So I think I already explained a bit, but a task basically means what do you try to do with the data? So you just have a data set, and you want to do an evaluation on top of it. So you say, okay, I want to do supervised classification on top of MNIST, and that's my task. Third one is a model, and I don't need to explain models too much. pipeline they have the parameters and everything else and runs are the configurations that you have for the pipeline and evaluations you generate from the pipeline so if you have something like that this kind of also ensures you to have a global knowledge base of all this metadata that you have and this does allow us to do a lot of operations on top of that data and even take that data and generate the same workflow again. Now, we also allow auto annotation, but this part is not very relevant to today's tutorial. So if we have enough time in the end, maybe we'll talk about it. Now, here is a very simple way how we use OpenML API. So is everyone familiar with Python here, or there are also some people who want to use our in today's tutorial, by the way. Okay, great, because that's the last time. But so we have a simple Python API where you have this data, you as users or me as a researcher, I publish my data, I publish my flows, I publish my runs, and those get evaluated on OpenML and those are then accessible through our web UI or through our APIs in Python, Java and C++. This does allow us to have a concept of frictionless machine learning. How many of you are familiar with the concept of frictionless machine learning here? Great. The idea behind frictionless machine learning is that you can share the data easily, you can import the data easily, and you can create runs on that very easily. So the idea is that you don't have any friction between all of these processes that you are trying to do on top of the data. There's a really good resource, it's called frictionlessdata.io. I would recommend to read about that if you're interested in this area. We also provide a nice web UI for these websites and these flows to make them reproducible and for you to also allow if If you have a data set and you want to see which pipeline performed better, so we have 10,000 users who have computed multiple data sets, multiple pipelines, we also have some bots which randomly put pipelines on data sets, and then you have those on top of them to generate these things. Now so how many of you have used scikit-learn, raise your hands, great. How do you get data set out of scikit-learn? You do scikit-learn.datasets, and then you get a dataset, right? But where does that dataset come from? So fun fact, it comes from OpenML. And we do have a really nice scikit-learn integration as well. So if you want to create a scikit-learn workflow only, you can just integrate that in OpenML. You can just use that workflow and publish that workflow directly on OpenML. So we do have a cyclone, we do have a PyTorch integration. These are in a bit better, PyTorch is a bit in alpha stage, but we are working on that. And now let's talk about, again, reproducibility by OpenML. So since we have all these exact data set algorithms and flows used in every experiment that the user performed, this does allow us to have reproducibility in our workflow. A simple way you can just make a flow on OpenML is that you import OpenML, you have a scikit-learn pipeline over there, you wrap everything into one pipeline object, then you run a model on a task, which I already explained, you have a model and you want to run it on a task, and that already gives you a nice run object. So a run object has all these evaluations that your model did on the task, and then you can publish that object, which will then be published on OpenML website. And then you can also get the same run. Sorry. You can get the same run here and then you can get the flow out of it. What we don't allow in OpenML is to upload weights of the flows because that can become unsustainable really fast. Sometimes neural network can reach around 10 gigabytes or 50 gigabytes for a simple experiment and allowing us to upload the weights as well, allowing users to upload the weights as well can be problematic for us so when you go to the flow you go to the schema of the model and then that's you have an idea of what the model is and then you can reproduce that model in by yourselves so we do have a small team and a small community we have around 150 000 yearly users 8,000 registered contributions, more than 500 publications have used OpenML and cited them. We have more than 20,000 data sets, but today we'll only be working with the active data sets, which are around 5,000. We have more than 8,000 to 10,000 models and more than 10 million runs. And this is, I think, a three-year-old slide, so we have much more now. This is our small team of people who are volunteers to open ML project. Now any questions about open ML right now? Can you repeat to say that it's not available for PyTorch yet, or you said something about PyTorch and open ML, what was that again? It's in the alpha stage. I have, I was one of the authors for that extension, and I did create it, but it's hard to publish, for example, a few things like if I'm doing a transfer learning workflow, so you do have a few bugs, but we are looking for more people to find more bugs and let us know so that we can change that. If you want to see all the projects under OpenML umbrella, they are here. So you just go to GitHub, OpenML organization, and you can get, you can see all the APIs, the main core API, and blog, benchmark suites, and all the benchmarks that we have created. One object I have not covered here is benchmarks. How many people here are familiar with the idea of benchmarks? Great. So what do I mean by benchmarks? So in OpenML, since we have all these datasets and all these tasks, we can make a collection of these datasets and tasks to ensure something, robustness of an algorithm or robustness of a system, because you just don't want to take random tasks or random datasets and say, oh, my algorithm can generalize, or my algorithm can make something. So we do allow users as well to create their own benchmarks. So if I decide to, let's say, take all the data sets related to heart diseases, and I want to make a collection of them, I can call it a benchmark and advertise it to people that, okay, this is my benchmark, I think all the algorithms which work on this benchmark work better with all the other heart diseases as well. But again, if the community doesn't use your benchmark, it's kind of useless. We do have two benchmarks out of our labs which are being quite widely used. The first one is OpenML CC18 benchmark. Anybody familiar with that? No. And the second one is AutoML benchmark. So these two benchmarks are being quite widely used by people. And second, we call benchmarks a collection of tasks. So if you make a lot of tasks for an algorithm to do, that's what we mean by a collection of tasks slash a benchmark. But what if you do your algorithms on that benchmark and you want to publish those algorithms? What do you call that? So we call that collection of runs. There's not really a word for that out there. So this was basically one of the best words we could find. So this is a very useful object if you think about it, because you have all these tasks that you want to do, then you have all the runs which people generated on those tasks, and then you can see, oh, what's the accuracy I'm getting on what task? And which run is the best for all these tasks combined? And this kind of thing can help a lot of users to select better algorithms and then use those algorithms to ensure that they can recreate them and use them on their systems. Now let's talk about... Now let's get to the hands-on session. I think we are already 20 minutes into the presentation, so it's a good time. Any other question about OpenML before this? Yes? Is there a way to publish or to run any collection of runs? Yes, so. Can you repeat the question? Okay, the question was, is there a way to recreate the collection of runs? I don't know if you can recreate the collection of runs because they're already run object, but you can definitely access that entire data in a dataset format, and then you can recreate individual flows one at a time, because what you get in a run object is something like this. So we call it a study, by the way, from the Python API. So sorry for all this confusing terminology. I know it can be a bit overwhelming for somebody who is new to this platform, but believe me, this works. So we call it a study, and a study is basically we have a Python object where you get all these task IDs, and then the only thing you get on top of that are run IDs and setup IDs. So once you have those run IDs and setup IDs, you already have the access to flows, and then you can recreate those flows by yourself. So you can, right now, you can recreate all the scikit-learn and R and WCAG flows in OpenML, but it gets kind of hard to recreate TensorFlow or PyTorch runs, because if I create my custom class in PyTorch, then OpenML only gets the name of that custom class. It doesn't really know what you did inside the custom class. Because our job is not to ensure recreating code, because that's what GitHub is already doing. want to recreate the model schema. Did I answer your question? You can. I don't think you can modify a collection of runs. You can just create a a new object and add run IDs in that object to edit that run. But if you mean that can you manipulate the statistics of the models inside the run? Yes, you can. I kind of don't get what you mean, but once you publish the run, you You mean after publishing or before publishing? So if OK, great. So I have already added the instructions for this on the Discord channel. Has everybody installed OpenML and cloned the repository? Does anybody have any trouble with installing OpenML or cloning the repository here, by the way? What's the concrete task? What? Yeah, like just first let's clone the repository, install OpenML, and then we'll go through the notebooks one by one, and then we can go through that. Yeah. So, shall I wait for two minutes or shall I just start with the, yeah, sure. In the meantime, this is the first time we are doing this tutorial for our developer audience. We are used to doing this tutorial for a more scientific audience, so I'm really excited for your review of this tutorial after that. So I've had a question in the meantime, first, you all can first make sure that you are able to clone the repository and install OpenML Python, that's the most important part. In between, I had a question, if OpenML is more for research groups or also for enterprise projects, and what if a pipeline should be reproducible within the team but not publicly? So OpenML, we do allow, we don't have anything private in OpenML, by the way. So the entire website code and the entire API code in the backend is available for everybody to download. And you can create your local clone of OpenML and just run it inside your network. And that's how you can ensure reproducibility inside your teams. We've had few teams who have used OpenML deployment privately. Some big companies have also used OpenML deployment, do also use OpenML deployment privately in their research team. More than that, I don't think this can be a part of a machine learning deployment cycle. We are not even aiming for that. We are aiming for more from a society or sustainable machine learning standpoint where if you want to create a research or a scientific study, then you can allow users to recreate those flows. But if you want to create a private model and you want only your team to be accessible through this and you want to use OpenML, then you can technically clone the entire OpenML, even the Python repository, just change the server in the back end, just change the server address in the back end and you have a running local deployment of OpenML. I hope I answered that question okay shall I start with the tutorial or anybody has any problem with the installation great so now let's Open the slides, and now let's first talk about, so you do have access to all the slides, I assume here. And today we usually do this tutorial in a more interactive way, where you can change a lot of stuff on the test server, but today our test server is in the maintenance, so We are doing it with the production server, which is also not an issue, but ideally you should not, how should I say, change the data sets on the production server, but feel free to do that if you want to. So you can just, first we import the OpenML object as a simple Python object. Okay, this thing doesn't work. Yep. Now let's do a small data set list. I realize I kind of added all the solution as well to my tutorial slides, which I should not have done. I would say all of it, but yeah, sorry for that, but we can go through it line by line and we can change the data sets to analyze more data sets. That's what we can do right now. So if you go to OpenMR right now and you take the data list, this is a very simple thing we are trying to achieve here. We just get all the data sets and try running it on your notebook and see if you can access all the data sets from there. Do we have that already done? I have a question. Where can I get the notebook? I added the link on the Discord server. Yeah. So once I have all this thing, now I want to do, let's say, Anybody able to run the first few cells of the notebook? Any problem with that yet? Okay, great. So let's say I want to do it with a credit G data set or some other data set, and if I you can even query the data set, so if I say number of classes is more than 50, I can even do number of classes more than 20. And there I have this name of the data sets. And the number of instances, number of features, number of classes. So you can play around with all these data sets over there and see which data set do you actually want. Because sometimes you have all this data set repository and you don't you want a particular data set but you don't know which data set it is. So then you can access through this. Now once I have the data set, let's just download some data set from OpenML. So how can we download data sets from OpenML? Easiest way is you have all these data sets here. Does everybody know the link to the website? It's openml.org. And let's inspect some random data set from OpenML. So is everybody aware about the credit tree data sets? Let's inspect that today. The ID of this data set is 31. So let's go here. So I have not made huge changes in the GitHub repo, I didn't have enough time. But we'll just start with the, I think that's where we left, download data sets, yep. So we'll start analyzing some other ideas of the data sets. So like I said, the first idea of reproducible workflow is just to have the data set and see how, what is this data set and what are we going to do with this. Yes? Yeah, sure. Is it better? Yeah, yeah. Okay, great. OK, great. So when you download a data set from OpenML, you use a data sets object. And we use the function called getDataSetFromOpenML. So feel free to go on the website and take any data set you want. Did somebody ask a question? OK, great. So right now, I'm taking the createdG data set. This is the URL for the data set. And we have the authors, we have the source, we have the citation. And this is German credit data set. And it comes with a small cost matrix as well, which is pretty cool. Now, this only downloads you the data set object. If you want to download the entire data set, you just go with the get data. Here we want to download data frame. And let's just call this data frame is here. Now, you can inspect this data frame. And is everybody able to access the data frame right now, by the way? OK. You're able to run that? So now I want to do my simple split on the data set in a scikit-learn style. So you always get four attributes out of the data set. If you want to get categorical, you get an x value, y value, a categorical indicator, the attribute's name, and from the get data set. Now I just want to see how it's looking. Is anybody trying any other data set than CreditG, by the way? Okay, great. How's that going? So, feels, yeah. Now I want to do a small analysis on this data set by exploring the visuality of the data. So, let's just go with the... Simple bar chart where we are trying to do a small scatter matrix of the class with the bins and so you have the one duration here and the other duration on the other side. I made this tutorial for another data set so I'll just get back to how it should look like ideally. Which data set are you using now? This one was created sheet, 31 ID. And if you use it for the EEG sample data set, which is 1471 ID, then this one makes more sense because it has the bars plotted right. So then you have the duration of the EEG samples, and that's what you're trying to achieve here. is to explore how the data set looks like. So this, yeah. So I'm using now the notebook from the GitHub. Yeah. I pulled, so I got a lot of updates. Yeah. But this looks different, so I wonder if it means that you changed something now. Can I see? Yes, sure. Okay. I can also reopen the. Oh, no. The sleeve. That's all right. So I did pull. No, that's okay. Can you just show me the notebook? Yeah, actually mine is a bit wrong though. So the one you have it on the notebook should be right, yeah, that's, yeah. It's a problem with my matplotlib. I don't know for some reason it's not showing the rest of the graph to me. Okay. Yeah. By the way, you should have a much bigger scatter matrix because I did make it like that, but for some reason I'm not able to see that in the rest of it. So you should have something which looks like, sorry. So you should have something which looks like this because that's what I was trying to achieve. and the code is the same, but sometimes matplotlib only shows one thing. I just might need to restart my kernel for that. But that's where you have all the bins here, and you can explore the data visuality here. Now let's talk about the next element of this tutorial called task. So I will go through a few elements. I'll go through tasks and models faster so that we can talk more about benchmarks, because I think benchmarks are the most interesting part of this tutorial. OK? So now, was everybody able to run the dataset tutorial easily? You grab the dataset, you analyze the dataset. You can grab multiple datasets and work on top of that. The second one we'll be working on is the task tutorial. So like I already said, the tasks are basically what you want to do with the data set. And it's a small list on how do you download and analyze tasks. So it will be a smaller one. You just go. I want to take all the list of the tasks from OpenML for supervised classification. You can even do it for supervised let's go regression. If I want to print my... I have all these tasks out there, which do allow me... This is a bit confusing, but I could see that. But that's where you get all the source data IDs, the number of features in every ID. But if you put in a nice Pandas data frame, which it's not showing right now, it should look much nicer in a normal data frame. But let's talk about all the tasks OpenML supports. So you can try multiple tasks here. So if I go to this and task type, you could already see that we support different kind of task types. So you can go with clustering, learning curve, supervised regression and supervised classification. The most common are supervised methods because we don't support anything unsupervised on OpenML. And then you have the clustering, which we sort of support, but it's much harder to achieve that. Most of the people just remove the label of the data to call it a clustering task. So let's create... So everybody is able to list any regression or clustering or classification task, I assume. By the way, everybody able to follow up with the tutorial or it's too confusing? If it's too confusing, just let me know. It's also the first time I'm giving it to developers. Now I want to see, let's say, I want to query the task. So we provide this API called task.query. And let's say I want to see between 10 and 100 instance. I want to see really small tasks. So I get all of these tasks, which has instance between 10 and 100. And instance is a number of values in the data sets. Yeah. Oh, sorry. It's a number of instances inside the task, the number of data points inside the task. So how many data points, how many flows have run inside the task. Yeah, so then you get this filter task, where you have all these instances. Let's say I want to see something really big. So since we have more runs here, less runs for these many tasks, so you have less task ID, but now you can see some really big tasks. So if you see the length of these tasks, you only have 59 tasks at these many instances. And now I want to see the tenfold cross-validation on that. So you also get different option of estimation procedures. So if I want to have more estimation procedures out there, we call them measures here, sorry, not measures. Evaluation procedures here. So you can take different evaluation procedures. So OpenMRO, what it does is that when you're uploading it, it does take your estimation procedure in count and it does take your evaluations in count and evaluate the model against it whenever you go for run model on task which we will get to later. So right now we are doing tenfold cross validation. Now you have these many tasks which use tenfold cross validation. So now we are from 59 to 24 tasks. Now we'll talk about a small benchmark that we did. It's called OpenML 100, where we took 100 different kinds of tasks existing from every domain possible. And then we try to evaluate algorithms against that task. So you can also search it via a tag. So if you want, you can assign your task a tag on OpenML website or OpenML API. And then you can query through that. So there you have the task. It's all supervised classification, as you can see, with tenfold classification. You have the target size. You have a minority and majority class size. This kind of thing can also be pretty nice, because sometimes you want to see, OK, I already have my data set. Now I want to see the imbalanced data sets which exist, and I want to train my models on imbalanced data sets. So I can filter it out through majority and minority class size. So every attribute that you see here, you can filter them out through Pandas API, but you can also filter them out through OpenML API. So that's how you have all the tasks here. And for example, if I want to do a query here, this is basically a database query that you write. And it gets parsed to our database. And through it, you can get all these data back. So let's, we have a small task here. Now we already see, so any data set which you use for data set tutorial, now let's do it for a task. So we have a small exercise here where we want to see all the tasks related to credit G. You can also take some, wait, yeah, now it will be better. So right now I'm fetching all the tasks here from OpenML in format of a data frame. And then later I'll try to print those tasks. Okay, it's taking a bit of time. Anybody else also trying to fetch it? Maybe that's why it's taking a lot of time. Okay, yeah. So now I wanted to do credit G, I wanted everything that people did on credit G data set or some other data set. You can write ID or name here of the data set and evaluate it. Now I have all the things any researcher or a student or a developer did on this data set. So as you can see, you have the task type ID, which we say what you did is some people perform learning curve analysis, some did classification, some even did clustering. And then you have a task type here. So we have task type ID and task type are the same. But sometimes you can make it different, but let's not talk about that right now. Now we have estimation procedure. So people use different kind of estimation procedure that data set. So now I know what these results are being generated by. So some are generated by tenfold learning curves, some are by 33% holdout sets, some are by tenfold cross-validation. So now you choose your evaluation measure. Like I only want the run which used tenfold cross-validation or holdout set because that's what I want to train my model on and that's that I want to analyze the run on. And then you have all the other metadata that you can find. So not every task has all the metadata because sometimes it just doesn't exist. Now let's download a task. Like I did here, I just downloaded a task for 31 ID. Let's take any other ID as well and see what's going on with that task. So if I take any other task, let's go with build classification, which is based on this build data set. I take the ID, and I want to download this task now, okay, this doesn't work, so let's just stick with credit G, and I want to get this task, and then I have this task which did cross validation on credit G, so unfortunately in here, the task ID and the data set ID is the same, but that's not how it's supposed to be. Every data set has multiple tasks associated with it, so every task has a unique ID. And then we are trying to do cross-validation on that here, and we have a cost matrix availability here as well. So if you don't have a cost matrix, then OpenML does not take account for the cost matrix. Do you know what's a cost matrix, first of all? Okay. Half of the people know, half of the people don't, so So basically it's a cost per class that you have and you can use it for model sensitivity or you can use it for calibration and other things through which you find out how sensitive is your classifier. Did I answer it a bit better or I made it worse? What? Confusion matrix is, I think, the one you get out of the results, right, yeah. matrix is what I get out of the result. Cost matrix is I want to generate the result, but what's the cost for a false negative or a false positive? And that cost you can account for while training your classifiers. So here, if you have the availability of that, you can train better classifiers. That's the... Yes? I'm not really sure about that because sometimes some people, you can even create a regression on top of this, and though it's a classification problem, you can just make the target continuous and then it might not take that into account. And if you want to do clustering on it, then also you don't have that cost matrix is just going there. In scikit-learn, if classifier does allow cost matrix to be taken into account, it does. And this is mostly through the evaluation. So that OpenML is doing in the back end itself. So OpenML is going to put that in the metadata and then evaluate. So some metadata of the performance do use cost matrix. And then you can see that. Did I answer your question? Okay, great. So now I wanna play around with multiple task IDs as well. So let's take, sorry. Yeah, we have a lot of tasks there which are sort of private. And we want to make those tasks a bit private now because we are still working on them and then we want to release them to public. But if it's up there, you should be able to access it most of the time. For example, let's do a tic-tac-toe classification. So tic-tac-toe, it's a really funny data set. It's basically every entry is the tic-tac-toe matrix, and then you're doing classification if you won or lost. And it's a really problematic thing for all the classifier because there's no pattern in it. So it gets really interesting. And then we call it AI. Yep, exactly. So you can only manage them directly on the cloud? Do you mention something of making them private for, I don't know? So right now, the, okay, so the question is if we can only manage the task in the cloud or right now we can make them private. So the private tasks here that we have are not accessible to public at all. These are just, we had a legacy system at one point, which is still in the existence, and we are in the process of removing that. So since it's a quite, like, around ten-year-old project, we've had some mess-ups, which created some random task IDs. And sometimes you can get the error that it's private or public, but we are sort of working on getting it right, but currently users cannot have private tasks. The idea in the next one year or two years is just to allow users to have private data sets so that everything associated with that data set is private. Yeah. But again, right now the goal of OpenML is just to completely allow you to use it. By the way, right now we'll try to create a task, does everyone have their API key? I wrote on the Discord to sign up for OpenML. Okay. No problem. I'll just give a small one. So if you sign up, you can just sign up with the first name, last name, email, and password. I have mine already here. So then you can can go to this profile page. You have your API key. You download your API key, and... You can always refer to OpenML Python documentation for more stuff. And let's just do an example of start type. So here are some, there would be an instruction here or how to add your API key here. I think it was in one of the notebook. So I'm just adding my own API key here. So try not to take a picture of that or something. And then you have this task which I want to create. So I want to create the supervised classification task on dataset 128. So if you want, we can just take a look at dataset 128. The easiest way you can access OpenML datasets is you can do openml.org slash D slash ID. And you'll be redirected to the dataset page. So I'm doing BNG cylinder dataset here. And let's see. So I want to submit that data set, so... Now I have this task up, and now I want to... I created this task and it got published. So if I look at the task here, now in BNG cylinder data sets. So I will have all the tasks associated with this dataset here. So most of the tasks were created a while ago already because it's a quite old dataset. So you can publish your task, but if it's not new, we don't create a new task. So if you say I want to do supervised classification on MNIST with 10-fold cross-validation, most probably somebody already created that task before you. So if you try to publish that task, OpenML will just not either give you an error or it will say your task is already published. How much time do we have left? Okay, okay, okay. I won't elaborate more on task. Anybody understand what the concept of task is here? I thought this tutorial would be too fast, but it's apparently too slow. So let's go to a small example of what we want to do now. We have a task here. It's in the run tutorial. So I have flows and runs here. And now I'm taking the greater g again here. And I'm writing a k-nearest-name classifier on that. Let's do something else. Anybody has any other classifier idea? Random forest, enzalm? No? OK. So we can do a decision tree here instead. So, I have a small decision tree classifier based on graded G dataset. Feel free to make a new classifier over there. You can have random forest, gradient boosting, anything. And that's how you make a small classifier and train it on OpenML dataset. But now I want to, let's say, let's go to the next cell. We can skip this cell. This is again about making a pipeline instead of a classifier. So you can take another data set, which we took here. And you just have a column transformer, a one hot encoder, and then you just fit it with the, you can have a classifier on top of that, and then you have this entire pipeline that you generate in OpenML. And now I have this run here, so if I have this task, so I took graded G31, I ran a decision tree classifier on top of that, and now I want to do run model on task, what happened here? Okay, so I need to provide authentication key. Any of you got your authentication key yet? Okay, great. So sorry, I need to copy it back. Wait a second. okay sorry just once again I need to configure something and then I'll be done so you can just Just configure your API key like this. And then you have a run which you run the flow, why is it giving me a again? Okay we have a one hot encoder here, that's why it was creating this problem, so since I take the decision tree, I'm just creating a decision tree on this. But I have the string feature, so right now it's giving me, it's a cyclone error normally, so if you, rather than that, if you take this X, sorry, my bad, I did not see that coming. We can just take another task which does not require pre-processing, so let's take iris this task instead. So we'll take task ID 59. And we can run, oh, what the hell is going so we can just do this transform our object and And. Sorry, I think I'm, yeah, maybe there's, what? Just a second. I can just take a small example instead. But basically you can do a small model on task from a classifier here. I can just put a better run here instead from the docs. It's like I made a small mistake over there. What was the task 59? Task 59 is a? Iris. Yeah, it was Iris, yeah. So if I have it here, I could just take a small example of putting a. Small decision tree here, another task, and oh, okay, yeah, okay, so for some reason my My system is giving me some errors, but is any of you able to run this instead or not? Which is already in the notebook? Can anybody run it or everybody is getting the same error? Yeah, the original one. If you run the 31, or 403, are you able to run that? No? Yeah. Oh, okay, then leave it. And so the idea was basically to put a run model on task here, but because of some problems or some bug, I'm not able to run that. But now we'll go to the most important part of this tutorial, which I wanted to focus most on but didn't have enough time, is benchmark suites. So now let's talk about how do we make a benchmark suite in OpenML. So what's a benchmark suite? It's like I said, it's the collection of tasks and runs, and if we have a data frame, first we can run... It's on the benchmark suites.ipy.nb, by the way. Okay. I thought it was already five minutes. So you can get all the suites from OpenML, and the moment you get the suite... So for example, I got all these OpenML100, OpenMLCC18. If you don't give the alias to a suite, we just give it a UUID, and you have all the other IDs here for the suites you can fetch. Now if you want to download a suite, for example, I want to fetch OpenMLCC18 here. You fetch the suite, you fetch the description of what the suite actually does. So most of these suites are made by researchers, but if you're a user, you can also make your own suite. So now we'll try to create our own suite. You can remove this line, a configuration for example, and just add your API key over there instead. So now I want to upload a suite. So it's a test suite. You can write your name here, PyCon, and benchmark suites. I want to have more task IDs for the suite. So I just create all the task IDs. I'm randomly taking task IDs for the suite and just having the UUIDs over there. So I will probably get the authentication error again. so I'll just copy my API key from here. Is anybody able to, everybody able to run this notebook? Thank you. Is everybody able to run this notebook or still running the notebook and having any problems? Yeah. I can imagine it's I did not account that everybody would be accessing all the same things from the same server. So we have some risk condition over there now. But I think that was the last part about my tutorial. how can we improve reproducibility by allowing different objects like runs, flows, data set and task and benchmark suites on machine learning by OpenML. So feel free to contribute more to the platform and feel free to explore the website more. This was meant to be as an introduction to this platform which is quite huge, to be honest. And yes, that's about it. So do we have any questions? Well, I think he also got one. No, okay. He. Yeah, just a second. They always give priority to the slider, so if you have access to slider, maybe print your question there, but if not, I'll come to you. The mask, sorry. Yes. Yeah. Okay, so we have the only one question in slider, For the moment, can anyone increase the benchmark by using augmentation data model training and achieved, Sauter? State of the art. Okay, so we do have this problem that if you create, so when you refer to augmentation, you can do that. And that's what we prefer in OpenML. But if you want to make the augmentation pipeline reproducible as well, Then you basically just wrap the entire object something like this. So you just go to your flow, and you wrap the entire pipeline. So I would then wrap the entire pipeline with a column transformer, one hot encoder, categorical features into one pipeline, and run that pipeline on top of OpenML. So yeah, you can do that on benchmark, but the benchmarks are never referred as just a single model. They are always entire pipeline. So yes, technically you can do that, but I didn't get the question completely as well. Well, I think if you have more questions, people from the web can always find you on LinkedIn or GitHub. And yeah, we have a few more minutes to take questions from the audience here. So yeah? I have one simple question. So then if I got correctly in order to use OpenML, you need to have the data set there, right? Yes. So the starting point to be using for my own project is to put all my data there, actually. So if you want to make a private deployment, if you want to have a private data set, you can just make a private deployment of it. But this is more about reproducing scientific experiments. So yes, you need to have first the data, and then you build on top of it. Because in the end, this platform is to make other people reproduce what you did. So yes, you need data set on that. Yeah, but then what I mean is everybody will have their own deployment, because no one want to check the data, and also many times you cannot check it. Definitely, but that's not what this platform is for. So the entire platform is for reproducing what you did, and it is for reproducing scientific experiments that academic labs or industrial lab are doing. And we want to reproduce those workflows. So you do want to ensure that your data is always the first starting point. On OpenML, if you go right now, you'll find even five copies of the same data set. And that's not by accident. Because some people take the same data set and they apply one-hot encoding, but they don't know how to put that in the pipeline. So they just upload the data set with one-hot encoding, And some people, they take different classes, and they want to see, OK, what if I make the data set in balance? How would the data set work? So that's the idea behind the entire platform, that you want to make this entire flow reproducible. So yeah, if you want to make the data set private, then definitely this is not the right approach to go. But then I think if you want to make your code reproducible without the data set, then maybe GitHub is a much better option than OpenML. Thank you for the question. Any other questions in the audience? Okay, moment. Thank you. Do you have any experience with publishers in the scientific community? So, So do you cooperate with the big publishers like Albuquerque or ACS, Nature? What was the question? If I have any? If OpenML does corporations with publishers. No, we don't do anything with the publisher's corporation or anything. We just allow users and researchers to upload all of their data sets over there. And we, like, we also hire people, students or employees sometimes, because we're funded by academic grants, to scrape the Internet and take all the public license data sets to OpenML. So if you go to OpenML right now, you find all the data sets which are publicly listed on Google data sets or UCI or ML data on the platform. So if the dataset is public licensed, we just copy it and put it on the platform so somebody can copy it easily. But yeah, we don't do any collaboration with the publishers as far as I'm aware. Thank you very much for the question. I think we have time for one more question. One question about the reproducibility. When OpenML creates this pipeline, is the seed always kept fixed? Because you have this Python seed, NumPy seed, Pandas has its own manual seed, then CUDA has non-reproducible algorithms that, depending on what kind of graphic card you use, produces a different output. How is this actually handled? Is the seed somewhere written down and kept the same? If you define your manual seed, and if it's in the scikit-learn object, it will be saved in OpenML as adjacent flows. So you will see that user defined that seed. But I know what seed you're talking about. If you talk about neural networks, it's much harder to do with them. And in OpenML, we try to make it as reproducible as possible. But even we can't ensure 100% reproducibility because, like I said, if you create a custom class, I really don't know what you did in that. And if you are not uploading your code somewhere with openable flow and data, then it's much harder to get it. So definitely with these random seeds, we can't ensure reproducibility. Or especially with these CUDA custom seeds. Okay. Well, one minute left probably till the end. Very short question maybe? No? Okay. Then we should probably all say thank you to Pravan for the preparation of the tutorial.