Tools that help you get your experiments under control

Speaker 1 [00:03]

Thank you very much, Alan, for the warm welcome. May I ask maybe for people to defragment a little bit because there are some people that still want to join, so move together a little bit. Unfortunately, we had some technical problems this morning. Fortunately, that means I had no time to be nervous.

Speaker 2 [00:20]

Tools that help you.

Speaker 1 [00:22]

you get your experiments under control.

Speaker 2 [00:24]

is the title of the talk.

Speaker 1 [00:26]

of the talk as you've seen it

Speaker 2 [00:27]

You've seen it in the program, but as it goes, the essence.

Speaker 1 [00:28]

in the program. the essence of what one wants to say sometimes only appears when you

Speaker 2 [00:33]

when you actually start doing the work.

Speaker 1 [00:34]

doing the work. So in retrospect, I think...

Speaker 2 [00:37]

Reproducibility in Theory and in

Speaker 1 [00:39]

theory, and in practice would have been the more fitting title.

Speaker 2 [00:42]

title for this talk because I want to talk about

Speaker 1 [00:44]

talk about how can we as data scientists work in a

Speaker 2 [00:47]

his work in a more reproducible

Speaker 1 [00:48]

a more reproducible manner, and how

Speaker 2 [00:49]

And how can tools support that?

Speaker 1 [00:50]

can As Alan said, I'm a freelance data scientist.

Speaker 2 [00:55]

scientist if that interests you.

Speaker 1 [00:56]

that interests you, read.

Speaker 2 [00:57]

reach out to me. The slides for this presentation are already on my GitHub.

Speaker 1 [01:01]

on my GitHub, and I also have this link again.

Speaker 2 [01:03]

I'll have this link again on the final slide. Consider the following situation. You just started a new data science project.

Speaker 1 [01:14]

You finally got access to the data and you're super.

Speaker 2 [01:14]

You f***er. and you're super excited to get started.

Speaker 1 [01:19]

and you start making your

Speaker 2 [01:20]

You start making your first exploration.

Speaker 1 [01:21]

We have the first explorations, a few plots, everything looks really nice, there seem to be no obvious problems.

Speaker 2 [01:24]

no obvious problem with the data.

Speaker 1 [01:25]

problem with the data, you've

Speaker 2 [01:26]

You're really itching to get that.

Speaker 1 [01:27]

to get that first simple model trained, and that looks promising too. So you go, you make more features. Maybe you go back and do some more data cleaning. And maybe you try out more.

Speaker 2 [01:40]

Maybe you'll try out a more complicated model. And it's all a bit...

Speaker 1 [01:42]

It's all a bit chaotic, but yeah, I mean, you're moving.

Speaker 2 [01:45]

mean you're moving quickly it's fun there's lots of progress but in the back of your mind you get this little nagging feeling maybe maybe there's a bit too much chaos

Speaker 1 [01:54]

Too much chaos.

Speaker 2 [01:55]

I don't really take I'm not

Speaker 1 [01:56]

really taking good notes of what I'm doing. I'm not sure that in a few weeks I'll remember what I did.

Speaker 2 [02:02]

what I did, what worked, what didn't work.

Speaker 1 [02:06]

So maybe you start doing something like this

Speaker 2 [02:08]

something like this. You have a false

Speaker 1 [02:08]

You have a folder where you have pickled files of all.

Speaker 2 [02:11]

files of all your models.

Speaker 1 [02:11]

all your models together with some JSON information, some metadata about it.

Speaker 2 [02:16]

data about your experiments.

Speaker 1 [02:19]

Or maybe you'll start filling out.

Speaker 2 [02:20]

start filling out a spreadsheet like that with information about your experiments. Or maybe you go all fancy.

Speaker 1 [02:25]

fancy and build a little

Speaker 2 [02:26]

and build a little web app for your experiment management.

Speaker 1 [02:30]

of those things kind of sound familiar to you, then this is your talk.

Speaker 2 [02:33]

then this is your talk.

Speaker 1 [02:36]

Let's look a bit closer.

Speaker 2 [02:37]

closer at this nagging feeling.

Speaker 1 [02:40]

feeling. What is behind this?

Speaker 2 [02:41]

What is behind this nagging feeling?

Speaker 1 [02:42]

nagging feeling that maybe there's a bit too much chaos.

Speaker 2 [02:45]

chaos.

Speaker 1 [02:47]

For me, a big reason

Speaker 2 [02:48]

A big reason is pride. I call myself...

Speaker 1 [02:50]

call myself a data scientist. It means I should be working in a science.

Speaker 2 [02:54]

be working in a scientific manner. I should take...

Speaker 1 [02:57]

Take good notes of my experiments.

Speaker 2 [02:59]

I should be able to reproduce my results.

Speaker 1 [02:59]

I should.

Speaker 2 [03:02]

And honestly, I worry.

Speaker 1 [03:04]

I worry that I did something stupid while moving.

Speaker 2 [03:04]

I worry. stupid while moving quickly.

Speaker 1 [03:06]

And if I can't go back and check because I've already changed it.

Speaker 2 [03:10]

changed that code, for example, that it

Speaker 1 [03:12]

If I didn't do something stupid, there'll be...

Speaker 2 [03:12]

didn't there'll be lots of worry.

Speaker 1 [03:16]

And that's sad because I want to trust my.

Speaker 2 [03:18]

my results. And I want that other

Speaker 1 [03:19]

And I want that others can try.

Speaker 2 [03:20]

Others can trust my results.

Speaker 1 [03:23]

And really, how can there be teamwork if other people in my team aren't?

Speaker 2 [03:28]

in my team aren't able

Speaker 1 [03:29]

aren't able to take my models.

Speaker 2 [03:29]

to take reproduce them.

Speaker 1 [03:32]

Reduce them and then add

Speaker 2 [03:33]

and then add their own ideas. As you can see,

Speaker 1 [03:38]

reproducibility as it is at the

Speaker 2 [03:38]

reproducing is very easy. that is at the core of all of these feelings.

Speaker 1 [03:44]

So today, I want to build some kind of framework.

Speaker 2 [03:48]

Framework of reproducibility.

Speaker 1 [03:49]

reproducibility for

Speaker 2 [03:50]

for data science in the theory part.

Speaker 1 [03:52]

theory part. And then I'll look specifically at some tools to tell you how they can help.

Speaker 2 [03:56]

they can help you achieve that.

Speaker 1 [03:58]

that. Before I do that, I'll have

Speaker 2 [03:59]

I'll have to do.

Speaker 1 [04:00]

to look a little bit at, narrow down the scope of what I want to talk to you about a little bit,

Speaker 2 [04:08]

it just so that we're all on the same page.

Speaker 1 [04:08]

just so that we all can see it. Everything I'm talking about today is kind of

Speaker 2 [04:14]

today is kind of guided by my experiences as

Speaker 1 [04:16]

experiences as a data scientist, if you can't find yourself here at all, and if you

Speaker 2 [04:20]

at all, and if you were completely different,

Speaker 1 [04:21]

work completely differently, I would love

Speaker 2 [04:22]

differently, I would love to hear from you.

Speaker 1 [04:23]

to hear from you get your perspective the situation I drew earlier is one of

Speaker 2 [04:31]

of model exploration here on the left hand side.

Speaker 1 [04:31]

model explore

Speaker 2 [04:34]

We want to try out many different models, many different features, different settings for parameters.

Speaker 1 [04:39]

But we probably will be using only one data set because we...

Speaker 2 [04:43]

because we need something to compare on.

Speaker 1 [04:45]

compare on. And errors are expected to happen. I mean, okay, it's annoying, this experiment failed, but okay, I'll fix it.

Speaker 2 [04:53]

Hey, I'll fix that bug and I'll restart.

Speaker 1 [04:58]

What I really value is

Speaker 2 [04:59]

you is flexibility I don't want

Speaker 1 [05:00]

I don't want to be held back by any tooling that makes me move less fast.

Speaker 2 [05:05]

Less fast, less freely.

Speaker 1 [05:07]

Contrast that with the other side.

Speaker 2 [05:08]

with the other side, when you've now found a nice model.

Speaker 1 [05:10]

a nice model.

Speaker 2 [05:12]

model type is fixed, your features parameters are all fixed.

Speaker 1 [05:15]

But it's likely that you'll have to retrain frequently.

Speaker 2 [05:17]

retrain frequently, for example, nightly if new data is coming in. You really don't want errors to happen

Speaker 1 [05:23]

Because that means that somebody will get a call at

Speaker 2 [05:23]

because that means... get a call at 4 a.m.

Speaker 1 [05:26]

At 4 a.m. in the morning.

Speaker 2 [05:27]

in the morning and has to go and fix something.

Speaker 1 [05:29]

So your system should be reliable and it should.

Speaker 2 [05:31]

and it should be well monitored.

Speaker 1 [05:33]

And I think the rigidity of something like a pipeline system can be really helpful.

Speaker 2 [05:37]

can be really helpful in this phase.

Speaker 1 [05:40]

Of course, no data science project has just one of each of these, but you're often moving between.

Speaker 2 [05:45]

between these different phases. And today...

Speaker 1 [05:48]

I'll be concentrating on the left.

Speaker 2 [05:50]

on the left face.

Speaker 1 [05:54]

The second scope thing is a little bit on terminology.

Speaker 2 [05:56]

A little bit on terminology.

Speaker 1 [05:57]

technology. Together with reproducibility, you'll all

Speaker 2 [05:58]

Usability, you'll also find the words repeatability or replicability.

Speaker 1 [06:02]

replicability. These range

Speaker 2 [06:03]

These range in meaning from...

Speaker 1 [06:06]

I can redo what I just did.

Speaker 2 [06:07]

to I can write up

Speaker 1 [06:08]

I can write up what I did for.

Speaker 2 [06:09]

up what I did, for example, in a scientific paper.

Speaker 1 [06:11]

paper somebody else can implement it and we'll get

Speaker 2 [06:13]

and will get the same results. Unfortunately, these terms aren't used the same in different fields.

Speaker 1 [06:20]

So what is called reproducibility in one field might be called replicability.

Speaker 2 [06:23]

called replicability.

Speaker 1 [06:24]

in another field. Today, I've chosen...

Speaker 2 [06:27]

I've chosen the word reproducibility simply because I've

Speaker 1 [06:29]

because I've heard

Speaker 2 [06:30]

heard it in use in data science a lot already.

Speaker 1 [06:33]

And today it shall mean the ability to obtain the exact

Speaker 2 [06:37]

The exact same model with reasonable efforts.

Speaker 1 [06:40]

There's two somewhat undefined words here with what is a same model and what are real.

Speaker 2 [06:44]

and what are reasonable efforts. And we'll talk about both of these things. Let's say you wanted to obtain the...

Speaker 1 [06:52]

obtain the exact

Speaker 2 [06:52]

exact same model.

Speaker 1 [06:54]

With that, I mean, if you pickle the two models.

Speaker 2 [06:57]

the files will be bitwise the same. Well, obviously, you need the exact same data, the exact same code.

Speaker 1 [07:05]

same code, the exact

Speaker 2 [07:06]

the exact same command line or environment variables.

Speaker 1 [07:10]

If your model uses randomness,

Speaker 2 [07:11]

randomness in any way.

Speaker 1 [07:12]

in any way like a

Speaker 2 [07:13]

like a random forest or a random...

Speaker 1 [07:14]

or random weight initialization in the neural network, you will have to set exact...

Speaker 2 [07:19]

exact same random seed different versions of

Speaker 1 [07:23]

of, for example, scikit-learn might differ

Speaker 2 [07:26]

might differ slightly in their implementations.

Speaker 1 [07:29]

So you need to use the exact same versions.

Speaker 2 [07:31]

the same versions of your language libraries.

Speaker 1 [07:34]

Different versions of Python, for example, differ also in

Speaker 2 [07:36]

will differ also in their behavior.

Speaker 1 [07:36]

their behavior. Look at the difference in floating point behavior between

Speaker 2 [07:39]

between Python 2 and 3.

Speaker 1 [07:41]

So you want the exact same version.

Speaker 2 [07:43]

same version of your Python interpreter. You might also want the exact

Speaker 1 [07:47]

the exact same version of any kind of

Speaker 2 [07:49]

of any kind of system libraries like CUDA.

Speaker 1 [07:49]

system. like Judah.

Speaker 2 [07:51]

or your operating system.

Speaker 1 [07:52]

rating system or the hardware the experiment

Speaker 2 [07:54]

where the experiment is run on.

Speaker 1 [07:56]

I mean, you kind of already think like the further we go down.

Speaker 2 [07:58]

we go down in this hierarchy, the less it does seem to matter, right?

Speaker 1 [08:02]

We expect Psyched Learned to be

Speaker 2 [08:03]

We learned to be

Speaker 1 [08:05]

robust

Speaker 2 [08:05]

robust against different system libraries.

Speaker 1 [08:06]

experience different system libraries against...

Speaker 2 [08:08]

against difference operating system.

Speaker 1 [08:09]

rating system. We expect our models to be

Speaker 2 [08:11]

to be robust against different random Cs.

Speaker 1 [08:14]

random seeds, because otherwise they would be kind of

Speaker 2 [08:15]

be kind of shitty models.

Speaker 1 [08:16]

shitty. So, today, let's concentrate only on this.

Speaker 2 [08:21]

on these parts of the hierarchy.

Speaker 1 [08:23]

And let's try to obtain not the exact same model, just the same.

Speaker 2 [08:24]

And let's... just the same model. In practice, however,

Speaker 1 [08:33]

I think it's rarely going to be.

Speaker 2 [08:35]

It's rarely going to be necessary that you actually need to reproduce something. What...

Speaker 1 [08:41]

And what is important is that you can if you need to. A little bit like backups. What is important in your day-to-day life?

Speaker 2 [08:50]

day-to-day life as a data scientist is more the ability

Speaker 1 [08:52]

It's more the ability to compare models.

Speaker 2 [08:54]

I've called the information.

Speaker 1 [08:54]

I've called the... needed for comparing models here on the left-hand side provenance I'm not super sure about that word again what

Speaker 2 [09:02]

Again, would appreciate some feedback.

Speaker 1 [09:02]

some feedback

Speaker 2 [09:06]

You see how the

Speaker 1 [09:07]

the information on the left-hand side.

Speaker 2 [09:09]

hand side aligns with the information on the right hand side. So, for example,

Speaker 1 [09:12]

So, for example, if you wanted to compare

Speaker 2 [09:13]

compare two models. You don't

Speaker 1 [09:15]

You don't care to look at the...

Speaker 2 [09:16]

to look at the actual data that these two models were trained.

Speaker 1 [09:18]

models were trained on. What you care about were they trained on?

Speaker 2 [09:21]

Were they trained on the same data?

Speaker 1 [09:21]

the same data or not. So you'd simply need some kind of identifier and that could be an explicit version or it could be a

Speaker 2 [09:27]

or it could be a hash calculated.

Speaker 1 [09:29]

calculated over

Speaker 2 [09:30]

the data set and how you do this comparison

Speaker 1 [09:34]

comparison. There's an obvious UI for that. It's something like this kind of experiment table.

Speaker 2 [09:40]

experiment table, often called a leaderboard, where you have

Speaker 1 [09:45]

some kind of matter, so it's one row per experiment, and you have some kind of matter information for every experiment.

Speaker 2 [09:50]

every experiment, like here I use the timestamp, as well as these provenance attributes.

Speaker 1 [09:54]

Revenants attributes we saw on the

Speaker 2 [09:55]

we saw on the previous page.

Speaker 1 [09:56]

as well as, of course, any kind of metrics that you.

Speaker 2 [09:58]

metrics that you want to compare your models on.

Speaker 1 [10:03]

I have seen a lot of tools allowing you to give a name for your experiment. I personally don't find that very...

Speaker 2 [10:10]

I don't find that very useful because

Speaker 1 [10:10]

useful because it will quickly end up filling up with useless information.

Speaker 2 [10:13]

useless information like this.

Speaker 1 [10:17]

I like to think of my experiment as being characterized

Speaker 2 [10:20]

characterized by their

Speaker 1 [10:20]

by their combinations of provenance at

Speaker 2 [10:23]

provenance attributes.

Speaker 1 [10:23]

So I want to sort, I want to filter, I want to search.

Speaker 2 [10:27]

by these kind of things and not

Speaker 1 [10:28]

things and not by

Speaker 2 [10:29]

Not by a name.

Speaker 1 [10:30]

a name I made up.

Speaker 2 [10:31]

at some point. Let's look a bit more in detail.

Speaker 1 [10:35]

a bit more in detail at these parameters.

Speaker 2 [10:36]

at these parameter columns.

Speaker 1 [10:37]

or columns like parameters,

Speaker 2 [10:38]

parameters are here

Speaker 1 [10:39]

here I mean anything that steers how

Speaker 2 [10:42]

you get from your raw

Speaker 1 [10:43]

from your raw data to your final results.

Speaker 2 [10:45]

to your final result.

Speaker 1 [10:46]

And that might mean steering the way you do your data cleaning, steering the way you do data and perturbation.

Speaker 2 [10:52]

imputation.

Speaker 1 [10:53]

Imputation, steering, how you, if you do oversampling or not.

Speaker 2 [10:57]

oversampling or undersampling steer

Speaker 1 [10:59]

how you do your feature engineering, and of course,

Speaker 2 [11:01]

and of course the hyperparameters that you give to the model.

Speaker 1 [11:03]

model. And there's different ways of getting your parameters

Speaker 2 [11:07]

parameters into the system, of course, command line parameters, environment variables, config

Speaker 1 [11:07]

into this.

Speaker 2 [11:12]

files, or simply having them in code.

Speaker 1 [11:15]

you're going to do that.

Speaker 2 [11:17]

It depends a lot on the tool you're going to use.

Speaker 1 [11:18]

on the tool you're going to use. And it's a thought I'll revisit.

Speaker 2 [11:20]

I'll revisit in the second part of the talk.

Speaker 1 [11:26]

It's also obvious that

Speaker 2 [11:27]

that all the parameter columns and all the matrix columns are extremely project dependent. So there needs to be some flexibility in this UI.

Speaker 1 [11:36]

In fact, I'd even say that at the beginning of your project, you don't.

Speaker 2 [11:40]

You don't, you won't.

Speaker 1 [11:41]

You won't know what kind of metrics you want to compare on at the end of the project, and you won't know which parameters will be important.

Speaker 2 [11:48]

will be important to you at the end of the project.

Speaker 1 [11:51]

And that means that for some...

Speaker 2 [11:52]

that for some older experiments

Speaker 1 [11:53]

All the experiments you have, you will not have.

Speaker 2 [11:55]

have stored relevant parameters. You will not have recorded

Speaker 1 [11:59]

have recorded.

Speaker 2 [12:00]

relevant metrics.

Speaker 1 [12:02]

I think this is a comment.

Speaker 2 [12:03]

a compromise, a pragmatic compromise that we need to

Speaker 1 [12:06]

that we need to make here that is unavoidable if you don't want to get into some kind of analyzes, paralyzes.

Speaker 2 [12:11]

is paralyzed at the very beginning.

Speaker 1 [12:12]

at the very beginning of the project or you want to set up everything.

Speaker 2 [12:14]

to set up everything perfectly from the beginning. The troubles now start when the left-hand side,

Speaker 1 [12:21]

side, the provenance, and the right hand.

Speaker 2 [12:22]

the provenance, on the right-hand side, reproducibility.

Speaker 1 [12:24]

Get out of sync.

Speaker 2 [12:25]

meaning what is in the table

Speaker 1 [12:26]

What is in the table doesn't.

Speaker 2 [12:27]

doesn't reflect the experiment that you've actually ran.

Speaker 1 [12:30]

And that can easily happen.

Speaker 2 [12:32]

Let's say you run an experiment

Speaker 1 [12:35]

made some changes, started it again, and oops, you forgot to do your game.

Speaker 2 [12:39]

to do your git commit of the new code.

Speaker 1 [12:42]

And if your provenance is the most recent case,

Speaker 2 [12:45]

most recent git commit hash.

Speaker 1 [12:46]

git commit hash, then of course there will be wrong information in that table. It does not reflect the actual experiment.

Speaker 2 [12:51]

experiment that was run.

Speaker 1 [12:52]

That was run, and your experiment will not be reproducible. Or let's say...

Speaker 2 [12:57]

you installed the newest version of scikit-learn

Speaker 1 [12:59]

so I could learn because it implements that fancy new algorithm.

Speaker 2 [13:01]

that fancy new algorithm you wanted to try out.

Speaker 1 [13:04]

Ah, but you forgot to...

Speaker 2 [13:06]

to update your requirements file again.

Speaker 1 [13:08]

The provenance information will be wrong and your...

Speaker 2 [13:11]

and your experiment will not be reproducible.

Speaker 1 [13:16]

Or you made some changes to your data set but forgot to calculate.

Speaker 2 [13:19]

calculate a hash over the new data set.

Speaker 1 [13:22]

data set. Again, your provenance information will be wrong.

Speaker 2 [13:24]

will be wrong, experiment will not be reproducible. So there's really a lot to remember for every experiment, right?

Speaker 1 [13:31]

every experiment, right? Need to ensure using the right data hash. You need to ensure that there's no local changes in

Speaker 2 [13:36]

changes in my Git repository.

Speaker 1 [13:37]

my Git repository. You need to ensure my requirements are up-to-date and

Speaker 2 [13:41]

I need to ensure I'm using the correct Python version.

Speaker 1 [13:45]

And it's too much responsibility. I don't want to have to remember.

Speaker 2 [13:48]

have to remember all of these things.

Speaker 1 [13:49]

Well, that's okay. I mean, we can tell a tool to do it.

Speaker 2 [13:49]

But that's okay. to do it for us.

Speaker 1 [13:54]

But I'm sure a lot of you are already thinking right now.

Speaker 2 [13:57]

Ah, that's a lot of hassle, like every-

Speaker 1 [13:59]

every time i want to run experiment that thing is going to nag me to commit my local code it's really annoying i want my tools to support me not to hold

Speaker 2 [14:08]

not to hold me back.

Speaker 1 [14:10]

Now you're going to be with me.

Speaker 2 [14:13]

One thing I've seen tools do is what I call the zip mode, where instead.

Speaker 1 [14:17]

Instead of using Git as your provenance and reproducibility source of information, you upload the actual...

Speaker 2 [14:26]

the actual source code directory and store it in the experiments database.

Speaker 1 [14:31]

I personally am not a super fan of this because number one, you're losing like the provenance, like the comparing different experiences.

Speaker 2 [14:37]

different experiments aspects.

Speaker 1 [14:39]

as well as like what are you going to do

Speaker 2 [14:41]

going to do with this isolated zip file?

Speaker 1 [14:43]

You have no history.

Speaker 2 [14:44]

You have no history.

Speaker 1 [14:45]

You don't have any of the nice tooling that gives you that.

Speaker 2 [14:47]

that kid gives you

Speaker 1 [14:48]

It gives you to compare it to a different.

Speaker 2 [14:50]

to a different version of your code. Instead, I think your tool should support a development mode, which basically is just best effort reproducibility, like as much hassle as you can stand to live with in your day-to-day life.

Speaker 1 [15:05]

And this is for any kind of just trying out type of experiments. And you might have a zip mode or you might not upload code.

Speaker 2 [15:11]

upload code at all.

Speaker 1 [15:12]

at all. The important thing is these experiments are still in your leaderboard.

Speaker 2 [15:16]

In your leaderboard view, you can still compare.

Speaker 1 [15:18]

compare them with other experiments you still have all the

Speaker 2 [15:21]

you still have all the nice tooling.

Speaker 1 [15:21]

nice tooling available to identify

Speaker 2 [15:24]

how

Speaker 1 [15:24]

and how they relate to other experiences.

Speaker 2 [15:26]

relate to other experiments, and really dig down and see different plots. As soon as you have found a nice model, you're too late.

Speaker 1 [15:34]

Your tool should allow you to switch into reproducibility

Speaker 2 [15:36]

reproducibility mode.

Speaker 1 [15:37]

where it really does

Speaker 2 [15:39]

It really then enforces

Speaker 1 [15:39]

And enforce it.

Speaker 2 [15:40]

that you are using the correct data and forces that you have no local

Speaker 1 [15:43]

you have no local changes.

Speaker 2 [15:44]

changes and forces that your requirements are up to date. And it's important that

Speaker 1 [15:50]

that in your UI, you can visually...

Speaker 2 [15:52]

distinguish between those two modes. So this is a screenshot from

Speaker 1 [15:56]

Screenshot from a software as a service.

Speaker 2 [15:58]

for as a service solution called

Speaker 1 [15:58]

solution called Valohai.

Speaker 2 [16:00]

where they use this nice little

Speaker 1 [16:01]

This nice little lab icon to identify.

Speaker 2 [16:03]

to identify what they call an ad hoc experiment.

Speaker 1 [16:12]

But let's drive this thing with the mobile.

Speaker 2 [16:14]

thing with the modes a bit further.

Speaker 1 [16:15]

Oh, it's a bit further.

Speaker 2 [16:16]

So, sorry, consider the third row here.

Speaker 1 [16:16]

Ha! row here. I'll call it reproducibility.

Speaker 2 [16:20]

Reproducibility++ mode.

Speaker 1 [16:22]

mode where you're not actually running the experiment on your local machine but you're running it on some remote

Speaker 2 [16:28]

remote machine. And this

Speaker 1 [16:28]

machine. And this gives you the added reproducibility bonus that you

Speaker 2 [16:32]

that you know your code.

Speaker 1 [16:33]

your code can run

Speaker 2 [16:34]

can run somewhere else, you know?

Speaker 1 [16:34]

somewhere You know you're not using some random file somewhere on the file system that you at some point linked to your project.

Speaker 2 [16:40]

to your project.

Speaker 1 [16:43]

It also has benefits if you need some special hardware like a GPU.

Speaker 2 [16:47]

like a GPU that you don't have on your local machine.

Speaker 1 [16:49]

So you start developing locally.

Speaker 2 [16:49]

So you start...

Speaker 1 [16:51]

As soon as you need to, you'll send this area.

Speaker 2 [16:54]

send this air experiment somewhere remote. We can drive that even further. Why not something similar to

Speaker 1 [17:00]

to CI continuously.

Speaker 2 [17:01]

continuous integration mode

Speaker 1 [17:03]

where it's not actually I who says this is an experiment I want to be recorded,

Speaker 2 [17:03]

where

Speaker 1 [17:09]

but rather I work on it.

Speaker 2 [17:10]

work on a different branch and as soon as I'm

Speaker 1 [17:11]

And as soon as I'm ready, I'll open a merge request or a pull request.

Speaker 2 [17:14]

or a poll request and the CI will

Speaker 1 [17:16]

I will pick up on it and run.

Speaker 2 [17:17]

and run my experiment on a remote machine. I've seen one software...

Speaker 1 [17:22]

Software as a Service Solution.

Speaker 2 [17:23]

so have a solution do something

Speaker 1 [17:23]

do something like that.

Speaker 2 [17:24]

like that that's called gradient and it that

Speaker 1 [17:26]

And it does similarly, like a CI system, plug into GitHub and directly...

Speaker 2 [17:31]

and directly tells you if the experiment worked or failed.

Speaker 1 [17:34]

Unfortunately, I have to tell you, this didn't really work.

Speaker 2 [17:37]

But in principle, I really like this setting.

Speaker 1 [17:41]

I'm not sure if this could be, should be called like continuous evaluation, continuous experimentation. Both terms I've actually.

Speaker 2 [17:47]

terms I've actually already seen.

Speaker 1 [17:48]

already seen other people use in data science.

Speaker 2 [17:49]

use in data science literature, meaning something completely different. So let's see.

Speaker 1 [17:52]

So let's see what will prevail.

Speaker 2 [17:54]

prevail. The question now is

Speaker 1 [17:58]

now is, how do you want to work? Which

Speaker 2 [18:01]

to work. Which of those modes

Speaker 1 [18:01]

Which of those modes works for you?

Speaker 2 [18:04]

How much hassle can you stand in exchange for the freedom from worrying?

Speaker 1 [18:17]

So, you might think, okay, let's tell me a little bit about some tools that are...

Speaker 2 [18:22]

tools that are available. There's a lot.

Speaker 1 [18:25]

This is all the software as a service solutions I found that try to do that. And there's new ones popping up all the time.

Speaker 2 [18:30]

all the time, as well as all of these open source solutions. Again, new ones popping up all of the time. I'll concentrate on the open source.

Speaker 1 [18:39]

the open source ones today. Let's remove all the ones that did not have...

Speaker 2 [18:44]

not have a git commit in the last year.

Speaker 1 [18:49]

So we left with these ones and then I picked the ones that

Speaker 2 [18:52]

that I've heard the most buzz about from other data scientists.

Speaker 1 [18:57]

And that is MLflow, Sacred, and...

Speaker 2 [19:00]

and DVC.

Speaker 1 [19:01]

Both MLflow, so there will be a tutorial on MLflow tomorrow morning.

Speaker 2 [19:04]

tomorrow morning.

Speaker 1 [19:05]

as well as another talk that does seem

Speaker 2 [19:07]

talk that does seem quite similar to my talk.

Speaker 1 [19:09]

my talk that also mentions DVC.

Speaker 2 [19:11]

DVC tomorrow morning.

Speaker 1 [19:12]

tomorrow morning, unfortunately, at the same time.

Speaker 2 [19:14]

at the same time.

Speaker 1 [19:16]

So if you want to get some more details than I'll give you today, please.

Speaker 2 [19:19]

please check out those events. Let's talk about Sacred. That's a library that you just simply pip install. You create your experience.

Speaker 1 [19:32]

You create your experiments, then you have the publicist's ability to add a research

Speaker 2 [19:35]

add a resource.

Speaker 1 [19:36]

source. In this case, I use

Speaker 2 [19:37]

In this case, I use my data set, and that means a hash is...

Speaker 1 [19:39]

a hash is calculated over that dataset and attached to it.

Speaker 2 [19:42]

to the experiment.

Speaker 1 [19:44]

You can log any kind of metrics as well as add additional artifacts. Here I have a plot of the predictions I made so that I really can dig deep.

Speaker 2 [19:52]

really can dig deeper into how my model is performing.

Speaker 1 [19:55]

performing. What I really like about Sacred is this idea of a conflict.

Speaker 2 [20:01]

of a config where

Speaker 1 [20:02]

where config is the combinations of different parameter values. And I found that there's some really nice automagic thing happening here.

Speaker 2 [20:11]

that I defined this

Speaker 1 [20:14]

config name, config call

Speaker 2 [20:16]

config called baseline.

Speaker 1 [20:17]

And then Sacred makes it so that I can just call my training.

Speaker 2 [20:20]

all my training script with the extra command line parameter.

Speaker 1 [20:22]

command line parameters with

Speaker 2 [20:23]

with baseline.

Speaker 1 [20:25]

And I can just go to my colleagues and say, if you want to run the baseline, just...

Speaker 2 [20:28]

Just run this command.

Speaker 1 [20:30]

command. And I found that really convenient to work with.

Speaker 2 [20:32]

to work with.

Speaker 1 [20:33]

However, a bit limiting when I wanted to do.

Speaker 2 [20:34]

when I wanted to do grid search.

Speaker 1 [20:38]

Sacred doesn't come with its own leaderboard UI, but there are several open source projects that build on top of the Sacred database.

Speaker 2 [20:46]

Here is a screenshot of something called

Speaker 1 [20:46]

Here's a screenshot. of something called Omniboard. I haven't looked at the other ones. It's perfectly usable. Like UI, you can click on each experiment and see some details.

Speaker 2 [20:55]

You can see some details, it's very nice.

Speaker 1 [20:56]

It's very nice. One thing I missed here.

Speaker 2 [20:58]

missed here is the ability

Speaker 1 [20:59]

is the ability to display parameters.

Speaker 2 [21:02]

parameters as columns. So you can't really filter by different settings that you ran your experiments with.

Speaker 1 [21:09]

And I like to use that to give me

Speaker 2 [21:11]

to give me some

Speaker 1 [21:11]

some ideas on how to progress, what kind of

Speaker 2 [21:14]

what kind of other experiments I should be running next sacred supports both

Speaker 1 [21:20]

both development as well as

Speaker 2 [21:22]

as well as a reproducible

Speaker 1 [21:23]

well as a reproducibility mode. However, you do have to remember to set a specific flag.

Speaker 2 [21:29]

flag to enforce reproducibility.

Speaker 1 [21:30]

reproducibility mode. Otherwise, it will not complain if you're

Speaker 2 [21:33]

if you're a local source.

Speaker 1 [21:34]

local source quote there.

Speaker 2 [21:35]

code directory is.

Speaker 1 [21:37]

is dirty.

Speaker 2 [21:42]

it to reproduce in sacred. There is a

Speaker 1 [21:45]

there is a good amount

Speaker 2 [21:46]

good amount of manual steps necessary.

Speaker 1 [21:49]

I personally don't think this is much of a problem. Like I said,

Speaker 2 [21:54]

I said important is that you can

Speaker 1 [21:56]

You can reproduce if you need to.

Speaker 2 [21:56]

reproduce I think, personally, it's okay.

Speaker 1 [21:58]

I think... it's okay if that includes some manual steps.

Speaker 2 [22:01]

manual steps.

Speaker 1 [22:02]

However, again, with the backup analogy, if it's annoying to try out your backup,

Speaker 2 [22:08]

backup, you're probably going to do it, right?

Speaker 1 [22:10]

to do it, right? So you're not really sure.

Speaker 2 [22:12]

if you can reproduce or not.

Speaker 1 [22:13]

reduce or not if you haven't actually gone through

Speaker 2 [22:15]

gone through these steps at least once.

Speaker 1 [22:16]

this. MLflow is actually somewhat similar.

Speaker 2 [22:22]

somewhat similar, again it's a

Speaker 1 [22:23]

Again, it's a Python package. In this case, you have to menu.

Speaker 2 [22:27]

You have to manually register any parameters you want to have displayed.

Speaker 1 [22:31]

which Seiko did

Speaker 2 [22:32]

sacred did for you automatically.

Speaker 1 [22:35]

What I really like about

Speaker 2 [22:37]

about MLflow is the way

Speaker 1 [22:39]

way that it unfolds.

Speaker 2 [22:39]

that... That it enforces.

Speaker 1 [22:40]

that your local environment, like virtual environment or condo environment, is clean.

Speaker 2 [22:46]

clean.

Speaker 1 [22:46]

in a super non-intrusive way.

Speaker 2 [22:48]

intrusive way, like you're

Speaker 1 [22:49]

like you don't have to do anything and it just checks that it's it's up to date and it's

Speaker 2 [22:53]

up-to-date and installs it's necessary I found that really really nice

Speaker 1 [22:56]

But really, really nice.

Speaker 2 [22:58]

Another thing I found nice is this idea.

Speaker 1 [22:59]

nice is this idea of this ML project file, which is somewhat similar to a Docker file in that it defines

Speaker 2 [23:05]

that it defines different entry points for.

Speaker 1 [23:06]

Three points for your...

Speaker 2 [23:08]

for your training process.

Speaker 1 [23:11]

As soon as you add this ML project file to your project,

Speaker 2 [23:13]

project file to your repository.

Speaker 1 [23:13]

to your repository, you can just give somebody the...

Speaker 2 [23:17]

URL of the Git project.

Speaker 1 [23:18]

of the Git project and tell them to do MLflow run URL of that project.

Speaker 2 [23:22]

and it will run and you will

Speaker 1 [23:24]

run and you will get reproduced.

Speaker 2 [23:24]

get reproducible results. So that is also pretty cool.

Speaker 1 [23:30]

MLflow comes with its own.

Speaker 2 [23:32]

its own leaderboard.

Speaker 1 [23:33]

One thing it didn't seem to do automatically is flag whether an experiment failed or did not fail.

Speaker 2 [23:39]

So all of these that are like empty in the last metrics column.

Speaker 1 [23:41]

matrix column there actually failed but it doesn't display it to me in a nice visual manner parameters however are done extremely

Speaker 2 [23:49]

done extremely well in this. So you can, you will have.

Speaker 1 [23:51]

and you will have like a dictionary-like parameters column and just can click on.

Speaker 2 [23:57]

Click on one of these and explode.

Speaker 1 [23:58]

and exploded to being its own column. And I found that really nice and convenient.

Speaker 2 [24:02]

really nice and convenient to work with.

Speaker 1 [24:06]

MLflow does have a development as

Speaker 2 [24:08]

as well as a reproducibility mode, however...

Speaker 1 [24:14]

It does, if you do not

Speaker 2 [24:15]

If you do not MLflow run some URL, but rather do MLflow run your local directory,

Speaker 1 [24:21]

It will not enforce you to commit your local changes. And there is no way to make it do it. And it will record the old commit.

Speaker 2 [24:30]

the old commit hash with a...

Speaker 1 [24:30]

which is now outdated, so you will have...

Speaker 2 [24:33]

So you will have wrong information in your experiments database.

Speaker 1 [24:36]

So be aware of that. You can already guess being able to reproduce is much easier in MLflow. You need to obtain the data in some way. MLflow does not care about data.

Speaker 2 [24:49]

about data at all, and then it's just run.

Speaker 1 [24:50]

then it's just running one simple command. The final one I want to look at is...

Speaker 2 [24:57]

want to look at is DVC data version control.

Speaker 1 [24:59]

version control. And you already heard in the name where the

Speaker 2 [25:01]

where the focus of this tool lies.

Speaker 1 [25:06]

you get

Speaker 2 [25:06]

Not a Python library, but rather a command line.

Speaker 1 [25:07]

another a command line uh

Speaker 2 [25:10]

a tool that has an interface quite similar to Git.

Speaker 1 [25:14]

And you use that to manage your data. So at the beginning of the project, you would add your data to your repository. And then DVC works in a pipeline.

Speaker 2 [25:27]

In a pipeline manner, meaning you set up a pipeline.

Speaker 1 [25:30]

So set up a pipeline from your raw data to your final results.

Speaker 2 [25:34]

And then when anything changes,

Speaker 1 [25:35]

changes, if you change any file that is included in that pipeline, you can just use one simple

Speaker 2 [25:40]

one simple command to rerun everything that is necessary.

Speaker 1 [25:44]

I urge you to read the documentation like the Getting Started.

Speaker 2 [25:48]

the Getting Started Guide for DDC. It's really, really nice.

Speaker 1 [25:50]

Really, really nicely written. Really impressive. One thing DVC does not try to solve is...

Speaker 2 [25:57]

to solve is a leaderboard what you if you

Speaker 1 [26:00]

If you want to compare your models,

Speaker 2 [26:02]

models, that is the only thing that you get.

Speaker 1 [26:03]

that you get, a very...

Speaker 2 [26:04]

A very simple text.

Speaker 1 [26:05]

Simple text view. There's no idea of like recording permit.

Speaker 2 [26:08]

parameters or something.

Speaker 1 [26:08]

or something like that, there's no certain

Speaker 2 [26:10]

There is no sorting, searching, filtering, nothing like that.

Speaker 1 [26:15]

which also means that a development mode is created.

Speaker 2 [26:17]

and mode is kind of a moot point.

Speaker 1 [26:19]

point. And as you can see, DVC really concentrates.

Speaker 2 [26:24]

really concentrates on data and code,

Speaker 1 [26:27]

doesn't really care about any of

Speaker 2 [26:28]

about any of the aspects of reproducibility.

Speaker 1 [26:33]

one thing I see a bit critical here

Speaker 2 [26:36]

critical here is that you really need to make sure that

Speaker 1 [26:38]

to make sure that your pipeline

Speaker 2 [26:39]

that your pipeline is set up correctly.

Speaker 1 [26:41]

If you add a new data.

Speaker 2 [26:42]

Add a new dataset or

Speaker 1 [26:42]

set or fire.

Speaker 2 [26:43]

file. Or if you

Speaker 1 [26:44]

if you add a new script,

Speaker 2 [26:45]

add a new script file and you'd

Speaker 1 [26:46]

script file and you do not include that in the pipeline, then DVC will not know that it needs to rerun something if you

Speaker 2 [26:53]

something if you changed that file.

Speaker 1 [26:55]

So there's definitely a need for code.

Speaker 2 [26:57]

a need for code review so that you review other people's pipelines.

Speaker 1 [27:04]

So reproducing with DVC is the most straightforward for all of the tools I've showed you because you don't even have to get the data. DVC does all of this.

Speaker 2 [27:15]

That's all of the things for you.

Speaker 1 [27:18]

And the question now, again.

Speaker 2 [27:19]

Now, Aganis, how do you want to work? I think these tools...

Speaker 1 [27:24]

kind of reflect how their developers want to work as a data scientist.

Speaker 2 [27:28]

scientist, that

Speaker 1 [27:29]

Does that match how you want to work?

Speaker 2 [27:31]

Maybe one of the other tools out there.

Speaker 1 [27:33]

other tools out there are the right ones for you. I hope that with the framework that I've given you,

Speaker 2 [27:39]

That you are able to go and evaluate.

Speaker 1 [27:41]

and evaluate how well

Speaker 2 [27:44]

the tool does on reproducibility.

Speaker 1 [27:45]

reproducibility and how

Speaker 2 [27:46]

and how well it matches the way that you...

Speaker 1 [27:48]

that you want to work.

Speaker 2 [27:54]

The reason I wanted to give this talk that I did not talk about at the beginning is envy. I do feel envious of software engineers.

Speaker 1 [28:03]

When I look at...

Speaker 2 [28:04]

at their, what seems to me, very much established

Speaker 1 [28:06]

very much established best.

Speaker 2 [28:07]

best practices like code review like

Speaker 1 [28:10]

review, like continuous integration.

Speaker 2 [28:11]

continuous integration, like unit testing, and how they have really nicely usable tools that support them.

Speaker 1 [28:17]

support them.

Speaker 2 [28:18]

in following these best practices.

Speaker 1 [28:23]

I do recognize, however, of course, data.

Speaker 2 [28:24]

Of course, data science is a much younger discipline. I hope that with talks like mine,

Speaker 1 [28:29]

like mine, and other similar

Speaker 2 [28:30]

And other similar talks at this and other conferences. We can have it.

Speaker 1 [28:33]

We can have a discussion about

Speaker 2 [28:35]

What should our best practices be?

Speaker 1 [28:36]

practices be and

Speaker 2 [28:37]

be? And how should tooling look like?

Speaker 1 [28:39]

What does schooling look like to support?

Speaker 2 [28:39]

to support these best practices.

Speaker 1 [28:43]

So the thought I want to leave you with today, the question I want to...

Speaker 2 [28:47]

The question I want to leave you with today is how do.

Speaker 1 [28:48]

is how do we...

Speaker 2 [28:49]

we, as data scientists, want to work. Thank you. Thanks, Kat, for this wonderful talk. Is there anybody who has questions for Kat still? Let's start with you up here.

Speaker 3 [29:15]

Yeah, thank you for a nice talk. You talked a lot about and focused on the reproducibility with these tools. I'm not sure, and I hopefully look it up afterwards, how the tools support data pipelines themselves and distribute the stuff that needs to be done, gathering data and everything. have you seen the talk about Kedro yesterday and can you do you know about Kedro which seems to be another new open source tool about data pipelines and that also might support this kind of reproducibility I'm not sure if that's a goal

Speaker 1 [29:52]

I don't know that, but can I ask you to maybe, I'm sorry, I kind of want to have this a bit more discussion also, learn from you.

Speaker 2 [29:59]

Where do you see the need for

Speaker 1 [29:59]

Where do you see? the need for these data pipelines in this

Speaker 2 [30:04]

in this setting.

Speaker 1 [30:04]

setting.

Speaker 3 [30:05]

Okay, for me, I'm not in this 100% data science workflow because we are training model like on a weekly basis and applying it. So there's more for us, the training data is constantly changing.

Speaker 1 [30:23]

Well, then, I guess, sorry.

Speaker 3 [30:25]

So generating the, okay, this is the training data we are using for training our models, which takes a lot of time, is like a big part of what we need to invest work in. And training the model is rather stable because we have this model that we found out is good and aren't working on it too much.

Speaker 1 [30:46]

too much then i think your use cache actually is following on the right hand side that i did not consider today and i think tooling i'm not saying that there can't be tools that

Speaker 2 [30:54]

There can't be tools that support both.

Speaker 1 [30:55]

support both. But I think tooling for the left-hand side will look, has very different requirements.

Speaker 2 [31:00]

requirements than tooling on the right-hand side.

Speaker 1 [31:03]

Yep. Thank you.

Speaker 2 [31:06]

Who has the next question? Okay.

Speaker 4 [31:13]

Hello, thank you for your talk. I'm not really a data scientist. I'm more a classical scientist using Python for some stuff. And I'm wondering if what you're saying here, which is merely, I think you apply it more to the machine learning and the data computer world thing, in how far you see overlaps or in your search have found connections with classical science experiments, where, as example, massive parts of the flow you do has nothing to do with software but has to do with, I don't know, telescopes or, in my case, electromagnetic fields you measure and your settings and so on. Can you comment? Have you seen some interaction with that world?

Speaker 1 [31:55]

with that world. Yes, definitely.

Speaker 2 [31:57]

Definitely, and you definitely need to.

Speaker 1 [31:58]

Definitely need different, maybe not tooling, different.

Speaker 2 [32:01]

tooling, different protocols to support classic science workflows.

Speaker 1 [32:04]

workflows I do see one difference though that your outcome like your your outcome of a classical science experiment is the often the

Speaker 2 [32:13]

often the essence.

Speaker 1 [32:14]

the essence of what it's

Speaker 2 [32:15]

of what it's all about, right?

Speaker 1 [32:16]

all about, right? For us, the outcome is the model. But in a classical science experiment, you want to have a paper that really describes the

Speaker 2 [32:23]

the essence of your experiments.

Speaker 1 [32:23]

effect. your experiments, what do you draw from

Speaker 2 [32:27]

you draw from this experiment?

Speaker 1 [32:28]

from this experiment and then somebody can take that and implement.

Speaker 2 [32:30]

and implement it on their own, and this is where I kind of see the difference. And we don't really have that, so classic science needs more.

Speaker 5 [32:44]

Thank you for your talk. It was great. And I wanted to ask you that you've been talking about all these tools like DVC MLflow and so on. Don't you think that they kind of cover all the important stuff, but it feels like one of the tools, one of them covers one important thing, another important thing. Can we do something like contribute to those tools or should we like develop something else? because in my practice, when I tried to use all of them, it was like, I need these, but it lacks something another tool has. So I ended up in building something more or less custom, which wasn't really convenient because when you're just one data scientist, you cannot build something really convenient, great in terms of code quality, UI, and all this kind of stuff. So what do you think about that?

Speaker 1 [33:37]

I completely agree with you.

Speaker 5 [33:38]

I,

Speaker 1 [33:39]

None of these tools really covers all.

Speaker 2 [33:41]

covers all the bases.

Speaker 1 [33:43]

So far, the best that we have, all of these are open source tools.

Speaker 2 [33:47]

source tools, so yes, you can contribute.

Speaker 1 [33:50]

I kind of want to take up again this idea of a discussion. Not just me telling you how I work and how that influences what kind of tools I would use, but rather that we have a discussion.

Speaker 2 [34:02]

have a discussion and find how we

Speaker 1 [34:03]

how we work,

Speaker 2 [34:04]

work. what our best practices and our tooling could look like.

Speaker 1 [34:04]

what, tooling could look like. And I don't think we're there yet.

Speaker 2 [34:07]

I don't think we are there yet. I completely agree with you. Maybe you are there. Hi, and thanks for your nice talk.

Speaker 3 [34:20]

Um, I

Speaker 1 [34:21]

I was a little

Speaker 2 [34:22]

I was a little bit wondering...

Speaker 1 [34:22]

bit wondering um

Speaker 2 [34:24]

is there still a place in your

Speaker 1 [34:25]

place in your kind of model

Speaker 2 [34:26]

kind of model exploration development workflow for Jupyter notebook or are you doing everything in like separate scripts because I I think most

Speaker 1 [34:34]

I think most of the tools you showed, they were all kind of working on Python.

Speaker 2 [34:38]

on Python scripts.

Speaker 1 [34:39]

Python scripts, so maybe you can elaborate a little bit on that. Absolutely, I didn't expect that question. I do love Jupyter notebooks as a tool for any kind of exploratory work, for plotting, for visualization. I do not believe that it

Speaker 2 [34:54]

that it is the right to

Speaker 1 [34:55]

Right tool for writing.

Speaker 2 [34:56]

for running reproducible experiments.

Speaker 1 [34:59]

I don't believe that because of issues that you get with out-of-order execution, of scoping issues.

Speaker 2 [35:05]

issues that are inherent

Speaker 1 [35:06]

that are inherent to Jupiter.

Speaker 2 [35:08]

to Jupyter Notebooks.

Speaker 1 [35:09]

So, yes, in my view of this, it's...

Speaker 2 [35:14]

Using notebooks for exploration.

Speaker 1 [35:15]

for exploration and then

Speaker 2 [35:16]

and then start using scripts.

Speaker 1 [35:18]

In fact, one of these libraries, I think Thackert or MLflow, actually forbids you to use them in a notebook.

Speaker 2 [35:18]

In fact, one of these book. It will throw an error if you try to do so. One last question, perhaps? So many people. I've not been in that corner yet.

Speaker 3 [35:36]

Hello, I'm a fellow data scientist, and what we heard a lot of times is that it's very important to document your stuff. I mean, you do experiment, you probably scoped your experiment before, and do you document it in your Git history, do you have a separate file, do you document your Jupyter notebook? i mean you can put your documentation at a lot of places but um i think it's very important if you establish some workflow like this that you have also done an overview what you have done and also to do it in a consistent way and i've tried a lot of ways to do it but i haven't come up with a good solution yeah i would say because always a little bit different i

Speaker 1 [36:28]

I can't offer you a good solution either,

Speaker 3 [36:28]

Thank you.

Speaker 1 [36:32]

especially for this whole writing some kind of report or final document.

Speaker 2 [36:37]

documentation about what you found

Speaker 1 [36:38]

what you found in this project. I'm sorry, I can't answer that part of the question. However, I want to point you again to the Git mode. I'm not searching for it now. Because in that mode, you need to have everything to run that experiment somewhere, right? It needs to be documented in some kind of file that sets up your continuous...

Speaker 2 [36:59]

your continuous experimentation framework.

Speaker 1 [37:02]

And thereby, you do have...

Speaker 2 [37:04]

have documentation.

Speaker 1 [37:05]

documentation by the sheer necessity of having it. So it's not text somewhere, it's in a configuration file. And anybody who wants...

Speaker 2 [37:13]

to manually reproduce on their machine

Speaker 1 [37:15]

their machine can look at that file

Speaker 2 [37:17]

and get all the information there.

Speaker 1 [37:18]

information they need on how

Speaker 2 [37:19]

They need on how to get.

Speaker 1 [37:20]

how to get set up. And I find that a really nice, like implicit documentation benefit.

Speaker 2 [37:24]

implementation benefit that you also see in a lot of software engineering tooling. Thank you very much, Kat. Thanks for the wonderful talk again.