The bumps in the road: A retrospective on my data visualisation mistakes

Speaker 1 [00:02]

Hi, I'm very happy to see you all here. My name is Artyom. I'm a software engineer a day and a physicist at night. Past eight years I've been working with different kind of analytics starting from computational fluid dynamics to business analytics. Today I want to share this story which is a bit the personal one but I think you will find something useful for any project you're making not only the data visualization but that's something that yeah visualization is something that i like uh in the end like i'm very easy to inspire person and yeah initially i thought that visualization is something that is well you just go and you just plot and that's easy as a piece of cake but as you might deduce from the title of my talk it was absolutely not the case the bumps on the road, there are many, and I'm here to share some learnings that I learned along the way. So that's how my journey looked like. And when I look back into my journey, I can say that, well, there were many mistakes, but if I try to summarize them, I could definitely say that I didn't know the rules, how to make the good data visualization, what makes the visualization good. I spent too much time on predefining the graphs without actually adding any value to the stakeholders that I'm creating those graphs. It was a nightmare to version my code because I love Jupyter notebooks. I ended up in a huge directory of various Jupyter notebooks that I cannot manage at some point and it was a nightmare. I was thinking that it's very hard to break data visualization so I learned that okay it's actually possible. I made a mistake when I had to reprocess all the data when you know like all the fire is around and then you just run with a glass of water trying to well to stop this fire from from continuing. I've been in that situation so yeah I learned few things and yeah so i have to reproduce the data and today i'm here to share some learnings that i learned along the way and maybe some practices that could help you to avoid those kind of mistakes but before we start to talk about the specifics let's just talk about visualization in general and as i said like for me when someone says like hey we have some data i mean can you help with visualization I usually very easy to inspire I go and I say yes of course let's get it and let's do it and I have like very colorful images in my head how it should look like but in reality what I've learned is that when someone comes to you and says let's visualize something you have one thing in your head but they have completely different thing in their head and sometimes after spending like hours on working on something you could end up in situation when well basically what you thought you're making a visualization the people for who you make a visualization they have completely different thing in mind so maybe they even don't any visualization at all and in my talk i will show the examples of matplotlib i love matplotlib i think it's widely used in, well, in data science, but it's not restricted to Matplotlib. You'll find the approach is universal, it's object-oriented approach, and you can apply it to any library you use or you like. And when we talk about visualization, actually the first question that I ask myself or the stakeholders that want that visualization is what kind of visualization do they need it can be the static visualization or dynamic visualization static is something that people usually put on reports like a4 paper and they print it like i don't know weekly monthly for their management dynamic is some the dashboard that requires some action from people so they go to some website they can interact with the data yeah they can explore something but that's a two big groups of data that we can think of but yeah remember before coding anything it's very nice to align on your goals and the fastest ways to align for me is to try to get sense of data and maybe to draw something and that actually works well because you can communicate with your partners is it something that you need? For example, if you have something running A-B test, maybe you need to see how the, well, the sales evolve over the time, or if you need to compare, for example, two different algorithms between each other, so probably will you pick some bar chart and see how it would look like. The key point here is that, yeah, having drawn that on a piece of paper can allow you to, well to save some time later and it's also nice to think about the corner cases because for example if you visualize how you wrote the awesome algorithm that performs like in nanoseconds and another algorithm is performed in seconds what you couldn't end up with is just like one huge bar for the poor algorithm and very little bar that is well not visible for your nice algorithm basically you miscommunicate the message so you kind of show that okay everything is okay slow but you don't show what is important your thing that you've been working on and uh again i mean the best visualization library or tool is the one that you know the most so it could be my plot leap or it could be something else but yeah i think at that point is enough hypothetical examples and and let's get to code, and that's my favorite part. And when I start any data visualization project, I start very simple. You could notice few things on the screen. So on a high level, there are two directories. The first one is the .github directory, and the second is the directory for the project we are starting. We will go over each of them. So the PyData Berlin is the directory that contains the data, the plots that are used for this presentation, and the .github folder, it contains the GitHub workflow for CICD thing that I will show you why it's important because in some time the visualization project, tend to become bulky and very unmaintainable but on a high level if you structure like that you'll see the benefits of of doing that and why i like to introduce the continuous visualization or continuous integration in my into my project is because it can work as a test and also what i try to do with my visualization i want every plot that i create to be reproducible And for that, I simply like to put all the plots that I ever create to the single file, which I called gallery.pdf. And the continuous integration, it allows me to create this gallery.pdf file. And as you can see here, it's the kind of plots that I generate for this presentation. And you can find them on the GitHub workflow attached to the, like, on the bottom. So sometimes it's nice to, well, not to introduce it like a separate file, but having them in the code because maybe you'll go and you'll show the GitHub repo. So for that case, I sometimes put some visualization in SVG format to the readme file. That also allows me to track and to showcase what my visualization code is doing. If you remember the first thing that I said that I didn't know like what is good and what is bad for data visualization and in my practice I usually use a few kind of plots it's like a chart plot, line plot, bar plot and tables and the opening iBook that I kind of very thankful that discovered it it was roger scientific visualization python plus mod plot leap it's a very nice book that teaches you the good design principles we can name them one by one now they are self-explanatory but i really advise you to go after and to check out his book he's like it's fantastic one if you use my plot leap so basically the first one is what i've started with it's like knowing your audience is the key, then you need to identify the message, adapt figure to support the medium, captions are not optional, don't trust the defaults, use color efficiently, don't mislead the reader, avoid chart junk, and message trumps beauty, and the final one is they get the right tool. Yeah, all of them they worth the separate talk and actually there are lots of talks around the design principles that you can find online but for me the main focus today is on structuring the project but let's look into action here so let's consider this example of Roger from his book how to make a good data visualization so he creates a deep showing that okay some things they are redundant they just create a confusion and if you try to optimize the graph try to keep in mind the less is more and that's actually the thing that I always keep in my head when I creating the visualization so in reality when you try to show as much data as you can in the end you may end up not showing anything and what I like specifically about that example is that yeah over time you see how it evolved from very complex graph to very simple one. And even without knowing the context you see how author tries to emphasize that something with bacon is going on. So this is the importance of clearing the message for the audience. Usually the line charts are simple, but I don't like the matplotlib defaults, and usually what I start when I take a new library for data visualization, I start to explore with basic things like that, trying to experiment how visualizations are structured and how I should interact with the framework by repeating this basic chart. So here you can see that, okay, it doesn't look like a default matplotlib figure, however However, there are little tweaks that need to be made. And what you could notice is that the title is left-aligned, the labels are directly on the plot, the Y label is 90 degrees rotated, so we don't need to rotate our hand when we read the Y-axis label. Yeah, so that's the first plot I make if I try a new library or something. With matplotlib, it's quite easy because everything in Python is an object. And matplotlib is not an exception. So everything in Python and matplotlib is an object. So you can easily modify their state before you call the figure show method. So let's continue. And actually that's what I showed that's the first thing that I do with my plots. You could notice a few things here. So I simplified the code a bit, so that's easier to consume, but you could notice that I usually like to use the Python files instead of the Jupyter notebooks, and for any plot that I create, I name them as a plot underscore something, and in this case, it's a plot common chart. And then, that's what I said, first you create the two objects, the figure object and the the access object, and then it's object-oriented approach so you can easily modify the attributes of your object to get to the state that is presented here. In business domain, I usually see the things that people, the data is complex, and people tend to use like, okay, on average we have something x. And I see another value of data visualization, And here it's like the point that it's very hard, like it's very easy to mislead the reader. And I recently found a very nice article that shows that you can get the completely different data sets, but the same statistics. And if you play this GIF here, you see that the data set changes, but the averages and all the statistics, they stay the same. So that's another argument of using the data visualization is not to mislead the reader. so and don't mislead yourself so usually people who requires data visualization they're busy and what they need to understand is like yeah we need to think how to create the effective visualization and don't mislead the reader if you go back to the structure so now we have a complex like more complex directory so it's basically the directory for this presentation so you So you could notice a few things. So now I'm showing you this src directory. There are a couple of Python files. Most of them, they start with the plot something. So each of them are executable by itself. So for example, if I want to reproduce the sine function, I just call plot sine, or if cosine, plot cosine. If plot waves, I just run Python plot waves, and it generates me SVG plot, yeah, it's ready. approach works well for me. Why? Because once I found myself like in very difficult situation when I had like tons of Jupyter notebooks where they have like pretty similar names with untitled something and I usually see people using them. It was a nightmare to find the like the plot that I created and to reproduce it and since then yeah I learned that okay it's very nice to separate your data from the data visualization and it also allowed me to structure the project a bit differently and think a bit differently so for example if i would recreate a plot like that if i would do it in a jupiter notebook i would probably create the first plot and then if i would experiment with that so for example imagine if i need to change the color scheme i would copy paste my cell to the new cell i will change the color scheme then i will repeat this process for another cell to get the third one, and it's not really scalable. So first you have this endless Jupyter notebooks with many of cells, then you tune one cell. If you want this change to propagate to any other cell, so you have to repeat this process again and again, yeah, it's not a good way to do it. That's how I do it now. So basically, yeah, I have a few functions, so I already talked about the plot underscore and visualized underscore, but you see how easy it is, because on the slide, the third function is the main function that generated these three plots. So you see that I create, like, I read the data, data, I believe, separately. I create the figure and axis containing of three plots, and then I just call the methods. And you can see by this simplifying that there are also some anti-patterns because I call the same function again and again, and then I call this a title. It also allows you to think how you can simplify your code. Another example is when you need to experiment with the data, so data is static, it's separate from your data visualization but sometimes you need to convey the message to your audience and that way it's quite easy so on top it's not misleading graph but you see all of other three are misleadings and why because that's the same data but different access sizes and if i would write the article saying that hey come on like the global warming is like is crazy i would use that misleading graph with exponential growth. However, there is no exponential growth. Or maybe if I want to convince people that, okay, the global warming is like, it doesn't exist, I would use the graph on the bottom. It's the same data, but with the different axis scales, it looks completely different. And when you, yeah, here it's not for visualizing and convincing you that, yeah, there is a global warming or not, but it's mostly for the code that is written on the left. It's the same building blocks that I used before. I used to reproduce this graph. That also brings us to another point. You see here the data visualization is separate from the data itself, and the tests are important. And sometimes you don't know how to write the test for your data, but you at least know something. So you know that, okay, your temperature, it would be a number. It wouldn't be in the string. So you can introduce some basic tests on your data that, yeah, you stay safe. Or, for example, if you know that, okay, the temperature of water, it doesn't exceed 100 degrees, maybe it's a nice idea to introduce that test, even if it's like not, doesn't show you much. But in case something fails, for example, you're processing start returns nonce, you will know that, okay, your test is failing because your data is not integers anymore. So you have this alert way earlier than before. Another thing that I found is that it's very nice to version control the plots this way. Because as I said before, for me, it was like endless amount of notebooks. And I didn't know how to use the version control effectively. But after I split the data processing code with the data visualization code, it's quite easy. And another thing that I really liked is that, okay, here I kind of get the GitHub example. But if you go to Visual Studio Code, you can use a plugin to go back into history. And then if you introduce any plots to your GitHub repository, you can basically open the SVG file and go back in the history. and then you can revert to any commit like in the past because imagine that someone comes to you and says that you need to change anything so it would be quite easy with this approach which is harder with Jupyter notebooks in my practice. Another thing that I found myself not doing at some point and I'm doing it now, learned by my experience is that not setting up the health checks and remember when I said that it's nice to have the gallery.pdf file. It works kind of a test for me because any time the data changes, the gallery.pdf file will also change. And for example, I can introduce the simple GitHub workflow here that is self-explanatory. So you basically execute the Python code, which is on the bottom. So I call assemble plots.py that assembles all my plot functions and generate the PDF file. And I know that if I change anything in my plot, If any of the plot function fails, the workflow will fail, I will get an email and I can go and fix it. So I immediately notified if something goes wrong and I can do it in a pull request. So yeah, before merging to the main, I already know that and that's for example how my assemble.plots file looks like. So, it may look complex, but it's not. But the key is on the line for a file in path, parent, globe, plot, underscore, pi. So what it does, it reads all the plot, underscore, Python files on your directory, and it imports all methods that starts with plot, underscore, and it executes them. It executes them and adds the plot to gallery.pdf. So that's why in a big project, I found it very useful to have this gallery.pdf file where you can just go and showcase to people. You might think of that, okay, I'm not coding, so we have a static environment like Tableau, Grafana. Yeah, here I also learned some lessons, and I made some changes to production dashboard that actually are not supposed to be there and yeah from my experience now i always keep two instances of the production dashboard so i have my staging dashboard and i have my production dashboard and before introducing any change in data or in visualization and pro dashboard i do it in staging environment communicate with the stakeholders if they are fine i push the changes to prod knowing the limits it's mostly about the data processing algorithms that you use because sometimes you yeah you write the test and it works for your specific case but maybe if you increase the data like a thousand of times it will stop working or maybe your algorithm works for a month but if you continue doing that way so in 12 months it's just stopped working so another benefit of knowing the limits is just like write a simple test multiply your data by a factor of thousand or million whatever your load is and see if it will continue to work because you could end up in situation where yeah where you need to reproduce something or you need to create a plot but you cannot because you need to refactor your code or optimize something and basically yeah that's my journey through this data visualization horror movie now i feel myself a bit better knowing those things but before it was not the case. The main messages are work collaboratively with your partners, aim for reproducibility, and for me, the version control is the key, so I know that at any point of time I can reproduce any graph. Back up the data, version the code, set up monitoring, and create a continuous pipeline that can help you to identify the bugs and probably save you, not life, but save you lots of time in the long run. Thank you, and I'm open for questions.

Speaker 2 [24:27]

Before we can go to questions, I would like to summarize again very briefly. Please pick up your wristbands for the social event at the registration desk as early as possible. Also, there is the lost and found desk if you have lost anything. Please pick up your Pi Ladies wristband if you are interested. also the next presentation which would be held in this room was shifted so there is no presentation next in this room now we can get to the questions after I thank Artyom for this great presentation I think the drawings were so cool if you have any questions just write them down at Slido also just vote the other questions up please first question how do you test the graphics you generate is it manual by eye or do you have some method of automatic yes

Speaker 1 [25:35]

Yeah, that's a great question. Actually, I write a test for my data visualization also because I found myself once, it's again, it's related to reproducibility. And if you remember from one slide, when I separate the data from data visualization, I had the line that says like pandas, read CSV, blah, blah, blah. so in my one in one case i draw a line plot where it was like accepting a column x and a column y but it accepted all the columns that starts for this y my data changed over the time so and i created a very nasty plot with the same plotting function where i had like not one line on my plot but tens of lines so what i test i test that okay i know that this specific grab that i ship to my my customers it consists of two lines and since everything in Python is an object you can write a simple test saying that okay get the access object and ensure that the number of lines is equal to two this is the basic test that I write every time now maybe there are other tests yeah we can just yeah just think what what is your visualization and write the test to ensure that is there

Speaker 2 [26:50]

Next question, Matplotlib versus Seaborn versus Altair.

Speaker 1 [26:56]

Well, it's a very hard question for me. I love Matplotlib because it's like a tool that, well, I feel the most comfortable with, but any other tool is great, so it's just like individual preferences. But it's also related to the domain because when I usually create the data visualization, I create them for reports. And those reports are usually static. They are printed on the paper. And in that scenario, my plot leap is, well, it's a winner for me because I can tune any object on my plot.

Speaker 2 [27:28]

There is time for just one more question question to the global warning Example when plotting temperature versus time there is no obvious scale. So how do you decide what excess scaling is not misleading?

Speaker 1 [27:42]

Yeah, actually, that's a good question. And in some books, there are some recommendations. But what I found myself doing is that experimenting with scale and trying to show and communicate with people and just showing them the graph and asking like, if what you can say, looking into the graph, because if you remember the bar chart example that I showed you that before had like five colors for each bar. and in the end like it was down to one selected bar and I say that okay here the message is clear that author is wanted to say something about bacon and the same works for the data visualization like with the line charts so I could end up like with the few examples of the different accesses and then I'll just discuss with people with whom I work like what you can say like yeah which one would you prefer or in my own judgment I could see that hey this is bad this looks better so maybe it's less biased

Speaker 2 [28:41]

Again, thank you for your great talk.