How to create effective data visualizations

Speaker 1 [00:00]

data visualization, and let's give a warm welcome to Dominic. The stage is yours.

Speaker 2 [00:12]

Thank you. Welcome everyone to this talk about data visualization. It's actually one of my favorite topics and since it's very visual and colorful, it's also very accessible and lightweight. So I hope you have a lot of fun during the talk and learn something along the way. The talk is divided into three parts. The first part will be about some general principles that will help you design better visualizations. The second part will be mostly about some practical tips and tricks, like the usage of color or the advantages of different chart types. And the third part will then be about how to put all this knowledge to work with Python. So let's start with the most important thing that should be at the core of your visualization efforts. That is the message you want to bring across. Now I could just show you a chart like this one. Actually this one here is super interesting. It's about money and happiness. On the x-axis, you have the GDP per capita of different countries. And on the y-axis, you have the life satisfaction, self-reported from surveys. And if you have a closer look, there's a lot of interesting insights to take away from this. So I could just leave you with this chart, but in general, the data visualization is more effective if I have a clear message I want to bring across. Something like, hey, the happiness increases with money, but it levels off after a certain amount. So the first thing is that I'm really conscious about the message I want to bring along and structure my chart or my presentation accordingly. Better, to be really conscious about this, write the message down, and at best, write it down on the chart, maybe as a chart title. You might think this is redundant because the message is already included in the data, but it actually helps to drive the point home, especially when your audience, as is mostly the case, is not as familiar with the data as you are. it on the chart and use additional design elements like highlighting or annotations to further support the message. So now we have a clear message. But what's also super helpful is to have a clear mental model about how data visualization works. And what helps us here, I think super useful, is an approach based on the grammar of graphics. So the grammar of graphics is originally a book by Leland Wilkinson written in 1999. Very comprehensive overview of describing a structure or a system of data visualization. And was famously adapted by the person on the right here, Hadley Wickham, in the design of its ggplot package for R, and the accompanying Tidyverse ecosystem, which you may have heard of, which also influenced later packages in other programming languages. So what this means, very simplified and practically broken down, is that if you have data like this data set with different variables as data frame columns, you can think about visualization as a mapping or encoding of the variables in your data on the data frame columns to the visual properties of your chart. For example, here you have the length and depth variables in your data mapped to the X and Y positions of the scatter point. And additionally, the species category mapped to the color of the scatter point. I think this is a very simple, clean approach that makes for a great mental model of data visualization and leads to clean interfaces and clean code like the actual code for ggplot here would be as simple as what I just described here you take the data frame map or encode its columns into visual properties for the specific chart point and there you have it a very nice descriptive way to think and think about data visualization and write the code and the interfaces accordingly. So now we have a clear message and a clear structure to describe our charts. But we still need to make it more accessible to our readers. And what can help us here is a design principle called the visual hierarchy. So here you have two website mockups. One on the left is really cluttered, and you don't really know where to look, whereas the example on the right is much better structured with certain elements that really grab your attention first. And this is what the visual hierarchy describes. visualization, visual elements are more prominent by the usage of color, larger size, boldness, and grab your attention first. Whereas other elements, by using smaller size or graying them down, move into the background. So instead of having all visual elements on the same layer and throwing everything at at the viewer at the same time, you can influence the order in which the viewer perceives the different elements of your chart and processes them, thereby guiding them through your visualization and making it more accessible. So one example where these principles work quite well is this visualization here. from the BBC, and it describes, you see it already, the chart title, the takeaway message, ocean temperatures highest on record. This was the sea surface temperatures in 2023, and you can see probably after the headline what you notice first is the prominent data point and line highlighted in red. So these are the elements that you notice first, while the other data lines representing all the previous years are grayed out and move a bit into the background. So I think quite a nice example of the structuring your chart, making it accessible, and also adding additional annotations or highlighting to make the data, you know, this is scientific data here, and by these comments, it's made accessible to a broader audience. So, however, here in this chart, we only have regarding colors, we only have gray and red. Usually you have a lot more colors for your visualization. What to consider regarding the usage of color? And the proper color choice depends on the type of data you have at hand. So quite often you may have categorical data, as you see on the left here, best choose some contrasting use, probably colors that are pleasing and not too bright or harsh on the eye. However, nowadays most graphics programs provide sensible defaults here. Use diverging data, like data that is centered around a middle value, 0 or 1, and diverges in both directions. Think of correlation coefficients or the visualization of a correlation matrix. So best choose a color map here with a neutral center, like white or light gray, and different colors on either side. Also very prominent ubiquitous are, of course, sequential data, just simply continuous values, which can best be represented by using either a monochrome color map going from light to dark saturated color, or one of the so-called perceptually uniform color maps, which are also nowadays often a standard for color maps in many graphics programs. One representation of sequential data I would discourage you from is the rainbow color map. Arguably using this was a bigger problem a couple of years ago when this was the default in several graphics programs like matplotlib. And this color map here is bad for several reasons. For example, there's no intuitive ordering of the colors, like red, green, blue, yellow. What is the order here? It's not so easy to make out as, for example, when ordering different colors going from bright to dark. That's the perceptual effect of red alerting the eye, and the sharp color transitions, Like going from red to yellow create artificial edges which aren't there in the actual data values. So I would discourage you from using this. Better use for heat maps or choropleth maps a simple monochrome or perceptually uniform color map like this one here. Pro tip for the orientation of the color map, use the end with the darker colors to describe higher values. This feels more natural, you know, describing a higher value, more of something with more color. Like on the example on the right here. Where possible, use intuitive colors. example are the climate stripes where the levels or the change in global temperatures are represented by blue for cool and red for warm. There's actually one variation of these climate stripes I really like in a very horrible way. I put this chart on the right here, the climate inaction stripes. This puts together the change in global temperature with the rise in atmospheric CO2 concentration and the dates of various major climate conferences to show how useless these efforts have been in curbing greenhouse gas emissions. A very famous example of using intuitive colors is using red for bad and green for good. This can be problematic for many people so just make sure to either use a slight variation of that like red or blue or include additional markers to make this red green accessible. Also regarding the choice of colors don't use for categorical data more than say a handful of colors. You can have a look at the example here and you immediately notice that this is very easy very difficult to process it's just too many colors if you go back to this grammar of graphics based approach of encoding variable in your data into visual property this means here that you've likely chosen the wrong encoding for your variable with high cardinality either encode it as some other property use a different chart type or also just reduce the cardinality in your data. Same goes for using color unnecessarily. If you look at the example on the left here, every bar has a different color, which is unnecessary since the order is already represented by the Y position. So again, with this grammar of graphics approach, this means you have encoded one variable into two visual properties, the Y position and the color, which is unnecessary. So best either have every bar the same color or use the color to encode a different variable in your data. like it is done on the example on the right here, which reveals some additional insights. Regarding encoding a variable as a visual property, are there recommendations which encodings are more preferable than others? And indeed, we know from cognitive science or perceptual studies that we as humans are much better to assess differences in values when they are encoded as a position or length compared to when they are encoded as area, volume, or color. Have a look at the examples here. In every example, it's the same values. And looking at the bar charts on the left, you could roughly make out what the relation between the different items is. And this gets more difficult when looking at the areas here, the circles and the pie chart slices, and even more difficult if you look at the chart on the right where the values are represented by color. Also, regarding pie charts here. So, I know they're quite popular, especially with the Excel crowd, I think, not so much with Pythonistas. And they are usually a bad choice. Looking at the examples here, you have three examples where the same data was represented with a bar chart and with a pie chart. And looking at the pie chart down here, it's quite easy to see which is the lowest, which is the highest value, how the different items are ordered. And this is much more difficult when looking at the pie chart above. So, better use simpler bar charts, however, I think there's maybe one narrow use case where pie charts are not so bad, that is, representing when something is the fraction of a whole, especially when it's around 25 or 75%. So looking at the example on the right here, you can easily see that the larger slice is three quarters of the whole, and the smallest slice is one quarter, and here is not so straightforward to immediately see this from the bar chart. However, I think this is a really rare exception. So some general tips on other chart types. If you have a chart like this where you have to represent aggregate values like the mean of some quantities from different observations. You would probably make a scatterplot with the values here, but consider using, as the example on the right, a box or a swarmplot, which in not only representing the average value, but also how the values are spread out can reveal some interesting additional insights. Using bar charts, which are quite often a sensible choice here, the default in most graphics programs is to use vertical bar charts, like the example on the left. And in many cases, especially when you have long labels, vertical horizontal bar charts work much better, and you don't have to tilt your head all the time to read the labels. And another pro tip, if you create a data visualization like this one with lines for different categories or scatter points, most graphic software automatically creates for labeled entries a legend with examples of the graphical representation and the label puts them somewhere inside or outside the chart. This is useful, but it creates an additional mental lookup task, like when you look up the plot, you always have to switch between the legend to remember which label is represented by which color. And you can make this easier for your reader if you put in the extra effort to put the labels manually in the correct color alongside the respective data lines. So we've learned about the most important parts of your visualization, the message you want to bring across, the structuring of your chart, tailoring it to your audience, lots of practical tips. Now how to make this work with Python. And in Python there's a vast data visualization landscape. I think this graph here has been circling in the Python community for ten years, originally created by, I don't know, Jake Vander Plaas or Nicolas Rouget, and, of course, we're not going to focus on all of these packages here, only the most important clusters, which are, in my opinion, Matplotlib and its accompanying ecosystem, and the newer JavaScript-based libraries like Bokeh, Altair, and Plotly. There's also the HoloVis, HoloViews ecosystem. Honestly, I don't have a lot of in-depth experience with that, but it might be interesting to look into that, especially if you have really massive data sets you want to visualize. So regarding the two clusters here, first of all, there's matplotlib. I think it's the most well-known, most popular, well-tested graphics package with a vast accompanying ecosystem. Most notably, I like Seaborn, which really provides a great interface for statistical high-level plots. It has a great configurability. You can do really all sorts of customizations. However, it suffers from its API being somewhat confusing. You know, it has this dual API, partly dates back to what was inspired by MATLAB, so not very modern and clean. Previously also had some poor stylistic defaults. However, this got better in more recent versions. And then in contrast, you have packages like Altair or Plotly Express, and they work, for one, interactively out of the box. You know, MATLAB creates static charts chart, which I think also originates, not only because it's older, but also it originates it from creating scientific plots for publishing in journals, but these here, they work interactively out of the box. If you create a Altair or Plotly chart from your data frame in a Jupyter or Marimo notebook, you have direct interactivity, and they are based on this grammar of graphics API, like ggplot. As you can see in the example here, you specify the dataset and the type of charge and then these variables to visual property encodings or mappings. And here we can see again how great it is to have such a clean and standardized approach. You know, when you come from ggplot, you immediately feel familiar with the syntax, which is different for matplotlib, which is more about juggling with individual arrays. When to use which? My personal recommendation would be to use matplotlib for non-standard plots, or if you need high customization, like special scientific plots, or if you need to make your plot adhere to certain academic journal publishing standards. A couple of years ago, I also would have told you that you can find for every possible customization examples on Stack Overflow. Of course, we use coding agents nowadays, and feel free to test this out if the coding agent works better with the customization of multiple plots than other libraries. For everyday plotting, data exploration and notebooks, or for creating dashboards, I think Plotly Express or Altea are good choices. Altea is already the default plotting backend in Polars, so especially when working with DataFrame, these are really the easily usable go-to libraries. Another mention here goes to great tables. If you have a case that you don't need like a super fancy visualization, but you want to show the actual raw values, Maybe include some nanoplot here, like these little barplots in the cells. CreateTables is a package to programmatically create nice tables like this all in Python code. So, to sum everything up. It's really important to be clear about the message you want to convey. Best write it down. on the chart and design your chart with the choice of chart type and the inclusion of annotations, comments, highlighting to support the message. Use the visual hierarchy to structure your chart and make it easily accessible to the viewer. Considering the grammar of graphics as a mental model makes for clean thinking and clean code. In addition, with adhering to the best practices regarding the usage of color or different chart types, all this put together enables you to create really effective visualizations of your data. Thank you.

Speaker 1 [24:31]

Thank you, Dominic. We have several questions here. The first question is, are there color maps which works best for colorblind people?

Speaker 2 [24:43]

I think, I go back to, so I think this would be here in the case, for example, for these, either for the monochrome color maps, which go from bright to darker saturated color, Like if you take the color away of them, you still have them going from white to black. So they work even without color. And also, you know, the ones that are below here, that are those perceptually uniform color maps. They are designed so that a change in value is reflected by a change in the color space of the color map and make this gradually. So the change is very continuous. I think these were also designed to work for colorblind people.

Speaker 1 [25:44]

Thank you. The next question is, there's a chart you're showing, the line chart in Timeshares. If I have more than six different lines in Timeshares, what should I do?

Speaker 2 [26:00]

Yeah, I think then you have the same issue Like you have with using too many colors if you have too many lines And I think it would depend a bit on the case if the lines are like Spread out then it probably work But if they're like crossing each other all the time, then it would just be too cluttered So it depends a bit on the actual data at hand One way to solve this would be like to aggregate your data or to use so called small multiples You know, have a different set of axes for each of your lines to not have them overlap.

Speaker 1 [26:40]

Thank you. The next one is the blue-red temperature graph comments a major protein crime, in my opinion, by not starting the y-axis at zero. What are the rules about accuracy in data presentation?

Speaker 2 [26:59]

Yeah, this is all.

Speaker 1 [27:04]

The, yeah.

Speaker 2 [27:08]

Okay, this one here, I know that it shouldn't start, that it should start at zero. That's true. I think here in this case, it works like this because you, for one, you need the space to put the labels into. I know what you mean that usually you should start the axis at zero to make the effect less dramatic than it seems here. I think it depends on, of course, the data if you really, if there's like a malicious attempt behind it for some time, for many types of data or the variation in the data, The changes would just be too small to see them properly when you have the y-axis go all the way to 0. But in general, it's good practice, yes.

Speaker 1 [28:16]

Thank you, and yeah, we have still two minutes, there's many questions coming, and this question, do you have any recommendation for visualizing large data sets, millions of points?

Speaker 2 [28:35]

Yeah, I think in Python, I once attended, I think it was at, was it another PyCon, a workshop by the HoloViews people, and one of them, maybe was it HVplot or Datashader, was designed specifically for this purpose of visualizing massive data sets. So if that is your use case, feel free to look into this ecosystem.

Speaker 1 [29:02]

Okay. We take one last question. Do you have one example from your experience where the plot really misrepresented the data?

Speaker 2 [29:16]

I don't have an example at hand, but I think it's something that you might need to look up if you Google for bad charts. Also, I think there's a lot of examples circulating online of bad visualizations.

Speaker 1 [29:32]

Okay, then, thank you a lot for the questions, and thank you, let's give Dominic a warm applause. There'll be one more session in this room, it's production machine learning from the time 2015 to 2017, 75.