Rewriting your R analysis code in Python

Speaker 1 [00:02]

Yeah, I'm Helena Schmidt, yeah, and I will talk a little bit about R and Python, and first of all, I want to know who does R at all from the people in the room? Okay, it's most of you, and who does Python, maybe? Okay, yeah, so, yeah, I'm interested in what did you already know from what I was talking about, but yeah, first something about me. I'm a freelance data scientist for about five years now and work mostly with Python and R and also give lectures on universities. And my background is, yeah, I have a PhD in gravitational physics and I also make a German language podcast about data science called Datenleben. And, yeah, in this talk, I will talk a little bit on when to rewrite your code and, yeah, some comparison of Python and R. And then a lot of examples for data analysis, pipes, classes, libraries, quoting, and combining best of both worlds. and yeah the first question here is when to rewrite your code and yeah one good reason to rewrite your code is when it's impossible to maintain the code base and unfortunately that happens a lot especially on our projects because most people that don't the program in r are not software developers and don't know how to get a code base that is actually maintainable. And I'm also working on projects fixing that for customers. And yeah. Also if you just want to switch programming languages, for example, in your team you have mostly people who are working with Python but the code was already in R or what I have done in the past mostly was rewriting MATLAB to R and that's also a possible reason and in one project I had the research project was done in R and then the idea was to integrate it into a larger software But, yeah, that was not easy in R because they already used Python and so on. Then it was cleaner to go with Python there. Yeah, when not to rewrite your code, maybe when it just takes too long and is then not useful. And what also can happen when you rewrite your code is you will lose some specialized features that you maybe totally forgot about because the people who wrote it are not there anymore and the people who need that feature don't know that it will get lost and then you start rewriting and nothing works anymore. And, yeah, so make sure you know exactly what you do. And also, writing new code means you will introduce new bugs that may have been already fixed in the past. But, yeah, now it's a new programming language. And it's hard to cover everything from the start. Okay, now I'm going a bit into R versus Python. R is basically the most specialized language in data analysis, while Python is the programming language that basically allows you to do everything. The strength of R, especially from my point of view, it's far easier to prototype your data analysis code. This I mean, yeah, when I see a data set and I want to know what's in this data set, it takes only maybe five lines of code in R to get a good overview of what I'm looking at. And in Python, that's not that easy. I tried that, but yeah, I always go back to R when, yeah, looking into data at the first time. So, yeah, that means the code is also more compact, at least for data analysis. And one very neat feature of R are the pipes, the data analysis pipes. Yeah, and also something called quoting, which I will also talk about today. And, of course, 2D plots with ggplot2. Do you have already worked with Python and Matplotlib in the past? If you know both, then you won't want to use Matplotlib anymore. But the strength of Python is it's a much more structured programming language, so the The code that you are doing is much more consistent and much more structured, while in R it's possible to write everything in one line and make it impossible to read. That's much harder to do in Python. It's also possible in Python, of course, but it's harder to do. And for example, stuff like try-catch is in Python much more consistent with raising different error types, and it's always somehow messy to do that, it's possible, but not that nice. And I also like really the asynchronous stuff, the asynchronous execution in Python, and in R there are some libraries that can do that, but it's also not that consistent. Okay. Yeah, the most obvious easy differences of both programming languages is in R, you don't have scalar values. It's always a vector of data. So if you define a variable and you put a number in there, then it's always the first of a vector of numbers. And in Python, yeah, you have to do that a different way, for example, with numpy arrays. And, yeah, then you can also add vectors and so on. Yeah, and the other very obvious difference is R is one of the few programming languages that actually start counting on 1, while most programming languages start counting on 0. which does not make much sense in our days. Maybe in times of C, where you calculate in an array the position of the data, it was useful to start at zero, but yeah, it's quite confusing for people who are learning it newly. Okay. Okay, so, why to switch from R to Python, especially, for example, yeah, if you just have more developers that can work on Python, that's also true the other way around, or you may want to have the general purpose features of Python. For example, you want to create an API server that does some data analysis, and then the users want to just ask the server what is the result, and it's easier to do in Python, but it's also possible to do in R. yeah and one other reason could be you have a very specialized library that exists only in one programming language and yeah I think in R there are more specialized libraries for data analysis but in Python you get everything else okay Okay, that was the general overview, now go to the examples, and yeah, I think the most basic libraries that you can use in Python that everybody should know about are Pandas and NumPy, and in R you can just use base R, but base R has the problem that you may face some really nasty things that have inconsistency in typing. For example, if you do a subset with the square bracket notation of a data frame, and then you have suddenly only one column you are asking for, then you get only a vector back instead of a data frame. And to mitigate that problem, there are libraries like Deployer, and the whole tidyverse is a good starting point for data analysis libraries. And I think if someone is using R for data analysis, you should always start looking there. Okay. Now start with the first example. Yeah. Who does know what pipes are in R? Okay. Yeah, pipes basically means you start with some position of the code. And then you have a pipe, for example, this is the new pipe that was introduced by the the latest version of R4.1, but there were already pipes before that with a different notation that came through some libraries, and basically what this does is it forwards the data to this function, and here the first parameter that you don't see is the data parameter that is filled by the pipe, and then you can do the grouping, for example, here, and then it is forwarded to the next function, in this case, the summarize function, and, yeah, this function is then called for each of the groups, and the result is a data frame, or a tibble, which is a special type of data frames from the tidyverse without some of the nasty stuff going on. And yeah, what this does is basically it counts the amount of data in here in each group and then calculates some statistics of it. And for this example, I use the palmer penguins example data. And, yeah, pipes don't exist in Python, but you can at least write your code like they exist because you can just concatenate everything with a dot. And then to make it readable, you can use this backslash to go then to the next line and make all the concatenations more readable. And then you get basically the same. But when you start with some R code that looks like the upper one and go to the lower one, which is Python, then you will face one problem. And this problem is, it's not a data frame anymore, but it's data frames below data frames. And yeah, for that we can look into the results. The first result is from R, and the second result is from Python. and you can see here this count belongs to this variable and this is one sub data frame and this is another sub data frame. So while in R you just get one complete data frame as a result, when using Pandas it's not always the case. So when you rewrite your code in Pandas you have to make sure that you know what you're doing because the structure is different. Yeah, and what you can also do with pipes that is really nice in R is all the filtering and subsetting and so on, but that's also possible with pandas but not with pipes, unfortunately. yeah the next thing I want to talk about is objects and classes has anyone of the people who wrote R code before ever defined a class okay two persons yeah there is something like object oriented programming in R as well and because R is a very old language that is yeah highly developed right now there's not only one possibility to write classes but many and yeah the earliest one were the s3 classes and s basically it was the commercial version of r that existed before r and that's why a lot of stuff is called s stuff and yeah S3 and S4 classes basically were completely different than what you know from every other programming language but there is one class system that is similar to the classes in Python and that's the reference classes yeah and a reference class is defined by the setRefClass function and then has a couple of fields defined and each field can have a specific data type. It does not must have a data type, but yeah, you can define it. And then you can add a list of methods to the class. and in this example I use a very simple class called basically a bank account where you can withdraw some money from and this example I take from the advanced R book yeah and one thing that is very special to R when you do classes is this assignment operator. Who has seen this assignment operator before? Okay, it's more people than people who have done classes before. And what this does is it looks in a scope above the current scope if there is already the variable defined. So you can then assign a value to a variable will, yeah, that is not local anymore, and, yeah, this is necessary when you define classes to use this assignment operator here, and, yeah, then I have some simple examples, for example, yeah, creating an object from the class, it's, yeah, I think with a new function It's pretty obvious what's happening here. It just defines the variables. It's also possible to have a constructor defined. That would be a function called initialize. And then it can also have different parameters and just the balance. Yeah, and basically it's the same as in Python. When you copy that without the copy function, You're only creating a reference and when you change it, it's basically the same again. So you have to use the copy function if you want to really have a copy of the object and not just a reference to the same object. And yeah, then I'm just showing here that when you do it on the first account, it has and withdraw everything, then the balance gets zero. And in the other case, it stays 100. And the same in Python. And for Python, yeah, I think most of you know, but in Python, you don't have the special assignment operator. You are using the self. And with self, you can define object-local variables like balance in this case. And while Python does not have the strict typing, it's possible to use the type annotation so that the users or the developers also see what types you are expecting. And the same is also with the withdrawal function. It also uses save as parameter, and then you can just, yeah, get the data from it. And, yeah, creating the objects is basically easier without the new, but that's not a big deal. But copying needs the copy, deep copy function, because otherwise you would just create a reference to it. this is where both programming languages have the same and what I also wanted to show you is both programming languages allow documentation strings so the first string in a function can be used as the documentation of a function of a class yeah, very good similarities yeah the other thing I wanted to talk about or the next thing. In R, you always use this library call. And this is basically the same as from package import star. And I think since most of you are Pythonists, you know why it's bad, because you're polluting your namespace. but it's also possible to not do that in R, for example, with this double colon. It's possible to use a function or something from a package that does not get loaded separately. And this is basically then the same as, yeah, importing the package in Python, and then using the dot to go to some functions for the package. Yeah, the next thing I wanted to talk about is the dot dot dot feature in R. Who has seen this before. And who knows what it does? Okay, yeah, the dot, dot, dot, you see it in the documentation a lot, and basically what this means is you can use every argument that is not defined here and put a lot of arguments that were not defined into the function, and then you can do stuff with it. For example, in this example, I just create a list from the dot, dot, dot, and the dot, dot, dot is filled then with everything that comes from this. So I can put, for example, two arguments to the function, the one and b, and one is an argument, and b is a keyword argument, which are the corresponding names in Python. So the same in Python would look like star args and star star quarks. Yeah, these names are arbitrary, but in most examples you will see args and quarks for this. Yeah, and this is basically then the same as using dot dot dot in R, as you can also see here. but I have to create the list to really use those arguments because without this list creation in R I can only bypass it to the next function and that may be what I want to do but not always and then you have the numeric arguments and then the keyword arguments also in the list and in Python it's a tuple respectively dictionary okay that's this feature and one other important feature that our health and peasant doesn't if this is called quoting or non-standard evaluation and basically what this does is yeah you can say to our yeah don't execute this code and since our users a lot of lazy evaluation of the code you can do that also afterwards for example when you define this function with the argument X, and then use a substitute function, you say to R, yeah, substitute X by the quoted version of X, then you can, yeah, basically say when I put 10 plus 10 to the function, prevent 10 plus 10 to be executed and the argument becomes a call argument instead of the already calculated result that you are maybe used to in Python. And this is one part in most cases, I already have shown you this in the examples before, This is exactly the reason why you can use the species and the column names without putting them into strings here, because R then knows exactly, yeah, okay, I don't execute the code, I don't know this variable name, but yeah, the function then will use for that in the data. while in Python the solution is most of the time just using strings and that's most of the cases the way to go but if you are using this a lot then maybe not one example yeah that I have here for example is I can call this function with this y argument, and y is not defined at all. It's not in some data. It's not defined anywhere. But when you use substitute, the code does not break, and then you can test if this is actually defined in the code. Yeah, but also you can do it just with using references in Python and names. But if you are using this extensively in R, then I don't know any good way to do it. And as I said, it's used all the time. For example, when just doing a simple plot, you can put the column names in here and then calculate it for example here millimeter to centimeter and decimeter conversion just in the plot code and then you get a yeah plot with different units and you don't have to do it in the data but only when you're plotting yeah but I'm not exactly sure how to do it in Python, so I cannot show you that. But this is a simple example that should be easy to deal with. But I think there could be some obstacles if you're using this a lot. Okay. But if you need this sometimes and not all the time and you want to work in R or Python most of the time, you can also combine the best of both worlds, and this means you can just execute Python in R, and then converting the data. For example, in RStudio, when you're working with RStudio, which is one of the major IDEs in R development, you can just make, yeah, a block with Python code in it, and then you can use this R dot notation in Python to get the data from R. And this also works the other way around. So this shows, yeah, the first line of the penguins data in Python code when reading the data from R. And the same, the other way around, is this R code, where you can see, yeah, with the Pi dollar, you can get data from the Python code. And, yeah, if you try it in R studio and you don't have the correct package loaded, it will tell you that and install it for you, and you don't have to think about anything to use it. Yeah, it's called reticulate. And it's also possible to do it the other way around. So when you're working, for example, with Python in the Jupyter Notebook, you can just, yeah, use the RPy2 package, and there you can load the R objects, and import some, yeah, package from R, and then run R code from Python. And yeah, the same from R would be this. You can also use the import function in R to load, for example, the daytime package, the UTC now, but since I also want to talk about obstacles, this is basically wrong. It converts the time zone twice and assumes basically we are currently in UTC plus four instead of plus two. So when I last executed the R markdown file, it was not seven in the morning. Also not in UTC. And you can also see this converts also to the correct R data types. But it's Python code in R. So, yeah, that's all I wanted to show you. Yeah, I had some references, for example, the advanced R book and the Palmer penguins. And you can contact me on Twitter or GitHub or just write me an email. Are there any questions? So, thank you. So if anybody have a question, I can hand out the microphones because I didn't have a question now on Slido. Yeah. Hey, thank you for the talk. Did you ever encounter any project that tried to combine R and Python like in the manners you showed on the last slides? Like, okay, importing Python libraries or functions. Yeah, anything in real life? or is it just theoretically possible to combine these two? The method I showed you in the last slides did not encounter those in projects, but what I encountered was that you can use R kernels from Jupyter and then run them through some other code that does the API stuff and so on. And that I already encountered in some projects. Any other questions? Hi, thanks for the talk. My question is, do you have any suggestions for transitioning Shiny apps to Python? Yeah, Shiny is most of the time ggplot in the background. I won't do that. I really, yeah, I think there's some clone of ggplot for Python, but I never got one for that. But, yeah, it's really easy to just write server code with flask in Python. And when you are using matplotlib, for example, you have to make sure really using objects. because matplotlib with all the examples that you encounter in documentations and so on, you are working on a global state and when you have, for example, two API calls that run in parallel, they will both write into this global state and then you will get a mixed up plot as a result. but it's possible to use matplotlib as a really object-focused tool but then the functions are called differently than in most tutorials so it's nasty but it's possible. I already did that. Hi, thanks for the talk. So I have a question about what's your experience or maybe a suggestion when, for example, there's a specific library that I would like to use, for example, in production, and it's written in R and my system is in Python. Like, is it worth kind of translating the code or maybe try something like Docker to try to make that work as well? Yeah, it depends. For example, what I showed last, you can just use that library from R or from Python with this reticulate package in R or the RPy2 package in Python. But this does not work in some cases where the data is transformed differently. It could break. And if it breaks, the easiest way, yeah, either you can use Jupyter, Jupyter kernels that you can execute. then you don't need docker but if that also does not work very well you can of course use just yeah docker put the data in and get the results from the other programming language thanks is there any other questions so I might have one then because I'm curious as well Regarding, it's more about the packaging. How difficult is this to package R? I mean, because I don't know any things in R. So how difficult is this to package it compared to any, like, simple Python package? And second question then is, like, when you want to have, you know, C extension or Cyton things, you can do that in Python. Can you do something similar in R? Yeah, it's really easy to package things in R because there's the dev tools package and then you can execute one function that just creates the folder structure and then you can put your code in and you don't have to think about it much. And there's also the rc++ package to allow you to use just c++ code and from there you can use everything that is in C also in Python. And it also helps you with converting the data types. Okay, so we can thanks Elena again.