reticulate: R interface to Python

Speaker 1 [00:02]

Thank you. So, please welcome Jens-Bruno Wittek. He will give a talk about our interface to Python, Particulate. Thank you for the introduction, and it's a pleasure for me to be here and have the first talk after the keynote. This will be a little bit more technical than the keynote, but I guess you are interested. So it is about Reticulate, which is a package for R which allows you to embed a Python session into an R session. And during this talk, we will see all possibilities this package allows you. At the beginning, there will be a short introduction about myself. but also already a introduction of the package then we will see how we can run Python code in our and convert objects create and modify objects and then there are some special features like interpreter reporting feature and at the And I have a machine learning use case with three ways how to solve it. So let me shortly introduce myself. I studied statistical sciences and there gained already first experience with R. And now I'm using both R and Python together. in a project working as a data scientist for about two and a half years now at ACCA and we have a team of six data scientists working mainly in automotive domain so here are my contact data feel free to connect i'm also happy to to share the slides the video will be will be uploaded next week i heard and i would also give the slides there yeah in aka i mainly have projects in the automotive domain but the company itself has lot of different domains that is working in currently we have 16,000 employees so quite large and still growing but now let's head to reticulate it was released by R studio which also create this IDE for working with R and although the first release on github was last year the official version was released this spring and since then it's on crown so the main repository for our packages the words reticulate means resembling a net so meaning that two or more things are are interwoven, are knitted, are combined, and this is definitely true for this packages, which interweaves these two programming languages. Also, we have the reticulated Python, which is the longest snake in the world. Yeah, Reticulate was created with the aim to have collaboration in data science projects. So when there is code already in Python, but you only know R or you have to work with R because some restrictions then reticulate lets you use the already existing Python code. And, of course, although both languages are really good for data science, so I like both, there are some situations where one has strengths, Another language does not have for example one Package or module was created in one languages, but it does not exist yet in the other so Reticulate lets you combine them There are related libraries so in the direction from R to Python as reticulate is there exists as well our python which however does not cover all those features just a subset of features and in the opposite direction from python to r we have r pi 2 There are nearly as much features as in Ridiculate However, it's not as straightforward to use So, for example, you have to do more work with declaring variables The conversion is not automatically as here However, it also runs in Jupyter and for Jupyter there as well exists IR kernel. To start with reticulate, of course you have to install it and then can load it into R with the library command. Of course the precondition for that is that both R and Python are installed. However, I expect that when you want to use them. Yeah, at the beginning of the code, first I want to show some features features regarding configuration which just will show you information so there is the PI config command and it will show you for example which Python is used and some other information regarding Python version reticulate is available for Python 2.7 and upwards so with all 3.x Python as well and it's closely connected to NumPy and works with NumPy 1.6 or higher. Then here another command to see your configuration also some helper functions like use Python where you can determine which Python you want to use so by default it uses the Python it finds first on the path variable but you can and specify the Python you want to use. There are helper functions to see if Python is available, NumPy available, or a certain module is available. So it's installed on your local computer already. computer already then there is special support for environments which means conda environments and virtual environments maybe you know already so conda has this anaconda distribution where a lot of data science packages are already pre-installed conda environment basically is also a virtual environment however there is more pre-installed and as virtual environments usually are used for single projects the conda environment is for usually used by multiple projects so there exists a conda list feature to show all available environments this is here those I have installed here then with condor create you could also create a new environment from our and use it with a use condor and then sometimes you want to install further libraries or modules and bison into your environment this is possible with condor install and packages you can add a character of your of a single package or character vector additionally to creating and installing installing packages it's possible to remove them and swell to commands for showing the binaries and the version all these commands also are possible for virtual environments the hint I want to give it's currently not yet working under Windows so there you should use conda environments in general when you want to install new modules use the PI install command there you can add the packages you want you can specify environment and also at a few other parameters so now let's go deeper into Python objects this is a first example imagine you have some some Python code so import numpy as NP and then create a two-dimensional NP array then in Python you can show the type of it, which is numpy ndarray. And reticulate allows you to have access to these Python objects in the R session you opened. This is possible with pi dollar sign x. so pi is the prefix which every time has to be used $ is the R syntax for accessing an object like in Python we have a dot for it and here the object is called x so when you enter this in R you will see the object which then automatically is converted to a matrix data type in R and this automatic conversion is one remarkable feature in my eyes there is one question it's a reference no and so this was one first example you might you yeah it's it's fine it's fine i will adjust accordingly um so it's a matrix so there are different but but limited number of data types in in python so the numpy nd array is converted to a matrix in r except when it's a one-dimensional one, then it's converted to a numeric vector. And when you have a Pandas data frame, this will be converted to our data frame. So there are different data types where we will head on later to. Now you might think how it's possible to combine these two sessions. And the first alternative we see is PyRunString. So when you enter an R PyRunString and have a string with Python code, valid Python code of course then you can run it and access all objects which are created there with pi dollar sign variable name of course it might be not that handy to enter everything in that string so it's also possible to run an entire script with pi run file so imagine we have a script which has the same content as the code above so here we get some random numbers then pi run file script name and it will make you all objects accessible into your workspace when you have a file it's not always a file which directly creates objects often we have files with functions so a Python file was was one or more functions was start with the typical DF define and here in this case let's imagine we have a function which adds the numeric of 7 to each number so when sourcing this script the function then the original Python function is available to your R workspace. In this case it had the name at seven and you can access it even without the pi dollar. So it's available in your workspace and you can use all arguments this function has and also the signature is basically the same. So this is an example of functions you have in a local file in the script. Modules aggregate famous functions or those which you might also use which some developer has already created. And it's possible to make all modules and their methods available into the R workspace. So, for example, here with the import command, we can import the module math, which is part of the standard library and assign it to a math library with this arrow and hyphen so our assignment operator. Let's imagine we create an object math and then with this object all methods are available inside R. So, for example, here I take 2 with the power of 3 and give the result back into my R workspace. And, of course, this is not only possible with the module math, but with all modules which exist in Python. You can, there's another command for loading a module from a certain path for the built-in library for main, some shortcuts. yeah then let's see how this conversion works we can see here the commands are to pie and pie to our usually you don't you don't need them when just using using it because of that automatically conversion however it might be that you create objects in one language and do not want to convert it so a lot of functions have this convert parameter which you can set to false and then stay in one language do another operation on that language and convert manually at the end so here in the middle we see a creation of a matrix in a in r so two times three matrix and the r class command tells us tells us that it is a matrix then when using the r2 pi command it's converted to a numpy nd array and has all the methods available to a numpy nd arrays which you can then apply to that object and when you go the way back pi to R then it's again a matrix so we've just seen matrices and when considering matrices which are higher dimensional so which is both supported in R you have arrays which are multi-dimensional and in Python the numpy and the arrays are n dimensional as the name says already here in a two-dimensional case it looks simple. However, when you have a larger dimension, you should keep in mind that R and Python are storing the data internally different. So in R and Fortran, you have a row major order, whereas in Python and C, the the columns are coming first when having multiple dimensions and when you are in such a case just have a closer look before and check your results with array reshape there is a special feature to to change the dimension of an array then Then in addition to arrays, we know there are other Python data types like dictionaries. So inside R, you can create a Python dictionary object in three ways. So there's the dict command where you can input our list. or just a comma-separated enumeration of key-value pairs, or use the pydict command, where you have two arrays or vectors in R, keys and values separately. So all these create a Python dictionary. You could also create a tuple or a function, so here we create in R a function with the name at five have the object at five in the workspace and with Python we could convert it to a Python object it's not yet available as a Python object in the workspace as I did not assign it to a new variable however you could do so and and then use this function within the Python session. This call variable command I will show later. Additionally to the data types just seen, we have iterators and generators so there are several functions which allows you to create and use iterators I would so far here it's now no examples as the code would be rather long. You will see it on the RStudio documentation where you have the possibility to create a R function basically with looping an array, having, if or else, to have a stop criterion. However, you could create your own iterator in R. However, as this is not natively supported by R, I recommend using the existing Python iterators like like range and so on which will be easier without going this way then when developing you might be interested in these features about attributes so for case we have an object X as an ND array and P array you could call pilots attributes and which will show you all methods available for this these are all methods you know from from Python here's just a subset so number 91 to 94 of that result and you could call any of these methods on that object you could also do check pi has attribute if certain method is available for an object one heavily used at attribute is the lengths where we have pi Len and sure it's also possible to get the value of one attribute so remember we had here the object as an array from one to four and for the case we want to have the mean we could call pi get attribute and then do a call on it however this is just for for demonstration usually if you want to get the mean value you could import and numpy library and then directly call NP dollar mean then it's possible to set the value for example the shape so the dimension of an object you could change it like this import and output with pickle is supported as well so we have PI save objects PI load object which lets you create a pickle file so from the first example it creates the file and then loads it makes the result available in workspace another example is capture output sometimes we have errors and also these are supported so the error messages are automatically converted to air our error messages the stack trace is captured and can be can be printed for example here on the next slide we see the error or the previous error which was in the files a few another features so I don't want to go into adapt so you can go set the seat and some other internal operations the help is supported so let's say you want to have the help for an object and when you which is a python object you can call pi help and in the r console a window will open with the python documentation now one slide to the repl to the interpreter you can start repl python from r and then it will allow you to enter python code in the same console you can also access our variables inside this interpreter with the prefix r dot so when there are when there's an object x available you call it with r dot and make it available within the the REPL and close the REPL with exit. Regarding reporting, there exists the R package knitter. Knitter also comes from the wordings like knit and weave. So, it was already created to knit different languages and currently supports 50 languages, which you can call via knitter. and knitter mainly is used to create files in html from our markdown or in latex or pdf and in knitter you can combine markdown or latex code together with code chunks which you will see there in the middle. And with the engine parameter of a code chunk, you can set the language you want to use. And there you can enter a Python code and doing all this stuff we have seen and the previous slides is sure is as valuable as well. And the... Feature of reticulate is that it allows knitter to To access objects from from different code chunks, so when you have Python code chunk and Another code chunk in our then it's possible to have access to all the objects from the previous code chunk At the end I want to show one example so for example we have data in R for simplicity I use the the iris data set which is already existing in R and assigning it to the object X and then there are a few variables and in this data set most will know it from machine learning use case you want to predict certain flowers and let's imagine we want to use the random forest classifier from scikit-learn to do it and on this and the following slides I have three ways how you could achieve this so one uses the first uses rather much our code so you use the import command to import the entire module then have it as sklearn object in your variable and then you can access it. It has the same properties as in Python just access it with the dollar sign to create an instance of a random forest classifier then use one method of this object the fit method to create your model and inside fits you need the X variables where you could here use NP arrays or Python object of the first columns from the from the data set and why variable so your the variable you want to predict just as an array and then the object is created all methods of that object for example feature importances available into the our session you can have a look at it and as well use the model for further prediction so it's not that sophisticated but I think it will give you a glance what can be done with it and as an alternative the possibility to source a function so let's imagine we have the Python function with the name classify which does all this in Python already so you import the module you create the module and return it and then there is the source Python command you give that parameter the the file name and the feature the function classify is available in your workspace you can give the the X object as an input of this function and then get the same output as on the previous slide also you could do a prediction if you want to do even more With Python code itself, there's the possibility to use the REPL, the interpreter. So, then in R, you just have the data. You use the REPL, do all your modeling inside the REPL, and then exit it. after that you have access to the results from python with pi dollar variable name here it's not printed as i i also did the slides with with nita and the um latex functionality it's only there it's the reason that they do not show the python output However, when you do it in a console, you have the results there. Now, I'm already at the end. A short summary. So, there are different possibilities to combine Python and R. You can use existing features. you can also use the language of your choice where you feel more familiar with and you can make a transition for your Python friends easier for your R friends and also create R interfaces for available Python modules and now I want to thank for listening and feel free to ask any questions you have. Thank you. Are there questions? Yes. Yeah, how is it with the logging? If you want to debug some stuff in Python or R, how does that work together? Can you repeat the question in the microphone? Yeah. So the question was about the interoperability with the logging module. Personally, I did not try it. We used Reticulate for a use case in the company at the end where we did the data management in Python and used ggplot for visualization. I cannot give any insights regarding the logging module. However, the import will as well work with the import function and make the methods available. And I'm pretty sure it will work from there. There is no more questions, please give them an applause.