Use Spark from anywhere: A Spark client in Python powered by Spark Connect

Speaker 1 [00:02]

Thank you so much for having me. Yeah, so thanks for bearing with me. Last talk of the day. It's always hard, I know. Everyone is a little bit sleepy. But, you know, there's exciting stuff in here. You know, first question might be, wait, what are we doing here? You know, like this PyCon, PyData, shouldn't this be really Python? This is what I want to start with. Everybody knows Spark, but Spark is this weird Scala beast, you know, like JVMs and stuff. Like, wait, what does this have to do with Python? So my number one question, really take this from a previous speaker because I like the interactivity. So how many people here have used Spark before? Oh, wow. Ooh, nice. Ooh. I'm a database person. I come from C++ background. All of the Scala stuff is still new to me a little bit. So how many people are using PySpark in kind of a regular basis? Oh, man. That's awesome. So how many people love debugging PySpark stuff? I mean, one person loves it. Wow. That's much more than anticipated. The reason is the evolution of Spark. Spark, back in the days, built in a way that we typically today associate with large data centres, large clusters. We've built all of these applications, and everybody that has ever deployed this ML app, this IoT app on Spark, essentially really boils down to deploying it together with Spark. There's There's no concept really of deploying an app next to Spark. It's always one thing. And the reason is that everything in Spark essentially is really tied down into the Spark driver. This is like this whole JVM interop between Python and JVM, this is really hard. But data applications that we've seen in many different places, they don't live in data centers anymore, just in data centers anymore. Everybody wants to use PyCharm for development, VS Code, Jupyter Notebooks, we want to write web applications not just in Python because we love it, but there's other languages like Go and JavaScript that people want to use, and on the other hand we have these edge devices. Everybody has a phone with lots of computing power, we have all of this other stuff, and the question is why can't I use Spark from these devices? Why do I have to always think in Spark in terms of I have to deploy this using Kubernetes, I have to go to Amazon and pay lots of money, or Databricks, yay, but at the end of the day, why is this so complicated? And really if you zoom in, as I said before, your application logic sits right on top of the whole Spark stack. So it's right next to the analyzer or the optimizer before it gets scheduled and executed. But if I want to embed Spark in my application, I get to this kind of big problem. I have to use SQL. I mean, SQL is awesome, don't get me wrong. But everybody that has tried to build a dynamic app using SQL knows that it ends up being like string concatenation. Like everybody that ever said, like, oh, you know, let's just add a dynamic filter to my SQL string here. Yeah, right. You build, like, a CTE, and then a with statement, and another one, but wait, how do I debug this? And then, you know, you talk to a data engineer, and they say, oh, have you used data frames? And you say, yeah, no, because, well, data frames don't really work there. So the question is, like, SQL only, but if I want to use my IDE, my notebook, the things that I'm used to, I'm a software engineer, you know, I don't like notebooks for developing code. awesome, for interactive stuff, for doing stuff really ad hoc, but if I want to develop an ETL pipeline, I don't use a notebook. Don't get me wrong, if I say this out loud, half of my company is going to yell at me because they believe notebooks are awesome. They are, but for their own kind of purpose. But if I want to use an IDE in a notebook, I have to basically deploy my code close to the REPL. The REPL runs right next to Spark, which means I have to run Spark locally or I have to SSH to know that I run Spark, and this is complicated. If I want to use a programming language that does not have any kind of interop with the JVM, I can't use Spark at all. If I want to use Go, I'm kind of toast. There is Spark R, but everybody that has ever done anything really with software engineering and Spark R, I mean, it works, but, whew, you know, it's a whole different beast. All right. So This is the reason why over the last year essentially we built Spark Connect. The whole goal of Spark Connect is to build this thin, client API that allows you to use the power of Spark from anywhere, like using it from your IDEs, from your applications, from all these different programming languages. How do we achieve this? When we looked at the way how the Spark API is essentially built, like the DataFrame API, the key APIs that have evolved over the last ten years, this is really like we have this DataFrame API that gives us a declarative perspective of how to access the data. If I'm saying Spark, read table, select, write and insert into, all of this is declaratively identifying what kind of operations I want to do on my workloads. We take this API, we translate it into a logical parse plan. Why parse plan? Because on the client side, we don't really know, like, is this a table, is this a file, is this Parquet, is this CSV, This is JSON. This information is handled best at the Spark server, so we want to push this there. We then take this unresolved logical plan, we send it over the wire using gRPC, and then we process it on Spark. You can use all the plugins that you're using in Spark, you can use all the power of Spark, the Spark ecosystem that you have today, continue using it, and use Spark Connect, and then the results are streamed back using gRPC and Arrow data to the client. So we started the project with this Spark improvement proposal roughly in May 2022, and now with Spark 3.4, actually the first version that Spark 3.4 was released last Friday is available as open source, and you can use it, you can directly get started. So what is the status of Spark Connect in Spark 3.4 today? Essentially if you're using Python for all of your data science and all your data management or data engineering processes, it just works out of the box. You can use DataFrame APIs, UDF, DynamicConf, all of this is there. We even have an experimental Scala client that allows you to use more or less the untyped data set APIs from Scala and use it as well and connect them remotely from IntelliJ, for example, to your Spark cluster without SSH-ing to something. However, there's a trade-off. This sounds like, oh, man, this is so awesome, why haven't you done this 15 years ago? And the answer is, Spark is 12 years old. So there's a lot of evolution that has happened in the different APIs of the system. And there are parts of Spark that still exist, they're still powerful, but they're not really client-friendly. So RDDs, for example. I don't know if people remember the RDD paper from 14 years ago already. I mean, I'm way too old, it seems. This was the beginning of why Spark is so successful. And RDD APIs was this kind of like this powerful improvement of a MapReduce and really showed like what you can do with a new kind of in-memory processing. But the downside is they're very low level. The DataFrame API is like high-level declarative implementation, and RDDs are this kind of low level, I really know what I'm doing, the assembly, so to say, of like data processing. But doing this on the client side, if you want to debug from your Python application, this is really not the way what you want to do. So here's really where it's saying, okay, please don't use RDDs, and we don't support RDDs in this API. And essentially, we are giving a simplification from this kind of three different APIs that you have today in Spark to only two focusing on the declarative APIs that can be optimised using the built-in optimiser catalyst and so on. If you have your application and you want to migrate it to Spark Connect today, there's a couple of things to be aware of. Number one that is, you know, for some people a little bit surprising is you don't have access to the driver JVM anymore. There are surprisingly many, like even Python applications that somehow use Java under the hood. And you might not even be aware of that, but this is something that, because now your client doesn't run anymore on the Spark cluster, you don't have access to the driver JVM anymore. In addition, the Spark context is this kind of like application singleton is not available in the same way, and we believe this doesn't fit really on the client because it's a server side concept. So what's the UX then? What does it actually change? So to basically show you how to connect to Spark, a remote Spark cluster, we added this this kind of remote property to the Spark Session Builder, which identifies a Spark cluster given a connection string. The connection string of Spark Connect is very similar to any JDBC connection string, so basically it's saying like, hey, this is my host name that I want to connect to, that's a port number, and so on. Spark Connect as a protocol, as I mentioned, is gRPC-based, so it's standard HTTP2, so wherever you can deploy any kind of HTTP2 traffic today, you can use Spark Connect, it works really easy. In Databricks, we run it behind an Envoy proxy. I talked to tens of customers in the last couple of weeks. There's like NYProxy. We talked about integrating this in QoB. And there are many different open source projects that give you this kind of flexibility of deploying HTTP2 proxies. So in Spark 3.4, as I said, Spark Connect is available. It's publicly documented. It's all in docs. And you know what's even better is in Spark, essentially, now they have these kind of live notebooks that are supported by MyBinder. So it's basically running a Docker container with Spark inside where you can run interactively, try this out in your browser, and run this Quick Start notebook for Spark Connect to really get an idea of how to use the system. All right. But the thing is, all of this is open source, but I get paid by a different company. And so I want to talk to you a little bit about where does Spark Connect come into play for Databricks, and why do we believe this is so important and such a game changer for us as well? So, as I mentioned before, you know, Spark Connect gives you the ability to connect to your data lakehouse, like, from anywhere. And this is really what we care about. Like, Databricks is the number one provider for data lakehouse platforms for giving you the ability to collect your data, process your data, do machine learning on top of it, integrate this into your MLOps flows and everything. So, wherever your data is, whatever your problem or whatever you're trying to do with your data, we basically give you the opportunity to integrate with us. and then using Databricks Connect, which is built on top of Spark Connect, you can then integrate from all different areas. So this is not just IDs and notebooks, but this is as well data applications. So you've seen outside, there's a booth on Plotly. We've shown how to integrate with Plotly, Streamlit, all of these new kind of data application tools that give you this interactive experience. And then, of course, we have partner integrations using DataIQ that are all using the same API to build new kinds of applications for the data lake. So, why do we care so much about IDE-based development? Because developers like it. If you're using Databricks today, you're thrown into web interface and then you have a notebook and you get started. This is awesome. You build your first pipeline. You schedule this as a job. This is great. But at some point, you're asking yourself, how do I write a unit test? How do I test this? And then somebody says, oh, I found a GitHub project that allows you to run PyTest in a notebook. And you say, awesome. That's really what I always wanted. this is what I dreamed of my whole life. I don't. I still believe that unit tests are run locally. And the question is, how do we give customers and users the ability to use all the muscle memory that they learned over time? They all went to school, university, whatever, wherever they learned their best practices, we want to give them the ability to use this. We want to say, okay, whatever you learned and whatever fits your job the way that you want to do your task the best, we want to give you the opportunity to continue using this. So this is why we believe it's not just in the notebook, it's from everywhere. In Databricks, we had something called DBConnect v1. This was our first approach to this topic, and it works. It's kind of clunky. It does its job, but it's not really extensible. It's not really fast. And, you know, like, this project has been in maintenance for a couple of years, and now, basically, with SparkConnect, we're pushing a completely new way forward in a much more lightweight way. So what it allows you, dbconnect, is whenever you have a Databricks cluster somewhere running in your infrastructure, you can connect from anywhere, from the IDE of your choice, being at PyCharm, being at VS Code or whatsoever, and you get the benefits of not just having debugging, but the larger your company gets, the more sophisticated your processes are around data, like data governance becomes important for you as well. So it's no longer just, oh, can I write my unit tests, but it's like, can I write my unit test in a way that still respects all of the data governance rules that I have set up in my company. So, and this is really what we want to do. And to get started, it's very simple, you know, like since today, essentially we're going public preview today, pip install, database connect, depending, version larger than 13, and now you can connect to your database cluster from anywhere directly, from your laptop, from wherever you want, and it's very simple. To get started, all that you need, as I said, you need the package, you take your workspace URL, your cluster ID, and off you go. I know this is kind of like, hey, this is not a Databricks show, but if you look at the slides later on, it's very easy to get started. So again, our goal was how do we make this backwards compatible? For us, the most important part is there's so much code written today in PySpark that you want to continue using. We don't want to push on you, or we don't want to prescribe you a particular way how you have to continue writing your code from now on, we want to make sure that the same open source way of using PySpark continues in the same way, and that you can simply use it, but you're using it now on Databricks. So if you take this example, you can run this on legacy, on plain vanilla PySpark, it works. You can run this on Databricks, it works. You can move this to DB Connect, and it works. And this is what we wanted to achieve. We want to make sure that whatever we're doing, it's compatible to the existing versions. So this is just an animated GIF that I did some time ago to show you, hey, how easy it is now to interact with a remote cluster. In this particular example, I just wrote a simple test where I'm scanning a table on my remote cluster, and I set a break point because my test was failing, and now everything that I'm used to being able to debug my code locally in Python, I can still do, so I can get all the benefits that PyCharm offers me in this case to inspect all the values, I can interact with the system, but my PySpark is not running locally. I don't have this PyCharm, I don't use what's called IntelliJ fleet or something like this, where you now have remote running instances of your IDE. No, this is my PyCharm running locally on my MacBook that connects using Spark Connect to a remote cluster, and this can be whatever. It can be single node cluster, it can be 100 node cluster, it can have a terabyte table, it can have a 20 megabyte table, it doesn't matter. All of the data that is available to me, I can use. And this is really the power, because imagine you have your complex ETL pipelines, and you're trying to debug why, for whatever reason, your zip code all of a sudden contains a letter, while you didn't expect this, because in Germany all the zip codes are actually numbers, until you realise in England that's not the case. But how do you do this? Yes, you can write a unit test, but sometimes you just want to look at the data. And this is really what the difference... Again, it brings back this combination of the ad hoc interactive analysis of the data and combines it with the standard software engineering approaches. In addition to just local development, this enables a whole different area of completely new use cases. It becomes very easy to write completely new interactive data applications with very little code that you need. So for example, as I mentioned, we have a public GitHub repository that shows how to to use Dash and Plotly with our sample data sets in I think roughly, I would say net lines of code is probably 100 lines of code to give you this kind of interactive map visualisation where every time you're changing something, it actually goes back to the server and runs a full query. Yes, of course, you could do this today already, but just with SQL. But again, now you have PySpark available, you can really leverage all of the benefits of that. And then another one is we have a partnership with DataIQ, where DataIQ is a provider that builds low-code, no-code ETL pipelines, and they have a breakout where you can write PySpark code, essentially, to integrate with Databricks. And now they don't have to run this as a notebook or a job in Databricks. They can interactively trigger this from their pipeline, making it very easy to combine SQL pushdown with PySpark pushdown and all of the other operations that they have in their systems. And, you know, As I said, you integrate from any application, like you can take this example, this is the most minimal, it doesn't produce this awesome map there, but the most minimal example to run a dash app is exactly these ten lines of code that you can deploy in a Docker container, run it anywhere, and just use it, and then you have basically a new application deployed using Spark. You don't need to explain why a new kind of data management system, you can continue using all of your existing data science tools that you have. integration as mentioned, DataIQ really the ability of pushing down additional PySpark code directly to the Spark cluster and making it available so that because, again, the interesting part is coming back to the previous talk, you're not the only persona in the company. The data scientists writing code, of course they know what they're doing, but there are many users out there that really look at this and say, like, I understand half of it, if if I'm lucky, but most of the time I can roughly point and click, and that's why there's so much success in this kind of no-code tools, because they give you the ability of onboarding users and onboarding knowledge that you couldn't access before. However, most of the time these kind of tools had a limitation that there's a limit to the complexity that you can express. With the integration that we're doing here with DataIQ, you really get the ability of saying I can onboard the users that have a good understanding of the business case and And they can express it. But in addition, I can offer them the ability to push down even more complex logic if they need to. Again, this is my job as a database personality. I have to basically give you an idea of a roadmap. So we went to the public preview today. The US is waking up. So I'm expecting the blog post to be published in the next hour or so. So today we have support for Python and PySpark, MLflow support is there for single node ML, Authentication and standard token-based authentication at it was an Azure GCP is coming the next couple of weeks I know as we are marching towards GA of the product like our our goals to have full support of the pandas on spark API as well The ability we will run Scala support in a private preview and as well full data support, you know Like for us, it's not just the on the client side They just reading the data But it's well writing the data using the power of the open Delta Lake to be able to integrate and as part of that comes as Well like making sure that we have the full enterprise integration auth and Azure AD and so on and then in the longer-term roadmap You know our again comes up because there are customers especially in financial sector that are using our and it's when it will not go Away, I mean we had the discussion earlier about like Cobol, you know, like it's been dying to it's been trying to die for a long time time, it's not happening. I think R will have the kind of same faith. It will remain there. And we have to make sure that we pick up the users from where they are. And with that, I conclude my presentation, and I'm happy to answer any kind of questions that you might have on this topic. Thank you so much.

Speaker 2 [20:00]

Thank you so much Martin, we have a bunch of questions on Slido So I'm just gonna jump into the first one does spark connect provide a mechanism to instantiate a remote spark cluster

Speaker 1 [20:13]

No, Spark Connect is really more the connection protocol. There are two open source projects that give you the ability to create the Spark clusters. One would be Kyobi, which is a relatively new one that is more like a SQL gateway on top of Spark, and then there is another one that I forgot which name it is, it was kind of dead but was just recently reanimated. So Spark Connect's job is not like instantiating the Spark cluster, there are other open source projects doing that, Spark Connects remodeled the protocol.

Speaker 2 [20:47]

How does UDF execute it now?

Speaker 1 [20:51]

So, the UDFs, the Python UDFs are executed in the same way. And so, basically, we... And this is... I don't even dare to say this. Like, we Cloud pickle the UDF closure and we send it over the wire. And this comes with all of the downsides that you've heard before. That's unfortunately the way that in PySpark the UDF execution works. We don't try to change this. If anybody here is interested in, like, doing something that is better than Cloud Pickle and more portable, like, please let me know. I'm totally happy to figure out a way how we get this in PySpark. But I think Cloud Picker, unfortunately, is the today's solution.

Speaker 2 [21:27]

Is the same Databricks connect code I wrote locally be prod compatible if I were to deploy it to Databricks via CICD or Terraform? Or do I need to adjust it?

Speaker 1 [21:40]

No, essentially the goal is that the code that you're writing locally,

Speaker 2 [21:40]

No.

Speaker 1 [21:44]

even if you're using PySpark open source locally, that this code is prod-compatible. We want to improve the fidelity of the inner dev loop of the engineers.

Speaker 2 [21:53]

Is it possible to see afterwards from which source the code ran? For example, for a cost analysis to check whether unit tests that need clusters are increasing costs.

Speaker 1 [22:04]

So, you have the visibility into from where the calls came mostly by the users that ran on your clusters. So, the same cost attribution that you're having on your existing database clusters, all of this works in the same way. So, there's no specific check for, like, logging for dbconnect access.

Speaker 2 [22:24]

For unit testing Spark apps in CICD, is it possible to have a local Spark version or environment in standalone mode that is identical to the Databricks runtime?

Speaker 1 [22:35]

Completely identical to the Databricks runtime is a little bit hard, because as Databricks, we have our own belts and whistles that we have around our Spark runtime. However, from a core functionality perspective, a local PySpark gives you roughly the capabilities of the respective Databricks version. Of course, if you're installing your local libraries, you have to make sure that they're installed as well on the database cluster.

Speaker 2 [23:02]

So far, in my experience, PySpark is much lower than native SQL in Databricks. Is it the nature we only use PySpark when SQL couldn't do more advanced analytics?

Speaker 1 [23:16]

So, in theory, that shouldn't be the case, because PySpark and SQL boil down to the same execution engine. They boil down to the same execution plans and run through the same analyzer and optimizer. A good example to see this is if you're searching for Spark SQL, you're actually not finding SQL text, but you're finding all the SQL, Spark SQL means data frames, essentially. So that's why it's very unlikely that SQL code, like string SQL, can be much faster than PySpark because it's the same. If you're injecting RDD somewhere in the middle, it becomes an opaque optimisation blocker. If you're using UDFs, obviously they become slower. So sometimes I've seen customers and users basically instead of calling the PySpark SQL function like upper, basically creating a UDF that does string upper, because it's convenient. They didn't remember the function name in SQL, so they just wrote a small UDF that does this. Obviously, this is tons slower than using the native SQL functionality. But otherwise, it should be the same performance.

Speaker 2 [24:26]

When would you recommend using one of the other methods to run Spark in IDE, namely VS Code plugin or Databricks extension DBX?

Speaker 1 [24:34]

So, the VS Code extension, like, we're integrating right now with the VS Code extension. The VS Code extension will be updated as well to support the native debugging capabilities. The reason for using the existing, let's say, DBX way of synchronizing the code to Notebook and Databricks is mostly if you're using not the most recent release of the Databricks runtime. So, if you are, for unfortunate reasons, stuck on an old release, then you have to use the old ways. can migrate to the Databricks 13 release, which was released last Friday, then you can use the new way of connecting to that.

Speaker 2 [25:11]

Are there plans to integrate Kerberos off?

Speaker 1 [25:14]

No, and the reason is, in Spark, generally, we try not to be prescriptive of the way that you deploy your cluster. If you look at Spark, you can deploy using Kubernetes, using yarn, using whatever you want, like local cluster mode, and Kerberos essentially is one way of doing authentication, but this is something that we believe should be handled outside of Spark Connect and outside of this particular implementation, this is something where a proxy in front of Spark Connect is a much better place to do this kind of authentication than trying to hook this into the Spark Connect protocol itself.

Speaker 2 [25:52]

How can the cluster dynamically check if it is started locally or from Databricks UI? Because depending on that session, construction differs, doesn't it?

Speaker 1 [26:05]

I'm trying to unpack this. So the easiest way to check how the cluster was started, I think, is checking the master property in Spark and the SparkConf, which should tell you if you're running in local mode or if you have a master UL set. But I think I'm not 100% sure what the question tries to get at, so it would be great to rephrase it and give me a little bit more hints on where this goes to.

Speaker 2 [26:31]

We still have time for questions, so whoever has that question can re-ask it again. Or maybe we have another question from the audience. Or we can go to coffee break five minutes early. What would you prefer? Okay. Thank you so much, Martin. That was a great talk.