Big Data Systems Performance: The Little Shop of Horrors

Speaker 1 [00:02]

So I was told I really have to stand here, otherwise a cameraman will be... There's no cameraman, that's a static camera, but typically you try to run around a lot. If I run around too much and you can't hear me anymore and don't see me on any screen, just raise your hand. So a little bit about myself. So basically I'm a researcher in database systems or big data analytics. I did a PhD a long time ago, and the three major topics of... This doesn't work. So the three major things I'm doing is large-scale data management. Basically, we look at large data sets and try to analyze those data sets, integrate those data sets, search stuff in the data. The second thing I'm working on is systems. So I like to build stuff. It's not so much about proving stuff that something in theory might work. We actually tried out and built prototypes to understand whether something works. And the third is indexing. So indexes like maybe you know it from undergrad classes, binary search trees, is a popular example of an index. In database systems, we use different types of indexes, typically B-trees or something that looks very similar. And other than that, I did a couple of other projects throughout my career, so I worked a lot on column stores. Hadoop, I also did crazy stuff. Actually, I was part of the German Science Slam finals in 2012 that happened here at ZKM. I only scored fourth place, but still was pretty good. Yeah, and more recently, I started working on data science stuff, of course, Python, and also blockchain. Blockchain is pretty interesting stuff. That's a new buzzword in the room. This is 90% database technology. It's very interesting. It's often presented as a completely different thing, but it's basically databases. And so the major thing I'm working is, you could visualize what I'm doing basically with this picture. So there's big data on the one side and machine learning on the other side, and where the two worlds collide or meet, there's interesting stuff going on. So you might wonder, where's NoSQL? Well, it's here, it's there to replicate, of course. Yeah, and what I wanna talk about is all these buzzwords. Each of us uses all these buzzwords, and some guy might say, hey, big data is so cool. Another guy talks about data lakes. Who of you heard about data lake? Oh, wow, that's impressive. Oh, that's a PyCon conference. NoSQL is another buzzword in the room, or Hadoop. And it a little bit resembles like a story from the Bible, like the Tower of Babel. People, like, speak different languages. Maybe they mean the same thing eventually, but they kind of use all these different words. It's like different languages. That's one way of looking at this buzzword world. And the other is an analogy for medicine. So if you remember, like, 200 or 300 years back, there wasn't, like, you couldn't study medicine. There were quacks. They would do quakery. that were like totally non-experts claiming that they would have the medicine for everything. They would go to the markets and say, oh, this little water will cure all of your diseases. Just buy it from me for 500 bucks and it will cure everything. And that was Quaker-y. And the layman would believe this stuff, of course. And it's very similar to what you observe today. So there are people claiming, hey, the data lake will cure all of your problems. I've heard that so often. It's total bullshit because, particularly if you ask into it, what is the data like for you anyway, right? And then it turns out it's often not more than a five system. Well, interesting. And the other, you could do the same thing for AI or as we had Pyke and you could do Python. Python will cure all of your problems. That's the only sentence where that is true. I'm just saying that because I want to get out of here alive, right? So you're all intelligent people and you would say, I don't fall for that trap. That happens to other people, right? Well, let's look at a concrete example. Assume you want to buy a TV. So how would that work? So let's assume there's a TV shop and there's a TV expert and there's a laywoman who wants to buy a TV. And don't worry, for gender equality reasons, I will switch roles in a moment. So for the moment, the laywoman, that's a non-expert and there's a guy that's an expert. And then the guy starts advertising this stuff. This model even has hyper-ray plus-plus rendering. whatever that means it sounds fancy but maybe who cares you wouldn't ever need that yeah or 500 hertz rather than 250 hertz you can't see that don't worry no way you can differentiate but yeah sounds great and you will pay the extra 500 bucks for that ultra hdmi yeah who's ultra hdmi doesn't mean anything and now how would that look like if you turned that into a big data shop a big data system shop. So, of course, we don't need TV sets. We need database systems or big data systems represented as cylinders. And as I said before, we will switch roles for the moment. We have to adjust the wording here a little bit, of course. And the guy is now the layman, and the woman is a big data expert. So how would that work? So she would advertise that. The leading no-seeable data lake solution in the cloud. I stole that from a web page. That's an actual sentence you you will find on a web page, I didn't make that up. Or blockchain enabled. AI ready, like HD ready. Build on the Lambda architecture. IOT, I-I-O-T, this exists, or even I-D-I-O-T. But she's just thinking that, she's not saying that, of course. Yeah, well, what's the problem of all of that? I think there are two major problems we are witnessing here, And the first is ambiguous communication. So what does that mean? It means that one guy says big data, and the problem with that is big data is just a symbol. Whether you say it or whether it's written down, it's just a symbol in terms of the language. It may mean very different things. So it may mean large data. Many people understand, oh, it's large data, right? Or maybe the four Vs, velocity, versity, also totally doesn't work, the definition. NSA, in particular for laymen, oh, big data. This is mass surveillance, right? or Spark, MapReduce, you name it. And now the problem is if some other guy or woman here said, oh, he's had big data, that guy may have a totally different interpretation of what big data means. So it might be that for that guy, big data means MapReduce, whereas for this guy, big data means large data. Hence, big data got translated or large data got translated to MapReduce because they used an ambiguous symbol for that. So it was large data in the brain of this guy, this got translated to MapReduce in the brain of that guy, hence all the confusion starts. And the more buzzwords you use, the more there will be confusion in the room. In contrasting clear communication, a great example of that would be a term like relational algebra. So who of you know relational algebra? So as a professor, I need to say at this point in time, please learn this. If you don't know relational algebra, it's really at the heart of all data processing. No, at the heart of data processing is not Pandas or Python. It's relational algebra. Pandas is just a variant of relational algebra, but to understand data processing, you need to know that. Typically, people who had an undergrad class will have seen that, and it's very clear what relational algebra means. It has one meaning, and the likelihood is high that if someone else hears relational algebra, he will have the same interpretation. So, in other words, here relational algebra got translated to relational algebra. This is clear communication. What you wanted to send was received as is without any further modification. And that's very important. So, with that, you can do a symbol-to-meaning landscape. You could also name it a symbol-to-bullshit landscape. And basically how this works is, on the x-axis, I depict the number of meanings associated with a specific term, from one, few, many, to infinite amount of interpretations. So here on the left, you will see those are terms, those are core techniques from data analytics. They exactly map to one meaning. However, the further you go to the right, you'll find terms like large data. Large is relative. What does that mean? Exabyte, petabyte, megabyte? I have no idea. Data science is very ambiguous. Some people interpret it as being the same is machine learning, which doesn't make any sense. It also covers data mining, data management, blah, blah, blah. Deep learning is ambiguous, MapReduce, Hadoop, and so forth. The further you go right, the more ambiguous it becomes. Here, for instance, you see the buzzwords that have the highest buzz, so to say. So big data, AI, NoSQL, and blockchain. There's no clear definition whatsoever. And don't try to even try to define it. It doesn't make any sense. And my advice is really, if you have a technical conversation, try to use words on the left. If you try to get venture capital, try to use words on the right. Okay, so that's my advice. And the second is this confusion of dimensions, dimensions of terminology. And there are three dimensions you see in these technical discussions. So the first is fancy-sounding buzzwords. We've seen them. Big data, AI, NoSQL, you name it. But there's a second dimension in communication. Those are the actual technical principles that you use to solve any data managing problem, any data analytics problem. That's dimension number two. And examples of that are those, predicate pushdown, relational model, relational algebra, and so forth. Let's go through some of the principles for a moment, just to remind you, if you haven't seen them, what they are about. For instance, for predicate pushdown, it's a symbol, but has a very clear meaning associated to that. That is, filter and project data as early as possible. For instance, assume a company that has a satellite. The satellite takes pictures of planet Earth and sends terabytes of those images down to Earth. Then on Earth, you filter those images. You're searching for whatever feature in those images. In contrast, assume you could also filter the data directly on the satellite And only the results are sent down to planet Earth. By that, you would save bandwidth. You don't have to transfer the entire image, but just the results. That's called a predicate pushdown. So the predicate is sent to the data sources. And basically, it's one of the most fundamental patterns in databases. Any decent relational database system does that automatically for you. And I could talk for hours just about that or write a book about that. The other is the relational model. So you model all data as multi-attribute sets. That's one of the pillars of data management. And if you don't know what the relational model is, please learn this. That's the first thing is to learn the relational model, and then its implementations like in Panda. Because that's not what Panda is doing. Pandas is doing. Typically, this is confused with tables, data frames, column stores, row stores. That's not the same thing. It's a conceptually different thing. And we will come to that in a moment because that is dimension three. That is concrete software platforms. I will get to that in a moment. And the third pillar here is relational algebra. So you query those sets through a combination of simple set-valued functions. That's again different to what you see in Pandas, which is not set-oriented and has kind of a weird interface here and there. So this is a very lean and simple interface. And every database system uses this stuff internally for query optimization. And if you understand that, You basically understand most of query processing you ever need to understand. So those are really concepts, and they appear everywhere, everywhere in data management. And here just as one example, those are all three concepts combined. Basically what you see here is what we call a query plan. Here you have two sets, two data sets. One is called R, one is S, and the data flows from bottom to top. And basically, those two data sets go into an operator. That's a join operator. So you join these two data sets. That's part of relational algebra. And then the results from the join are filtered. That's a filter operator. That's also relational algebra. So what this means is you join these two data sets, and then you keep only those tuples where the name equals yens. And that's basically what this planner is saying. If you haven't seen that in your undergrad class, please check out my YouTube channel. I have a couple of videos explaining all of this stuff there. And what a predicate pushdown would do here, it realizes that you could filter out before doing the joins. Rather than doing it after the join, you push it down, you push it through the join, and then it looks like this. So you first take the data set R, you filter and keep all the tuples where the name equals Jens, and then you push them into the join or pull them into the join, whatever, and join it with S. And that's way cheaper than the stuff on the left. And many tricks can be done if you realize this pattern. So that's dimension two. There's a third dimension I already alluded to. That's dimension three. And that is all about platforms. So we talk about this a lot here in terms of there's a Python library XYZ that does whatever. And I just want to advertise here that's all cool. There's so much cool stuff out there. and there's so fantastic libraries for Python, but to really solve a problem, it's not at the level of library, you have to go back to the concept first, and then you think about what is the library. That's really, really important. Yeah, and to illustrate that, I'd like to tell you a little war story. And now you will see again all of these things I explained in the first part of my talk. So let's try to apply this idea of separating really the concepts, separating these three dimensions. So that's basically what we've seen with a certain customer or client. So that was a description of the problem he had. So basically the client has a big data problem. You understand what the problem is about now? Because he said big data? No, we have no idea, because again, this is ambiguous, may mean any of those things, so we have to find out what that means. He has a Hadoop cluster, is that clear? Is that ambiguous or non-ambiguous? Any volunteers? How big? The cluster? That doesn't matter. You don't have to look at size here. Even the term Hadoop cluster is ambiguous, of course, otherwise I wouldn't draw the green rectangle around it. So there's a cluster with, it may mean a cluster with HDFS installed, it's a distributed file system, or it's Hadoop MapReduce. And that's typically confused, because what many people do these days is, okay, you have a cluster with HTTPS installed, on top there's Spark. Yeah, but it's not Hadoop MapReduce necessarily, so that was ambiguous. In this case, it was a cluster with HTTPS. Yeah, then they did some Spark, wrote some Scala, Parquet, whatever, right? And this is already a confusion, because Spark is at dimension three. That's a platform. That's a concrete implementation and framework, And the question is, why do we have to worry about that at the problem level already? I first have to solve it conceptually. So that's the confusion. But, of course, if you remember from school, there was Moses, and he went down from Mount Sinai, and he brought the 11th commandment. If there's big data, thou shalt use spark on my produce. That's the 11th commandment. It was lost. It's not in the Bible. But that's how it is. And, of course, this is total bullshit. Because, I mean, just by – but you see that very often. People say, oh, that's a big data problem. I probably need to install Spark, right? No, that's really not the right reaction. Well, anyway, then the next thing was program. Well, program is a bad term from the point of view of a computer scientist. I'd like to talk about software or software architectures, layered architectures, libraries, stuff like that, but not program, right? Program to me sounds like, well, is that spaghetti code? Is that some flexible query processing involved? Is it a clear layering of an architecture? Is it some monolithic fluff? I don't know. The program was really weird. And what we found was something like that. So basically, the customer tried to engineer a car. He didn't know that he was re-engineering a car. He didn't know that the concept of a car existed. He just wrote code that would do something similar as a car. And then he ended up with something like a cardboard car. In contrast, he could have done something like that. He could just get a state-of-the-art car like this Tesla. Actually, this Tesla is from 2015. I was informed on Twitter. But it doesn't matter, the comparison still holds. So sometimes it's better to just get a car and pay the price rather than reinventing a car and paying all the maintenance costs, reinventing the wheel, being way slower than, this is way slower than that one I would expect. So it's a clear trade-off, you better get this one. However, the client didn't do that, and he was wasting so much money, so much money money for burning that maintaining that and stuff like that so let's continue um there were more details so we learned it's all about sensor traces stored in hdfs using parquet so here i should write shouldn't write parquet i should write arrow right you know arrow most of you it's a column layout yeah it's more there's the same thing just the main memory as parquet Parquet is a little older and basically allows you to speed up analytical queries. However, here it was used in the wrong way. So the company was believing, oh, it's pretty fast, but then looking at the data, we learned, no, it doesn't make any sense here. Because the data was something like that. So that's the data we looked at. Basically, those were traces from sensors. They had some machine that is monitored using sensors and there are thousands of sensors, so basically you had logs of this type, there was a timestamp, a sensor ID, so which sensor measured something and then the value measured at that time by that sensor and then there was some fluff no one really cared about. So then you would basically get such a long trace and then you would see through those traces with the SPARC program. And the queries that they're interested in are something like that. So when is sensor x operator value? So as an example, when is the value of sensor 42 greater 15 or as a conjunct, when is the value greater 15 and sensor 77 smaller nine? So basically you were looking for time intervals where that would hold true. And important to note here is that there's only few attributes in each query, not the thousands of sensors that are available, just very few of those. Well, and so what we did was we basically understood that if you have, oops, if you have queries like that, it doesn't mention any of the fluff, so the fluff that's seen here can easily be removed. It has removed all of that from the data, and basically we already have much simpler data. And that was trivial, and the next thing is the thing that they were using Parquet, a columnar layout. They could have done that with Arrow. It would have led to the same problem. So there was no clustering here on the attributes, and that's exactly what you want to need, what you need. So we changed the format from timestamp sensor ID value to sensor ID timestamp value. It changes the lexicographical order. Or if you wish, you could also do it differently. You partition the data by sensor ID. It's just a very easy and old trick from databases. It looks like that. So basically now you don't store them in timeline order, but you store them first by sensor ID and then in timeline order, which means all data collected by one specific sensor sits physically clustered on disk. It's a very old trick, but it wasn't played here. And that is actually what was intended by using Parquet in the first place. Which means now when you have a query, you only have to look at this specific partition. Only if you're interested in the measurements of value 23, you just look at this partition. If it's 24, you look at that partition rather than sieving through all of the data. That's basically also what columnar databases like HANA do. It's a 20-year-old trick or even older in databases. So all of what I'm saying to you here is not saying, oh, I'm such a great guy. I had these great ideas. Don't get me wrong. What I'm telling you with my talk is those are all very old ideas. And if you apply them well on the conceptual level, you can get a tremendous boost in performance. That's the story of this talk. Yeah, and again, that's a technical principle from database management that means we don't compress data to save storage. In data analytics, you compress data to save I.O. There's a problem in data analytics is getting your data from the storage medium to the CPU. That's a bottleneck. It's not, we have so much, this don't cost anything. You can store whatever you want, but the problem is streaming the data to the CPU, be it from a hard disk an SSD or even from main memory to CPU that's a bottleneck so you rather work on compressed data or you rather make sure that you only transfer little amounts of data and that's what we did here yeah that's basically done what we ended up with and then the format changes before we kept the 23 here you don't have to store it redundantly within each partition you kick it out basically now the 23 is gone, it's only stored once, if you map that to files, you could say, okay, you have separate files for each partition, if you wanted to do it like that. And that alone gives you a factor of 500 speedup. Why is that? Because data analytics is almost always I.O. bound. The problem for us is not CPU operations, but the I.O. And if you gain in I.O., it almost always translates directly to an end-to-end saving in queue response time. So that's what we see here. But there's more you can do. So one thing you realize here is that those, there's a redundancy in those timestamps. Say they repeat this 12 o'clock, 35. It's just one second after 34, but they repeat the entire timestamp, which is totally redundant. So it's an obvious way of compressing it, is to difference encode that. Who of you heard about difference coding? Yeah, some of you, good. Yeah, so that's also now, yeah, it's a standard technique in databases. And basically, now I subtract the different values and end up with something like that. So it just stores the difference to the previous value. This is this plus that. This is this plus that. So you have to add it up, all of those, and then you end up with 36. And that gives you another factor 12 improvement because you save IO. The data you're looking at gets smaller. You have to stream less data through the system. And by that, the entire query runs much faster. Again, multiply that with a times 500. You end up with a factor of 6,000 speedup, just by conceptually looking at the problem. Yeah, but you can do more, of course, because another thing that comes into play is now data locality. You have a much better exploitation of the storage hierarchy because the access to specific sensors is skewed. It's not that all of the sensors are requested uniformly because some of the sensors are more important than others. They will request it more often, and they will walk up in the storage hierarchy. So that's basically the effect that we call, we cluster the data by hotness along the storage hierarchy. And if you remember what the storage hierarchy in a computer system does, it's basically not transparent. Expect when you go from disk to main memory, but all the caches, it's not transparent. Still, you can do something about that by laying out your data in specific ways that the storage hierarchy works better. You don't have to just stick with it and say, oh, whatever the storage hierarchy, I have no influence on that. That's not true. You can play with the data layouts, and then you can speed up the queries. So basically, in a situation like that, it could be that the hot data sits here, the cool data sits there, And this will go up in the storage hierarchy, which means that query performance will improve, of course, for the hotter data. And this is hard to quantify, but we expect a factor 10 speedup, which in total translates to a factor 10,000 speedup. So there's some hand-waving here. It depends to a certain degree on the query patterns that are used by the customer, but I bet that we are totally in this ballpark. there's some positive side effects on something like that because when you do that you realize the data could be stored in compressed format in the first place so currently the sensors collect the data in a highly inefficient way you could directly push that to the sensors yeah that's a kind of predicate push down you push the compression to the sensor to the data collectors and you get another factor 12 in storage the really cool thing is there's no need to heavily paralyze query processing here. So the initial thing done by the customer using a Spark cluster doing fancy query processing on HTFS, there's no need for doing that. You can do that on a smartwatch, all of this query processing because it gets so much more efficient. Yeah, and only the overall data sizes are now the limit for the client. So typically in that specific use case, we learned certain people in that company are interested in certain sensors. so they will only look at subsets of the sensors so they can pull those subsets to their machine, to a laptop. There's no need to use a server for that. And you can do more like anatomy detection or fancy data science stuff, whatever. So process recap. So what did we do? So the client had a big data spark thingy. That is dimension one. We went back to understanding the client's original problem. What does he actually want to do? understand what's the conceptual problem then we combined a couple of fundamental technical principles of data management data layout compression and hot cool clustering that was on dimension two and the synergies triggered io savings of a factor 10 000 which more or less directly translate to end-to-end query time savings and all of that is totally software platform independent so you translate it only afterwards to a specific platform i understand that in practice a customer sometimes says, okay, but you have to do it in Spark, you have to do it in Python, that's okay. But even in that situation, step back and ask yourself, okay, first solve it really on the conceptual level and then let's map it to whatever the customer wants. It's a huge difference rather than directly coding some Python stuff. It's a huge difference. So takeaways from my talk. If you understand one thing from my talk, please, that should be this slide. And now it's the time to make a picture. So, design solutions on dimension two only. That is where you solve problems. Consider dimension three as an afterthought, and dimension one, realize that dimension one is solely about marketing. As a footnote, I recently renamed the name of my research group from Information Systems to Big Data Analytics. Big Data Analytics? Well, that's fancy-sounding buzzwords, right? But it made a huge difference, because before, no one gave me phone calls like, Do you want to participate in project XYZ? As soon as I rename my group, I get fancy phone calls. Hey, you want to have that money or you want to be part of that project? Just the term big data is enough for doing that. So marketing, you have to do marketing as a professor as well. And the important thing, please do not confuse these three dimensions unless you want to explicitly fool people, which is not my style. You shouldn't do that. That's basically the gist of this talk. So if you're interested in any fundamentals with respect to database management and big data analytics, I have a YouTube channel. I have about 100 videos there. Half of it is in German, the undergrad stuff. The more advanced stuff is in English. I'll also use that for my classes. Take a look. So it's here, youtube.com user Jens Ditt. And the other thing I would like to briefly mention, I also consult people with respect to these problems. we just founded a little company now we are officially a gmbh it was just two weeks ago it's not it's not official and we do projects in that space for us data science is really the combination of many things machine learning modeling simulation but also all the database stuff when you look at real projects you often see it's it's not this little modeling thing there's so much infrastructure around this modeling thing problem that you have to really tackle well in order to provide a decent solution and here's some of the example projects we're doing so one pet project as we try to predict weather using deep learning so recent recently did a project together with a German weather forecast DVD where we were able to predict lightning so when the lightning strikes both in space and in time and we have a very high accuracy for that so So the German weather forecast was very happy about that. The other is we can also predict temperature relatively well. And we are currently in power with equal performance to the one used by the German weather forecast model. They use a rather complex model used on numerics. We do it with some machine learning tricks and on the same ballpark. Other things we do is we look a lot at logistics. So, for instance, for trucks, you have a huge fleet of trucks. You want to optimize the fuel consumption, want to lower that. You can do that, of course, with machine learning. We have a project in that space. Other projects are with respect to assembly lines. So how do you make the best use of an assembly line and predictive maintenance? How do you avoid downtime in those assembly lines? We have customers in that space. And the other is data silo integration. So you have all these different databases. You want to integrate that. And that's one of the problems that's typically totally underestimated by companies, integrating data sets in different formats from different databases. It's a huge problem, and we have something to say about that. And let me say one thing about data science and ethics. So data science can be used for good and for bad, of course. There was this big affair with Cambridge Analytica in the U.S. that helped Trump gain power. So when we founded our company, we thought about that, and we said, okay, we make it part of our satsung, as our foundational document. We exclude anything that has to do with political or analysis of data in society. If you can read German, that's exactly taken from our satsung. So we're not allowed to work with parties, politics, or influence people with regard to any personal data, because we wanted to make sure that this is part of our Satzung. If it's part of our Satzung, this means if we do it, we would lose liability with respect to GmbH. The Satzung was really strict in Germany about that. We must not do that. Otherwise, we might have to pay a high price, and we thought that was important. So if you're interested in this stuff, talk to me. I don't buy it typically. Thanks for your attention. Thank you very much for this really interesting and entertaining talk. Now we have a couple of minutes for some questions.

Speaker 2 [32:29]

Thank you very much for your talk. I'm working in the life sciences where most of us are self-taught and we're going through all the stumbling blocks that you kind of laid out. What are some advice you can give to university research labs that may not have the resources to bring in consultants and may have very small research groups that may be able to be more productive early on in their research and their design.

Speaker 1 [33:11]

If I understand your question correctly, the advice you would give to the researchers at university or the people at the company?

Speaker 2 [33:20]

at the university.

Speaker 1 [33:22]

To be more productive early on That's a that's a huge thing I mean that's difficult to answer depends on what what problems you're trying to tackle I think the biggest problem I'm overseeing is it looks like there's only a single weapon to solve all of these problems and that's deep learning these days And that's how I started when I try to learn the stuff Oh, yeah, deep learning will solve all of the problems in the future. Then I learned over there's this random forests, right? there's gradient boost. It's pretty cool in many situations. And then you learn, oh, there's modeling, there's Markov chains, there's simulations, there's k-means clustering, there's entropy stuff. There's so many tools. And the problem you see in practice is typically that people are too quick in making a decision on a specific tool, even on a conceptual level. So for me, the biggest challenge currently still is understanding okay, that's this kind of problem that requires a solution in this space. For instance, predictive maintenance, if you have a predictive maintenance problem, that's not necessarily random forest or a typical classification or regression problem. There's a special class of algorithms there that's called anomaly detection because the classes are highly unbalanced and you want to take a look at that, like one class classifier, stuff like that. But you first have to learn that when you enter the field. So the biggest obstacle is probably when you work in this field is that you start with a solution that doesn't really match. So I think my advice would be the big important advice is really try to understand how to map classes of problems to the right classes of solutions. And there's a learning curve associated to that. But once you get that, it really helps a lot. And sometimes that means, okay, that's that class of solutions. I have no clue about that. I have to ask a colleague about that. rather than diving into it and trying to solve it with the wrong tool. We still have some time for some questions.

Speaker 2 [35:27]

Again, thank you for your talk. I've got some questions about the usage of relational algebra in your...

Speaker 1 [35:32]

Algebra in your projects, do you really

Speaker 2 [35:33]

Do you really map in your second dimension or on your conceptual level your data access pipeline or whatever in relational algebra?

Speaker 1 [35:43]

or do you do it in some

Speaker 2 [35:44]

Do it in some other pseudocode.

Speaker 1 [35:46]

pseudocode because when

Speaker 2 [35:47]

When I had relational algebra in university, I was like, what is that?

Speaker 1 [35:52]

Yeah, I mean, not for each and every project. If you quickly understand, okay, here relational algebra doesn't help us. But, I mean, I can tell you a story from databases because that's repeating history over and over again. And I see that with younger PhD students, they do the same mistake. You also see that with papers that get submitted to database conferences. And that is the following. A new data model appears, be it objects or XML or text or key value-like data, time series, whatever. So what do people do? They reinvent a database system from scratch. They really, from scratch, do all the groundwork. Over time, they realize, oh, we also need this feature and that feature. We edit, edit, and edit. And if you look at that, you see, oh, they're reinventing stuff that already existed in relational database technology 30 years ago. They're now building it into their system. Eventually approaching what a relational database system would be doing except this data model and if they want to support a different data model But then when you look at that eventually some other guy appears and says yeah But if you change the relational data model a little bit in this part or extend it by this specific Operator have another relational algebra operator that does that a relational algebra can do the same thing and that happened many times It happened for XML where we had dozens of XML database systems 15 years ago they were all killed by relational systems. Same happened for RDF. Killed by relational systems. Doesn't make sense to build it from scratch. Same for text. The same, I'm sorry to say that, for any NoSQL system. NoSQL is technology from the 60s, basically. You shouldn't be using that. It's all included in the database system. And that's repeating history. Reinventing what has been done in the 80s and the 90s. Sorry for sounding arrogant. When I was young, I didn't believe that. The guys with the gray hair said, oh, we did all of that back in 1969. Yeah, I wasn't born there, right? What the heck? I don't care. I do my stuff. I might say we're kind of right if you look at this technology. So, yes, we use it heavily for this kind of stuff. Is there someone else? So, to also be arrogant for a minute, maybe the researcher's marketing sucks. So, everybody knows pandas, nobody knows relational algebra. Why is that? And maybe, well, to formulate Isaiah's question, do you know of any projects that are trying to give us relational algebra or database building blocks that we can use instead of reinventing everything from scratch? Yes, Spark does exactly that. Spark's relational algebra plus plus, full stop. Many projects did that 20, 30 years ago and didn't take off for whatever reasons at the time. So it has to also be the right time. I mean, the problem with pandas, and that's also what I learned when I taught it in the class this summer, was the interface at times is really weird. You have one square bracket and two square bracket things, and then selecting stuff and sometimes, hey, there's another bracket here or not, whatever. I'm still filling with that and ending up Googling and some stack overflow to the rescue. I mean, we just developed a small relational algebra for our students. We haven't put it open source. Maybe we should do that. But it's relatively easy. And once you have it, you can write it as a relational algebra and have this wrapper around Pandas. And as I said in the previous talk, in the keynote this morning, if Pandas had a decent relational algebra interface, all of the optimizations from databases would be so easy to apply. Basically, you would have a smooth transition from Pandas tables to query optimization and database systems. I believe they would really give a boost to Pandas and all the Python or PyData ecosystem. Someone should just do it. I don't have enough time to do that, but it's doable with little effort. That's all I'm saying. We still have time for one last question.

Speaker 2 [40:10]

I have a short question. I'm a physicist. I'm not a data scientist. So do you have a recommendation for a book about rationale?

Speaker 1 [40:21]

Rational Algebra I think you can put any introduction to databases any introduction to SQL there's a very old book by Johannes Kerke and Ragu Ramakrishna, 20 years old it has a cover with a cow on top, it's also called the cow book you can use that, there's a book from Stanford they're very old, but I mean if you had to learn that I would just, I mean I have videos in German on relational algebra but if you want to have it in English check out YouTube the videos it sounds there will be some proof it's so easy you learn that in two hours and then you're done ok, good, thank you there's still some questions left

Speaker 2 [41:20]

When talking about relational algebra and talking about your issues with pandas and its interface, it sounds to me like, are you mixing dimensions two and three in this discussion? What's the difference between the problems you're identifying in pandas? Can you hear me?

Speaker 1 [41:40]

Yeah, I can hear about that. Can you try to rephrase your question? I'm not really sure what the question is about.

Speaker 2 [41:41]

Yeah. Well, you were asked about pandas and how it relates to relational algebra and how could it be improved, and then you talked about square brackets. Isn't this dimension two versus dimension three discussion?

Speaker 1 [41:58]

discussion? Yes, yes. Relational algebra is dimension two. Pandas is dimension three. And the problem is that what is clear, how to solve it in dimension two, was not done as it should be done in dimension three in pandas in this case. And that's very confusing from my point of view when I try to teach relational algebra, which I do in my classes, and then I confront the students with pandas, and then we see it's not exactly how it should be, right? It would be much easier because that's basically the lingua franca of database people is relational algebra. And if you had that in Pandas, any database researcher would directly understand it and could apply his or her stuff to Pandas. And that's my whole point. And if you did that and really synchronized the two words, that would be a huge opportunity for synergy there. Yeah, then let's thank Jens again for this really nice and interactive talk. Thank you.