ZODB: The Graph Database for PythonDevelopers

Speaker 1 [00:03]

Hello. Today I'll be speaking about ZODB, the graph database for Python developers. So most people, when they think of a graph database, they think of a property graph database, typically Neo4j, written in Java. And so here's their canonical example. You can see the nodes, you can see the edges. If you look on the right-hand side, you can see it's a sort of very natural English language representation of a property graph database. So I'm actually working with something called object graph databases where every object is a graph. And actually I took this image not from Neo4j, I took it from Graphigus. Graphigus implemented this on top of the ZODB. And so basically they said every node is a Python object, every edge is a Python object, and there are connections between them. So here we have the data model for GraphiGIS. At the top, you can see a root. You can see two containers, one container for all of the edges, one container for all of the nodes. And on the left-hand side, you can see a couple examples of the nodes. On the right-hand side, you can see a couple of examples of the edges. And typically, in the ZODB, you get an abstraction, you show the tree, and you don't show all the intersections, all the connections between the particular nodes. but they're there. And I say containers because these look like Python objects but actually they can have literally millions of items in a particular container so they look to you as a dictionary you access them by key but on the file system they're sorted as a B-tree and all of that is transparent to the developers it's really easy. So when would you use an object graph database instead of a property graph database? Well if you have a lot of variables so if you're doing a social network it's very natural to do it as a graph and people have a name and a birth date and an age and all kinds of other information attributes on a Python object. If you're doing a computer network, same thing. And of course, any of these networks get very large very quickly. So what you need to, it's really important, particularly in social networks, to have hierarchy. In computer networks, you can have a basic principle in human factors is no more than seven items in any category. So in computer networks, you can have, here are the computers in a room, on a floor, in a building, on a street, in a city. So hierarchy is very important. And particularly when you're dealing with taxonomy, hierarchy is absolutely critical. So I'm building a taxonomy of Python videos. This particular image has 40 nodes on it, but real-world images have a lot more. So here you have my taxonomy as a circular image, a circular web. The root is in the middle. 759 Python videos about another 100 categories and you can watch it display you can move it around you can zoom into it you can slide across a particular node so if you slide across the node see how fast it displays it's a single page web app I download the JSON, it responds instantly you can hover over the different nodes in the edge, in the middle and you can actually zoom into a particular node and you can click on that node. You see there's not only just a parent but there's this extra connection and that's the connection to the conference. And the point is I talk about trees but this is really a graph database. Okay, so how do we use this thing called ZODB? Well, it's incredibly easy to use. Here we have a simple tree leaf object with initialized method, set the name and a render method which returns the name. And so to create this as a zodb object, instead of subclassing off of object, we subclass off class persistence. So you can subclass off the persistent object class, you can subclass off the persistent container class, and there's a persistent set. Then you subclass off those, your objects become persistent, your graphs of objects become persistent, your applications become persistent. So it's really easy to use this. If you want to create a leaf object, you just say leaf is a tree leaf, and you say root.leaf is that leaf. But in real applications, you create lots of objects. So here we create two leaf objects, and we add them both to the root. So we have root of leaf1 is leaf1, and root of leaf2 is leaf2. So very simple application, three nodes. and of course there's some boilerplate that goes around that so you have to import some stuff the first time you have to create a database multiple ways of doing that you get a connection to the database you get a root object from the root object you navigate to the objects you want to change you make some changes you do object transaction.commit and everything works and all of this boilerplate I never even see it the frameworks do it for you I had to look this up for this talk so it's just python so say you want to change the name of a leaf you get the root object you ask it for leaf one and you give it a new title yellow leaf if you want to do a query you can do a very simple query over over the nodes in a tree in a in a in the tree for key item in root node items print the key in the item so really easy i'll just python if you want to delete an object, you get the root object, you say delete root of leaf1 transaction.commit. So it's really quite magical, right? I'm not messing with the database. I just take the root object, I pull it out, I navigate to the object I want, I do whatever transactions, whatever functionality I need to do, I do a transaction.commit, it all works. Really very magical, very easy. And it really is a graph database. So here I have two examples. I add leaf1 and leaf2 to the root. But I'm actually going to turn it into a graph, so I'm going to say leaf1.sibling is leaf2, and leaf2.sibling is leaf1. I do transaction or commit, and it all works. This is a really simple example with three nodes, but you can do a million nodes, a branch away across the tree, you do it, it all just works. It's magic. Okay, so we live in a data science world, and things like PyData, sorry, data frames, pandas, PySpark, these are really primarily focused on tables and relationships. Yes, pandas do have a parent pointer, but they don't have a child pointer, and they certainly don't have a collection of heterogeneous child pointers. They have a statically bound parent pointer. So I need to do some, so I'm doing some analysis. The first analysis I have to do is I have a tree, and I add more videos, and so I want to recalculate how many videos there are on any node of the tree. So if I go to a leaf and it's a video, well, that's one. And if it's a category, a container, the number of videos underneath it is just the sum of the children's. So really easy calculation to do. So I submitted this to St. Petersburg, Peter Pye Conference, and they had a big raging argument inside whether to accept me or not. And they came back and said, we need a more complicated calculation. It's not my fault it's so easy to do this stuff. I thought you wanted it to be easy, not complex. But anyhow, I thought about it for a day, and I thought about what can I do. I said, I got it, I got it. We'll do it all in SQL. Sometimes people laugh at that. So here's a more complex calculation. One of the things I have to do is calculate the best videos in any branch of the tree. And so that's based on upvotes versus total votes. And so for any node, you have to look at all the nodes, just child nodes, and then you have to sort them. And I don't want to do that for all the nodes and serve them to the client, so I just serve the data and do this on the client. I don't want to go into the JavaScript. It's how I do it. It's pretty easy. This is a Python conference. But the interesting thing is how I serve it. So what I do is if you have, if you mantle your data as hierarchy, you can serve the root node and then you can serve the children node. So I have a series of records, one after the other, where each parent is served before the child. And so the fancy tree libraries that I use, what they can do is as they're downloading it, as they get each record in, they can update, they can do a callback, they can update the record. So a user can come into it, he can look at the homepage, or even some child page, it can come up quickly, and while he's spending 5 seconds or 20 seconds reading it, you're downloading all the data in the background. So currently I don't do this, it loads quite quickly, but you can probably get up to 10,000 nodes, a very responsive system, and you can also do lazy loading if you want to do larger trees. So this is very simple, I don't show all the records, I don't show everything in each record, I just showed the ID of the object and the parent ID, and that's enough for the client to reconstruct the tree. And it's really easy to generate these templates. So I have a template. I use chameleon page templates. Any templating language is fine. And there's a generator. So you start on any node. You can say all tree objects, which gives you everything underneath that, and then you just iterate over it. So it's really easy to create, to serve this kind of JSON files. how do we store all of these well we store all of this with python pickles so what pickles do is they um take a graph of objects and they compress it down they're very python centric they compress it down there's no redundancy in it they take any arbitrary graph of python structures and do it really easy but we don't want to store the whole tree because the whole tree can be way too big you know hundreds of gigabytes you don't want to compress the whole tree as a single pickle, so what you do is you store it as multiple pickles. So here we have the file. We record a transaction is at the end of the file, and all the objects in a transaction are all recorded next to each other at the end of the file. And so say there's a link. I mean, there's always a link between the objects. The way that the links are stored is you store the class ID and you store the object ID. And so then when you read it in then when you read it in the next time say you want to read in this second object what you do is you read in the second object and it has a referral to some other object and so the referral you call it a ghost object right because you don't want to read the whole thing and you don't want to read in the data you only want to read the data in when it's needed so it's got a referral to a class and you just leave you've got the class information you've got an object in memory and you leave the information on disk until it's needed when it's accessed if you access it and then it pulls it in. So you end up with a bunch of different transactions written to the end of the file. And you can't see this item number three. But here you can see a particular item. There are going to be multiple versions in every transaction. And so this is really good for any kind of content editing system. So Plone is a content management system. It's based on this. You keep previous versions of the content. And so you can go back when you're editing. It's really nice to be able to go back to a previous version. When I edit with Emacs, or if you do Vim or VI or something like that, you get the current version and the previous version, but you don't get any other versions. You can save it in GitHub, but it's only every now and then when you do a major transaction, whereas here you get a whole series of minor transactions. It's just brilliant when you're developing something. You can store it in a relational database. You have a table, which has three different columns. One is the object ID, the other is the version number of that object, and the third is the pickle for that object. and then the relational databases perform a bit better than the file storage. Of course, you have to scale these things up, so you have multiple clients, so you have a load balancer, and then these three items, basically in a row, the three items are application servers. So the request comes into the application server, it queries the Xeo database server where the data is actually stored. So the application servers keep a cache of the object. If that object gets updated, they update the Xeo server. The Xeo server invalidates the other caches. So one of the things I don't have to worry about is I don't worry about cache invalidation. Very nice. There's actually a whole bunch of things I don't worry about. I don't worry about, I tossed out the database, right? I just have a file that I store. So I don't have a database. I don't have to have a database administrator. I don't have to know how to be a database administrator. Okay, that's another simplification. I don't have a database schema. I don't have an object relational mapper. What else do we have here? Oh, I don't have referential integrity problems. I have automatic garbage collection, so I don't have to worry about that. And I don't have any manual reads and writes. So just huge simplifications from using a graph database. How do you actually use it in production? Well, you can use Flask. That has a very primitive API, so we're not going to talk about Pyramid. That's the mainstream one that everyone uses, much more advanced. And then what I'm using is Chromlech. So Suhail Shalfu took 10 years. He cleaned up all of Zope. He separated it out from the ZODB, so he actually uses it with a relational database. And then I released on GitHub, Python, Lynx, ZODB demo, where I connected back into the ZODB. So the guy's just brilliant. Okay, so how do you use these things? You have to traverse to an object. So you start at the root object. This works really great for a web request. So slash takes you to the root object. Slash libraries takes you to the one to the right. slash libraries slash software slash libraries takes you down two to the right, slash software slash libraries slash database libraries takes you down three. So you can navigate. The URL just makes it really easy to navigate to a particular object. But I'm doing a taxonomy, and as I add things, I change the structure and I move things around. So the URLs would change all the time, so that's not good. So I do canonical URLs. Every object in this model has a unique name, and then you can navigate it either by traversal or just from the root, it's got an index into the object. So there you have an example where you have two different ways of accessing an object. It's a graph. And in reality, it's really much worse than that because every object has a reference to its conference and every conference has a reference to all of the objects. And in fact, what people are asking me for now is for every object to have a reference to its speaker and every speaker, because like Beasley they really like, They want to have a list of all the talks by a particular speaker, so it's getting really complex. And so that gets a real problem when you go to delete, when you make a change. So here we have the management view. This is what the developers use. So for every node in the tree, you can see all the different objects in it, and you can rename it, or you can cut that object, you can cut that branch, you can copy that object, you can paste that branch, you can delete that object. But the delete is quite difficult, because some objects just have one parent. Some objects have a parent and a canonical URL, so that's two different references. Some have three into the conference, or maybe even four into the person, so you have to manage all that. So one of the things that Chromlech does so brilliantly is it gives me these adapters. So I can have this simple interface that just assumes there's an object with a delete method, and then it has an adapter. So say I take one of these objects. I haven't done this before, so let's give it a shot. Okay, and say I want to delete this, right? It goes ahead and it deletes all of the rest. Oh, shit, that was my best talk. Oh, damn. That was really stupid. See, you should always practice ahead of time. But, let's see if you were paying attention. Subversion database, so here are the transactions. You can see there's the link transaction, I just have to undo that transaction. it's all done and I can go back page page the talk is there again so all as well so it's really nice to have a versioned database so you can fix mistakes like that okay

Speaker 2 [16:52]

I'll be the last to say.

Speaker 1 [16:53]

Okay, so people ask, how does this compare to a relational database? So you understand a relational database well. You understand the idea of a graph of objects. So the big difference is that all the objects are written to the end of the file. So if you just have a single object being updated in a transaction, a relational database, but say you're updating 10 objects in a transaction, then the ZODB will look faster. And of course, different things are better at different things. So in the ZODB, if you want to iterate over all the objects in a node, it's really easy to do that for item in node print item. But if you want to do that in a relational database, you'd have to do a join across however many different tables it is. But some things are easier in a relational database. So if you have a person, you have the age, and give me somebody who's between 21 and 35, that's really easy to do in a relational database. But in the ZODB, you have to create your own index. It's not that hard to do. You can look it up on the web how to do it. So let me give you one more concept here. So let me give you another example. JavaScript. So I have a tree of JavaScript objects I actually edit on top of the ZODB. Most people edit on top of the file system. Maybe often they have these really long files. For complexity, you really want to have only like seven functions in any particular node of the tree. And so let's take a look at this. Here's the breadcrumb functions. So it's very natural to compare editing on the ZODB to editing on the file system. But in a file system, you have directories and folders, and directories just have folders of subdirectories or files. Whereas here, you can have additional instance variables. So we can see the breadcrumbs across the top, very much like the file name. But here you have a title, and so you can write notes about the different version and stuff. And here's an ACE editor. I just skipped the ACE editor part. But the interesting thing is you can have a test URL. So when I'm developing, when I make a change, I just go down and hit save and test. Yeah. No idea what this is. Okay. Anyhow, it worked earlier, but anyway. Okay. So a bunch of questions people always ask, how do you store this stuff? You can store it in file storage. If it's an image, you can store it in blob storage. If you're doing the Xeo client server stuff, then you can store it in Xeo on the server, and you can actually use these tools to cache on the client. Relational storage, Oracle, MySQL, or PostgreSQL. If you're using PostgreSQL, then you can use the indexes of the PostgreSQL, very high performance to index it. Who's using it? A lot of people in the Pyramid community, not quite clear who is and who isn't. All of the Plone users, so just in the government of Brazil, there are 100 different websites, the president's office, and parliament, lots of things like that. Speed, you get thousands of transactions a second. Reliability, sorry, scalability, they went up to hundreds of gigabytes of main memory and terabytes of blob storage. If you need more than that, there's a related product called Neo. That's not Neo4j, that's Neo. And they do 80 terabytes in production, 160 terabytes in test for really large databases. Number of objects. This is 2 to the 64th. Security. The connections between Xeo clients and Xeo servers are encrypted, so you can run them across the internet if you want to. And there's a really fun product called ZeroDB. What they did is they did all the encryption and decryption in the JavaScript client so you can host in the cloud, and the cloud servers know literally nothing about you. They even encrypt the indexes. And it's an excellent example about it's all Python, so it's really easy for them to make these changes. If you want to do chat, so it's not like a real-time server. If you want to do something like chat or real-time application, I recommend MQTT. MQTT also does a tree of objects, so it's a published, subscribed tree of subjects, so it's really easy to connect it into the ZODB. Okay, so I invite you, I have some business cards, if you could pass them out, stickers. There's a little folder, if you're interested in staying in touch, you can just check that you're interested in the ZODB stuff. I hope, I guess I have time for questions. Go ahead.

Speaker 3 [22:11]

Thank you, Christopher, for your talk. Yeah, questions?

Speaker 2 [22:19]

Thanks, Christopher, for promoting ZDB. ZDB has been around for about 20 years, almost, and it's a good start for hierarchical data, but you promote ZDB for graph databases, which is complete nonsense for several reasons. Because a graph database is more than storing data, you need great capabilities, like performing all algorithms that need on graphs, for example, like calculating your connected neighbors and so on. And there's no support in ZDB. So in 2018, with better solutions and building solutions for graph database like ErangoDB, which is a decent graph database, where you have basically documents, can build your graph on top of the documents, query them with an arbitrary query language and have all the support for graph queries and so on. But using ZDB in 2018 for graphs makes no sense for several reasons.

Speaker 1 [23:29]

So let me make sure I understand you. So there's a problem, you're using Python instead of a query language, which is one of your concerns. The other concerns?

Speaker 2 [23:36]

The other concern is basically that the ZDB was never built for graphs. It doesn't scale in a reasonable way for graphs. It's okay for hierarchical data like we use in ZMS systems like Plone and so on. The ZDB was an excellent solution for that. But in particular, using the pickled storage, using pickles, is not compatible with, let's say, other systems. So when you build...

Speaker 1 [24:01]

Is not compatible with which systems?

Speaker 2 [24:02]

Yeah, for example, when you build large systems, you don't only want to access your data from Python. You have always components like JavaScript or other languages, and it's impossible to access Python pickles from JavaScript. So you need something more neutral, and using a new database or the SSL ErangoDB is a much better solution than using ZDB, which just serves you for some particular use cases.

Speaker 1 [24:37]

So, yeah, so it's really easy. It takes me a couple lines of code to serve it as a JSON file. I can understand in really large organizations where you have a little bit of this and a little bit of that, you know, you don't want to be focused. But, you know, as a standalone developer, to just do the whole thing in Python, productivity is great. I did this whole stuff myself. Okay. And scalability, you know, hundreds of gigabytes.

Speaker 2 [25:03]

Yeah, but in the context of this CM system, but not as a craft database. Yeah, I don't know. You're making claims.

Speaker 1 [25:13]

Let's talk more about this afterwards.

Speaker 2 [25:14]

Let's talk more about that.

Speaker 1 [25:15]

I'd love to get this feedback. It's not clear I'm catching what the problem is.

Speaker 3 [25:18]

All right, do we have some other questions?

Speaker 1 [25:21]

is there any notion of replication in the odb yes so there's a soap replication server and so what it does is it's remember how it writes the transactions at the end of the file so it passes those over to the soap replication server so you can get free open source is a replication server does great job for you I should add that slide good question

Speaker 2 [25:54]

Can you say something about storing large binary objects like whatever one gigabyte of binary into this? Is it efficient somehow?

Speaker 1 [26:04]

Yeah, so it stores something called blobs, and blobs are a file on the file system where you can store blobs on like S3 and Amazon. And so if you store blobs on S3 and Amazon, what it does is it stores it to a local file, and then after the transaction is committed, it moves it out to the ZODB. So all that happens very quickly. So your storage of blobs is as efficient as your storage of file system. I guess you have a tree there's a tree of blobs and it has some way of figuring out where it goes in the tree so yeah, that works quite well

Speaker 3 [26:39]

We still have some time for a question. If not, I want to make sure that everybody knows that we are still looking for volunteers who would volunteer for a session chair, for the registration desk, and for the video things. Yeah. So let's give, again, a warm applause for Christopher for his nice talk.