Postmodern Architecture: The Python Powered Modern Data Stack

Speaker 1 [00:02]

Thank you very much, and thank you everyone for coming along today. The last kind of real talk of the conference. What an amazing turnout. So my name is John. If you want to tweet at me throughout this talk, then it's just John underscore Sandal on Twitter. So I am CEO of Coefficient Coefficient. We're a data consultancy. You'll find out more about what we do, because I'm going to tell you about the kind of stuff that we do for our clients. So, I'm going to walk you through an introduction into the modern data stack, more importantly, the Python-powered modern data stack. I'm a Python-ister. And I'm going to do that by introducing you to my fictitious company, Megacorp, who sells software. So welcome to Megacorp. Imagine you are just starting a role as head of data at Megacorp. And you're being welcomed to the team. You've got a lovely-looking office there. And we're going to introduce you to some characters to help illustrate our journey today. So first of all, let's meet on our day one, the CTO of Megacorp. And the CTO says, we've got some data systems. We've got SQL Server. That's our main product database. We've got some products, some customers, some orders. There's a financial system that has all of the main sort of legally stuff in there, the bank accounts, the transactions. We also do lots of analysis in Excel. Different teams do their own thing. We've got some dashboards as well. They're actually just Excel, but we call them dashboards. There's some other systems around, but we don't really use them for analytics. It's something the developers do. Some of you might be feeling seen and heard at this point. Let's meet our software engineers. The awesome Pythonista in the room comes to all the PyDatas and the PyCons, knows their stuff. So what are they doing? Well, they decided they wanted a better analytics data warehouse. So they wrote some ETL pipelines using Python. They've got Pandas, SQL Alchemy to take stuff out of various different upstream data sources, load them into BigQuery. They've got a really nice ETL pipeline being orchestrated by open source tool, Apache Airflow. They've got a really good testing. They're using all of the best practices, all the things that you're learning about in the other talks of this conference. They've got some scheduling, some of it's with Airflow, but some of it is just stuff they do every Monday morning at 9 a.m. manually, they just run it on their laptop. And maybe there's some bits and pieces that run using Cron. But by and large, they're trying to implement software engineering best practices to their data pipeline. So they're applying dry programming principles, don't repeat yourself, where that stuff is happening in the SQL server layer. They're using store procedures. They're using views. And the other thing we've got here is there is a product analytics platform. I'm not going to name it. There's loads of them out there. I've just made one up called Events Logger. It logs events. That's what it does. There's loads of tools that do this kind of thing. So what's the kind of best practices that you may be aware of? And if you're not aware of them, really cool tools that I'm going to tell you about. So data validation within your pipeline. Again, this is reasonably good best practice. So, there's a really lovely tool called Pandira. There's also things like great expectations that do a similar thing. And it allows you to create a schema. If you're a Django developer, maybe you don't even need this because you're managing all of this in your Django layer. If you're not got Django in the mix, then I really love the combination of SQL Alchemy and Pandira. It's a little bit like Pydantic++++ for data pipeline validation. So, you can specify your schema. You can then run a data frame in pandas in memory through your schema and make sure that it validates on the types are correct, that you don't have string fields coming into your date times. And you can also even validate whether they've got nulls in them, if they're not meant to have, if they're an integer, that they're unique, or that they're between a certain value range, or if they're categorical data type, again, that they meet things that you've seen before and you don't have someone, again, maybe in the marketing team suddenly throwing in new field categories that maybe you need to map in. This is very common in the product analytics pipelines. People like to create just new events, and maybe you need to merge them into old events because someone's changed their casing. Other people don't care about us casing as much as us Pythonesters. So we can validate our data within our pandas layer, and Panda is a wonderful tool for doing that. What else is our software engineer doing? They're doing other little bits and pieces that require Python in the mix. This can't all just be done in SQL. So here's a nice little example of, but maybe they're doing some lookups via an API. They need to map all of their transactions from pounds and euros and other currencies into dollars so that it can be quickly analyzed in one query in that data warehouse. So maybe they're using APIs to do this, but maybe this is one of their Monday morning jobs. They've got to do using Apache Airflow. But they've got so many other things to do at the moment, and this kind of just works. As long as they don't go on holiday, it kind of just works. So let's see what else is happening in Megacorp. We've got the head of sales and the data scientist doing some cool stuff together. So let's see what's going on in the sales world. We've got a lead generation agency that they're purchasing leads from to sell their software to. And every Monday at 9 a.m., the lead generation agency uploads some CSV or maybe it's an Excel file into an FTP bucket. So more data coming into the organization. This is really, really useful, valuable data because it's what's powering the entire commercial operation of Megacorp. But there's a manual process around it. They take it out of the FTP bucket as an Excel file. They load it into their CRM tool. And they actually spoke to their awesome data scientist who thought, I can automate this a little bit for you. So they use Python and Pandas. They even use a bit of scikit-learn on the side to build a lead scoring algorithm so they can target the most effective leads first. But to be honest, she's not really sure how to evaluate that. She's evaluating it based on people they've spoken to in the past. They don't actually have the downstream data in their CRM tool or anywhere else easily accessible that's joined up across all of the different systems. So they might be speaking to a lead from one organization, but maybe someone else in that organization with a slight different email is signing up to the tool. They don't really tie any of this together. And also this seems to be wowing everyone at the organization. Wow, we've got AI in our lead scoring. Very cool. So this is also working operational. She just, again, it only comes in every Monday at 9 a.m. So she just runs this script every Monday at 10 a.m. and it pops using the nice API Python client that generic CRM has. She can just load that data straight into that tool so the sales team can quickly create their own automated target list. So we've got some cool stuff happening in sales, in engineering. What about finance? What are they up to? So in the world of finance, we've got SQL Server. That's your core database. Kind of that has to be accurate. The amount of revenue that's tracked there should really match the amount of revenue that's tracked in their generic accounting system. But the head of finance, maybe they have their own analyst because normal data analysts don't understand finance. There's all sorts of things and weird words like journals and revenue that people don't really understand sometimes, especially when you start talking about tax and gross and net. So they have their own analyst. They have their own manual processes. It's all Excel-based. It works. Don't come and mess with the financial systems. They're too valuable to disrupt. But, hey, we've got lots of manual reconciliation, and things mostly match. within our level of materiality, as long as it's within $10,000 or so. We don't really care if there's a slight error. It's finance. So then maybe it's day two. I've had my introductions, and I speak to the CEO. And the CEO is saying, how can we improve? I get this feeling that we're probably working in sort of circa 2018 style data infrastructure. What does 2023 hold for us? And so I walk the CEO through it. I say, you've got some really good skills in the team. You've got data engineers. You've got Python, clean tested code. Your data is being validated, at least in one tiny part of your pipeline where you've got a BigQuery data analytics system. You've got cloud scheduling, at least for some of your setup. You've got machine learning, optimization within your sales process. You're using Python and Pandas really effectively to do what they're good at. What could be improved? Well, at the moment, you're kind of struggling with your data engineer is actually just getting larger and larger and larger cloud servers to process their memory because, hey, why not just have a two terabyte RAM system on your BigQuery or on your Google Cloud Platform because it's kind of cheap, especially if you're just using it a few times a week. But really, this is not going to scale very well, especially if you really start flying and you've got terabytes of information flowing through your pipelines. So what is out there to help you move out of in-memory ETL jobs? You've also got a lot of tech debt. There's a lot of code maintenance. You've got a small team. You've got one data scientist and a few software engineers. And they've got a lot of things to do, let alone helping the rest of the business figure out what they can be doing to advance themselves, especially when some of the rest of the business doesn't really want to advance what they're doing in terms of data maturity. So you've got lots of organizational silos. You've got analysts in different teams. Some of them are going a bit rogue. You've got a data warehouse. They're not even using it. They're just writing SQL queries directly against your database. And they're bringing in all sorts of bad data. They're bringing all of your testing, QA teams, data, because they don't know what to filter out. So you've also, as a result, got lots of different dashboards showing different numbers. And when you say, how much revenue did we make this month? If you ask three different people, you're getting three different answers. Again, is anybody feeling seen? This is something I've seen quite a few times with our clients. So what am I going to do? I'm going to map out their systems. At least let's start figuring out where the problems are. We've got leads. We've got the core database product platform. We've got events, which is sort of going somewhere and not even being looked at by the product team. We've got financial data. We've got manual reports. We've got a data warehouse. We've got a CRM. Some of these are kind of siloed, and the SQL server does a little bit of talking to the CRM. It tries to tell it when someone's signed up for an account so that the sales team can reconcile and not sell to people who are already customers. But it's a little bit rough and ready. So what does better look like? Introducing the modern data stack. So, first of all, moving into the theory part of my presentation. Let's talk about ETL, ELT, ETL, TL, and a few other technical jargon that I found an absolute minefield when just trying to understand what the modern data stack is, because it's a very buzzword-y set of stuff. So, let's keep it simple. ETL, some of you will be quite familiar with this. You extract data from some system. You transform it, probably in memory, using Python and Pandas. you load it into some other system. So ephemeral in memory, that's your transform layer. That's the stuff that you do day in and day out. And then you load it into wherever it's going to live as a persistent storage. Then we heard people talking a lot about ELT, this sort of paradigm shift, where instead of doing your transformation layer in memory using Python or Pandas, you actually extract your data from source. You load it into your data lake, something like Databricks, or your data warehouse, something like Snowflake or BigQuery. This was covered very nicely the other day by, I think, Florian in the wall stack presentation. Data lake being basically throw your data into it, and it's kind of raw, but we can start doing joins on top of that, and the data lake, like Databricks, is moving kind of in the direction of data warehouse, where we can do transformations in that layer, too. If you've used things like Hive as well, with Hadoop, similar sort of concept. We just throw stuff into it, and then we start trying to join it together. Your data warehouse is something where, again, you can load stuff into it in a little bit more of a structured format, and you do your transformations in that layer as well. So typically, your data lakey warehouse or data lakehouses or whatever they're going to call them next year, these are your load transform single storage unit. You put data in there, you transform it using things like SQL views, materialised tables, UDFs. Again, we're getting into jargon, so let's cover that jargon. SQL views. This is a virtualised table. It's created by a query. It's not kind of real. It's there. It's read-only. You can't change it. It's a bit like a hologram. A materialized table. This is creating that physical copy of the result set of your query so it can quickly turn a view into a materialized table in many tools these days just by adding a config flag such as materialize. And these can be updated periodically or on demand. The upside of these is your downstream query is going to be faster if you materialize your tables. The downside is it's going to require more storage space and also who's updating it. And is that process running? Or is it something that someone has to run manually? Or is it potentially falling over? Who's going to get an alert if that falls over? This gets very complex very fast, which is why people kind of like the SQL views, because they just run in that moment. And then we also hear about things like UDFs, the user-defined functions. Think of it as just a function. But these are written usually in things like SQL or JavaScript, depending on your platform. So if it's BigQuery, then that has JavaScript UDFs. Snowflake, people like talking about it. And they're here because they also support Python, Java, and Scala in their UDFs. So here's a little example of a Snowflake Python user-defined function. It's defining the runtime version. It imports some packages. There's a little bit of XGBoost going on there. This isn't doing much useful stuff. It's just printing out some package versions just to show that it can. But these are the sort of things that you get in your data warehousing tool layers. So back to what we're talking about. ELT, lots of things we can do in that layer. But the modern data stack I like to think of as slightly different because there's an extra TL in there. So extract, load, transform using your transformation tool of choice. It could be Python. But I'm going to be talking to you about dbt today. And then we load it into its final resting place at the end of the day. So we start thinking about a raw data lakey dumping ground. Try and keep it relatively clean. Otherwise, it becomes a data swamp, I've heard it called. And then we actually clean it up. We transform. We take just what we need. We do our typecasting and our other transformations, and then we load it into a really clean, documented analytics warehouse. So here's your typical modern data stack. We've got various upstream sources. We use extraction tools, open source or SaaS. We load them into a raw data lake. We do some transformation. We generate a clean data warehouse. We then do cool stuff with it, reporting or data mining, or even reverse ETL. There's some of those little data bits that are really useful to have in those upstream operational data sources. So here's how this might look for Megacorp. We've got our various data sources on the left. We use some extraction tools. The relevant tool for you is going to be completely dependent on your needs, depending on your level of technical maturity and your level of funding. But there are open source tools. There are commercial tools that have different features and price points. And then there are tools that we could use for reporting in open source, like Dash, Metabase, both DBT, Aware. There's also the usual go-to commercial providers. You've got your data mining tools, which we talk about every day at PyDatas, and you've got your reverse ETLs. There isn't much open source in the world of reverse ETLs, so if you're thinking about your next open source project, maybe consider that space, because the paid-for platforms are very good and very expensive. We're talking about sort of $1,000 a month base pricing for some of these, although what do we use internally at Coefficient? We use Make, because it's like 10,000 operations for $10 a month. I am literally, like, my time is more expensive than $10 a month. I'm just going to go for the thing that serves my needs for cheap. So give that a go. It's previously IntegraMAT, and they've just rebranded and slashed their prices. So this is the sort of process we might be talking about. And it's a lot better, I think, than this process, because stuff is beginning to talk to each other. We can integrate across all of these different systems, across the CRM, across the financial system, make sure they're talking to each other, and that if we've got valuable information on our warehouse, that can be fed back into those upstream systems. So finally, showtime. What does this look like in practice? First of all, our head of sales, we want to automate extraction from our CRM system into BigQuery. I'm going to use the open source tool Airbyte. If you don't want to orchestrate this yourself, then you can pay them some amount of money. But to be honest, if you know Docker, you can just run Airbyte yourself and deploy that using your favorite cloud platform. So I can find my favorite CRM tool. I can see what it's got available for me. I can run this locally on my machine. It's just a Docker thing. And I can then go into that dockerized thing. Once it runs, it gives me a web UI. It's very beautiful for a piece of open source software. And I can set it up. And what will that look like once it all gets integrated? It looks like this. This is my Google BigQuery warehouse. It's got literally everything in my CRM has been neatly packaged and presented into small, raw staging tables that I can then see how they link together and so on. Just to be honest, this is actually using 5chan because 5chan also costs me about $5 a month. But it's a very similar output to what Airbyte would give you. So I can see all of my tables, I can see some of the information in those tables. My entire CRM instantly and perpetually synced into my data warehouse for about five minutes of my time and $5 a month, or even free if you can do some Docker deployments. What about the lead generation? Again, we've got FTP, connectors within Airbyte, Other open source tools worth mentioning, Meltano, Singer, Bonobo. You can run these and orchestrate them yourself, or you can pay Meltano a little bit of money to do it on their cloud. What can we do with my Xero for the finance team? So again, the financial extractors, they do exist in the open source side. So we've got Meltano's Xero extractor. As we'll see, there is a full-on GitHub repo for each individual extractor. So if it's not working for you, if something breaks because Xero broke their API, then you can open a ticket and go fix it yourself. It's just connecting to the API, and the core software is handling all of the upstream transformations and extractions. So when something changes in my upstream accounts, then that immediately gets synced into my downstream database. Right, next up, dbt. So we've got loads of raw data in our staging data warehouse. How do we then do that transformations? So you'll be used to using Python to do this. But dbt is a very nice open source tool. We can hook it up to our data warehouse using our profile. So this is a very quick sort of three-minute dbt 101. And this has been one of the greatest things that I've learned about and really learned myself in the last 12 months or so. So very excited to show it to you, because maybe it's useful for you as well. So we pointed this, in this case, to my BigQuery warehouse. And then let's have a look at my dbt project file. So we can see there's loads of confects. It's a bit like a pyproject.yaml. We can also see that there's going to be the downstream warehouse and models are going to be all set up here. And I'm even saying, for my zero tables, I want you to materialize the output as views, at least as a default. You can change this on a table-by-table layer, but that's my default. So let's go and have a look at my zero models. How's that going to look? So my zero models have a zero schema. And this schema basically just defines what tables are going to be used. There's like 300 tables in it. I don't need them all to solve a simple problem for my finance director, such as, are you underpaying or overpaying your VAT or your tax this year? So we choose the tables that I want to use. And then for each of these tables, I'm going to create a mini extraction layer. This is all about applying software engineering best practices into your data pipelines. You can do this using SQL. You can also use Python for this if you wish. And then what I'm going to be able to do is from those clean staging tables, I'm going to be able to bring them together into a single downstream analytics table, which brings them and applies all the joins and the mergers. How does that look once I'm done? Then I can see. I can run dbt run. dbt run is going to power all of those tables into my downstream data analytics warehouse. It's going to then materialize some of them. Some of them I only need as views in the moment. They just get used as staging posts to keep everything really clean and separated, trying to apply dry programming principles so that any piece of typecasting is done once and once only as early as possible. Any other transformation layers like converting pence into pounds or dates into UTC, for example, again, only happens once into the staging layer and then it's done by the time I bring into my little VAT analytics table which actually solves the problem for the finance team. Other nice features of dbt, it's got testing built directly into it. So you can specify in your dbt schema, you can say I would like you to check the following columns are unique, that this one has not got nulls in it, that these are unique together, there's a relationship between this one and this one, that it meets these pandera-like data quality validation And just like using PyTest, we can run dbt tests in our CI layer, and it's done for us. And then finally, a really nice thing I like about dbt is that if you add descriptions in your YAML files, it's going to generate some documentation for you. So if you're working in an organization where nobody's really talking to each other, and half the problem is because nobody knows what's in the data warehouse or what these column names is, or when you say net revenue, is that really net revenue? Is it gross revenue? Let me do some data lineage tracking to figure out what this revenue number actually means and where it came from. And there are really good tools that plug into dbt that will do that for you. Where did this column come from in my BI tool? You can trace it all the way back and see that actually that's not net revenue, that's gross revenue. Go and talk to the finance analyst and make sure that's correct. So this is your generated dbt documentation project, beautiful HTML website. We can also do machine learning. We can do Python. It doesn't have to be SQL-only, so this is dbt's latest 2022 update. Some very cool things. You can also materialize tables as incremental, so it can watch an upstream table, see if there's new rows, and then just load in the new rows into your downstream tables. So finally, to wrap up, the future. The CEO is going to say to me, what skills do we need in our team? Is it data visualization? Is it engineers? Is it analysts? Is it MLOps? Is it DevOps? What is it that we need? And the modern data tech answer is this new role, the analytics engineer. And to be honest, I'm still fighting off people saying that data science isn't a real thing. Even in 2023. I'm sure some of you have experienced that, too. Analytics engineers are going to have a rough time for the next decade. I do think that there's something in this. So they are someone who sits in between your technical team and your business team. They're much closer to the business problems. The kind of stuff that the finance team is saying, you know what, we just got our own analysts. They understand what we need. They're going to do this thing for us. and are less encumbered by the problems of orchestration and so on. Now, there is a relationship to the future of work here. We've not only got new roles appearing in tech and data, we've got new roles appearing in the business side. We just completed last year a online course for the Chartered Institute of Taxation in the UK. This course was creating an online diploma for tax technologists. And I think we're going to see this across every major industry. You're going to have legal people who are being told your job's going to get automated. They're not going to get automated. They're just going to learn new skills. They're going to get more techie. They're going to understand everything that we're talking about today because I've just written an entire diploma that teaches tax and accountants how to do this. So this is the exciting new world where we can show a slide like this to a load of accountants, which is, first of all, relevant to you. but it's also relevant to them because they're the ones that are going to have to start making these decisions or talking to the rest of the CTOs and their team who are going to be helping them to orchestrate the tools they need to do their job. So this is a slide from that used with permission from the CIoT, but also a very relevant slide to my talk today because it shows you that in every single stage of the modern data stack, there are open source and there are SaaS tools. What's relevant for you is going to be dependent on, do you know Docker? If you don't know how to Docker something, ask ChatGPT. I'm sure it will help you. Because if you can, you can really use those open source tools. So to wrap up with CEO, they're going to say, how does it help me? It's going to save you time and money for your organization. There's reduced cost and time to get up and running. You can sync an entire upstream data source into a load of nice, clean, relational tables in your data staging warehouse. And you can do that in about five minutes. This is much faster than the months it used to take me back in 2015, for example, doing this using Python alone and really struggling through the API documentation of some upstream data source. The speed of processing. If you're using some of these tools, you can process stuff at scale. This is what DBT and Python and Snowflake and all these other big data warehouses are designed to do. Terabyte, petabyte scale data processing using the same stack that works locally on your machine as well. The licensing, commercial, open source, cloud-sourced. Up to you. Go with the thing that meets your budget and that you can use. Finally, this is going to give you much better data. You can integrate data from all different sources. it's scale-ready, you're going to get much better data accuracy. If you really want to go crazy into the modern data stack, you can bring in additional, sometimes quite cheap tooling that'll tell you if you've got some product analytics pipeline that's fallen over because some developer forgot to include that call in your mobile app this month. So you will be able to get alerted, and you can fix these issues rapidly. You can also see if there's type or data quality issues happening the moment they happen. So I just want to finish off with my vision of the next stage of data maturity. Many of you will be working in organizations where data is probably critical, the one in the middle, because you're at a data conference or a Python conference. But as we move into data-driven, you can start moving towards self-serve analytics. Some of our best clients are sort of in the self-serve analytics space. They're able to answer what happened, but anyone in the organization is able to answer what happened last month or which of our A-B tests was the most performant when we're looking at the downstream conversion rates, not just the things easy to track, the things that are right to track. But next, I think I'm looking forward to using these tools to do more. It's not about just saving time or money or doing things faster. It's about the opportunity cost of what else can you do? Now you've got all this new time on your hands and everything is just being handled for you. We can do more predictive modeling. We can predict what's going to happen. Or we can start treating our data systems like I treat ChatGPT, which is, hey, what should I do in this scenario? How do I best negotiate with this contract? These are things that AI is really good for, and you can make it really bespoke for your organizations. Give your CEO tools where you can actually figure out, what should I do? Thank you very much for your time.

Speaker 2 [26:51]

Thank you so much for such a ground covering talk so many things to ask questions about We have quite some but we only have a little bit of time left. We might go a little bit over because the next thing coming up is cake and coffee, so First question for you who will maintain this whole system of ELTL who is going to be responsible?

Speaker 1 [27:18]

It's a really good question. So at the moment, you probably need a lot of different skills to work on, let alone maintain one of these systems. You need the Python. You need the DevOps. You need to have good understanding of the command line. You need to have good understanding of SQL. And you also need to understand all the nuances of your finance team's hacks in the way that they use their bookkeeping system. What this hopefully reveals is a new way of working where only one person in an organization, or in a large organization, only a small team, can handle all the DevOps. They handle the orchestration. they put the tools out there, and more people, especially if you're looking at your next five, ten hires, are working that analytics engineering layer. They're closer to the business because all they need to know is SQL and a bit of Python. If you need some data scientists helping them out with the machine learning side of things, great, but it means just a lot more people who can work at the SQL, Python business interface instead of all of the DevOps. There's less of that and more of the value drivers.

Speaker 2 [28:14]

Are your slides available somewhere?

Speaker 1 [28:16]

I will put them up on GitHub, and I will tweet that on my Twitter, so at John underscore Sandal, and you can find out where these slides are going to end up.

Speaker 2 [28:26]

Given our limited time, I'll skip the question if you're a fan of Brooklyn Nine-Nine or we can maybe cut this short.

Speaker 1 [28:32]

How did you guess?

Speaker 2 [28:36]

Is there a modern data science stack, a modern stack for MLOps?

Speaker 1 [28:43]

What I would love to share, my favorite resource I've learned in the last year in terms of the modern MLX stack is there's a really good free online course called fullstackdeeplearning.com. It covers a lot of these tools. It covers things like DVC and weights and biases and other sort of machine learning, monitoring, observability, tooling, a lot of it open source. It's completely free. You can pay if you want, like mentoring or tutorials or whatever, but it's completely free. And it's really, really high quality. So that's fullstackdeeplearning.com.

Speaker 2 [29:10]

Could you go into the details about Pandera versus Pydentic?

Speaker 1 [29:15]

Cool. I think Pandera is slightly more data-aware of the kind of things you want to be using. So Pydantic helps you to specify your types. It's type-aware in the Python typing sense. But Pandera has quite a lot of nice tooling for specifically saying things like, this is an age. We shouldn't have 200-year-olds in our data warehouse. Which, having worked a lot with the UK government departments over the last year, there's a lot of that. There's a lot of very, very, very dead people in core critical systems. So if that's you, check it out.

Speaker 2 [29:45]

The actual AI models would also be in the data warehouse. Will the data scientists themselves put them into the warehouse or how can they check if everything is fine?

Speaker 1 [29:54]

Good question. This is kind of what is comfortable for you to do. Do you want to deploy your AI models as just here is a model asset that is going to be put into S3? You're using DVC, weights and biases, these kind of things, and you've got your model registry, in which case you can put it from there within your data warehouse layer, within your DBT layer, for example, and then you can do your machine learning there. Or you could have an API if that's your thing. I think it's really up to you.

Speaker 2 [30:19]

Okay. And with that, I think we'll wrap it up. Thank you so much again. It was really interesting.