Multi-tenant Conversational Analytics

Speaker 1 [00:07]

First of all, I think thanks a lot for joining today. I hope you're enjoying the conference as much as I am. First time I'm here actually at PyCon and PyData in Germany. Did, let's say, witness other PyData conference elsewhere, but I have to say in terms of venue, in terms of presentations, it's been very nice so far. Speaking about, let's say, speaking at other PyData conferences, I think in the past I've mainly talked about more infrastructure-related topics, either more towards data engineering, building data platforms, building ML platforms. But today I want to spend a bit of time on a topic that I've worked on mostly for the past half year called multi-tenant conversational analytics. We'll dive into it, what it means. But I think what's different in this chat compared to chats I gave in the past is really the end-to-end, let's say, look at it. Not just from an infrastructure perspective, we'll definitely dive into that as well, but it's also about customer facing, like front-end-wise. How do you, what do you show to customers? How do you show it? But also from an AI perspective, how can you make sure that customers are actually seeing the right thing? We'll dive into that a bit more. Very briefly, just a bit of background on myself. If you're curious, this is my personal website, you can see a bit of the stuff that I've been doing so far. But I guess to keep it short, in terms of my experience, mostly worked at, and already has been set at, enterprise companies or scale-up startup companies operating in the fintech space. So my background is mostly within financial companies and, let's say, startup versus enterprise perspectives. And what do I do? I've mainly been busy with data, so data engineering, ML engineering, back-end, I write blogs, I think it was also mentioned, the Substack link will be later. And I love speaking, so I love presenting at conferences, sharing what I do. Let's go a bit into what is conversational analytics. So, I think probably most of you know that when you interact with the well-known chatbots of this time, what you can do is you can, for example, upload your data. Let's say you have like an Excel file or a CSV file with your data in it. Of course, it will be shared with, let's say, with the chatbot, and it will be doing things, let's say maybe transformations with it, it will read it. But what you can do with it is you can chat with your data. For example, by uploading something and then just asking questions, like maybe do some simple transformations, maybe do a bit of visualization. But it's one of the first entry points to conversational analytics, just being able to, from a file that you have, being able to chat with your data, and the chatbot actually calling maybe some Python tools, maybe some JavaScript tools in the background, and it can, let's say, visualize and transform your data. Next up, I think, is one level deeper in the conversational analytics space, and these developments we've been seeing quite recently, I think mostly over the past, I would say, two or one year, has been around, but what, let's say, if your data lives somewhere else? What if your data lives in your data warehouse? What if your data lives maybe in a database only? How can you then talk to your data? And this is some example. So an open source tool, Vana AI, for example, facilitates for you to connect to a lot of databases, a lot of data warehouses, and it will facilitate an interface which you can use to chat with your data. You can ask specific questions around your data, and it will answer it. It will display your data in the chatbot interface, and it will also visualize it for you. Another example is Databricks Genie. And a lot of other data warehouse solutions have similar things that they currently are building or already offering to chat more with your data. What is new and what is really recent I think been coming up is the generative BI space, so generative business intelligence, and this is really the next step when it comes to conversational analytics. So not only having just a chatbot in terms of interacting with your data in more natural language, it can also generate full dashboards, generate filters on dashboards. It can generate all types of visualizations and do it in, let's say, a very generic way. Really, let's say, I think new in this space. But what if I tell you that, let's say, from an engineering perspective, all of this is very far from trivial. You want to do it yourself. And this is a bit, I guess, of my experience. So what if a stakeholder comes to you? Let's say you're engineering responsibility at a company. And they say, I want to have a chat interface to interact with my data, because that's the hype, right? That's how people envision things to look like. You don't need UI anymore, right? You just need to have a chat interface. And based on that chat interface, you can do everything. Well, I can tell you it's far from simple to be doing something like this. And it's becoming simpler by a lot of, let's say, good open source tools which facilitate this. But still, putting it all together is far from trivial. And let me show you a bit on, so let me first introduce maybe a little bit of context into the use case that I'm presenting on today. So I want to introduce you very shortly to Palm, for the company that I work for, and the context, let's say, of what we are building, let's say, conversational analytics for. And Palm is, let's say, a cash management tool, and cash management means managing bank accounts, so having visibility on your bank accounts, especially for very big companies that have, let's say, that operate in a lot of countries. It's very important to know, let's say, what is the balance on the bank account, what are the transactions that are coming in, what type of transactions are coming in. and one crucial element to this is the analytics part so you can imagine that if you have a lot of this all over the world you want to know exactly what's going on and you want let's say from our side we want customers to be able to interact with with that data it's very very quickly like I think you can see like all that say maybe different things that we focus on cash management bank account management but the reporting piece is always let's say very much central. And that's also why conversational analytics was one of the things that we're like, if we can do that well, then it unlocks a lot of potential in terms of not having to build a very customized dashboard, not having to build a lot of self-service analytics for our customers, but being able for them to interact more in their own language with our tool. What were some of the requirements that we were focusing on when we wanted to explore the conversational analytics space? Well, first of all, we wanted to have fast iterations. We wanted to be able to change, let's say, the AI. We wanted to change the AI, let's say, prompts, the AI system behind it. We wanted to be able to change the front end in styling. We wanted to change a lot of things very rapidly and experimented with that a lot. Another angle I think that we were focusing on really was there are let's say for example well-known BI tools which are trying to do this already have something like this but we want it to be mostly based on open source or manage tools that we already have let's say that we already are capable of using for example in our own cloud environment to reduce dependency but also to make sure that we're in full control and then on the other hand I think talking about AI we wanted to experiment more with other agentic capabilities. So conversation analytics is one of them. Meaning that you can talk to your own data, visualize it, etc. But there are a lot of other things that we wanted to be able to experiment with in a chat interface and then easily iterate. And that also brings me to the multi-tenant part of this. So what what makes this difficult is that it's not only or it's not only internally facing, because of course we can also internally use something like this, but But the multi-tenancy part makes it really tricky. You're customer facing, and therefore expectations are a lot higher, but also the context that is there is a lot different for different companies. I'll show you a bit of how it looks like today. So if, let's say, a customer would go into our platform, it would look like something like this. I'll try to make it a little bit bigger. They would come into, let's say, the chat interface, and they'll be like, all right, what can I do? So I'll say, speaking at icon.a, and the audience would like to know what you are capable of. So what we will look at today here is this part. So data queries and visualizations. The chatbot can do more than that. And that's what we're experimenting with in the agentic capabilities. But what I want to focus on is the data queries part. So bear with me. This has, let's say, this specific version has no context about anything. So what I'm doing here, let's say, is rolling the dice a little bit. Because of course, let's say, there's always this unpredictable element when it comes to LLM and AI systems. But let's say now I want to just show me my cash flows for March 2025. All right, let's look at it a bit. What does it do? So it briefly described what it did. So it looked up all details. It gives, let's say, a preview of data that a customer can browse through. By the way, this is more data than we usually show. But it gives a bit of context on the different aspects of data. I cannot scroll through it, by the way. You can browse through it and see it a bit. And it also gives a graph. But is this really what I'm looking for? No, I don't think so. So can you aggregate this by day? I'll say chronologically just to be sure. And let's see what it does. All right, this looks a little bit better. So looked up total money, transaction date, total amount, line chart. You get a sense. And then can you filter only for USD transactions? So it's a bit of this, right? You can chat with it. You can see previews. It's straightforward enough. You can do stuff. It's simple. But you probably want to know, well, what's going on here? It's a pretty simple version. It can do more than this. But I just wanted to highlight the simple use cases that it can do. But even the simple parts are far from trivial if you want to do it well in a multi-tenant setup. So let's go back a bit into this part. Multi-tenancy, what are kind of the requirements that you need to have if you want to do this well in a multi-tenant setup? Well, first of all, it depends a bit on the data setup that you have. But let's say you have stakeholders or customers that you work with. You can spin up different instances of a database. You can segregate it. You can have different tables. But an easier way, in my opinion, to do it is to have row level security. So just have an identifier or multiple identifiers in a table which you can use to identify specific customers to have them only being able to query that data. Another element to it is the schema that you have. So when it comes to the data structure, it's very important that it's not a very complex data structure. And of course, you can imagine that if you are trying to service this to stakeholders and you already have an existing data structure which has a lot of tables and a lot of columns, this becomes a lot more trickier. You can still do it, but it becomes trickier. I'll get to it, how you can deal with that a bit better. But having one schema across, let's say, everything, so, for example, I showed you cash flows. You can think of, in our case, of bank balances, bank account balances. Bank accounts, if you have, like, specific structures in place, generic schema makes it a lot easier already mentioned row level security I think data security is another part I think is very very essential for it to not let's say have other let's say people being able to interact with data that they should not be able to interact with as well as expectations already shared I think expectations are a lot higher if you deal with external let's say stakeholders but maybe even internal stakeholders and the part of pre-training having context around what has been asked, but also preparing beforehand, even before someone interacts with a chatbot interface, already knowing what type of questions they would ask. And then, of course, the interface itself, if you're hosting something just for yourself or internally, it's a lot different than you have, let's say, if you do it more externally facing. You need to think about how to embed it, how to authenticate, you need to think about how how to deal with history, how you do retention and all of that. What else makes this difficult? You can probably already see that there's a lot of different components when it comes to building such an interface. Just the amount of different data sources that you might have. say a database, a data warehouse, there's a lot of different varieties that you might need to take into account. On the other hand, you might already have some semantic layer, maybe you use Looker and you already have LookML set up, or maybe you use Tableau and you already have the semantic layer already set up for you, which you can use to also power conversational analytics. But in the easiest way, let's say, I would say is to focus on SQL, because it's generally supported by all the different solutions in one way or the other. Knowledge base, I mentioned around training, around examples, let's say positive, negative examples perhaps, you need to be able to store that in a certain format. Conversational analytics is nothing without LLMs and potentially also an AI framework behind it. Interface, I mentioned a bit around being able to iterate quick. There's a lot of different types of interfaces that you can use. But I think the two main ones I think that are right now are very interesting for a lot of use cases that are more externally facing, either the Vercel agents, the AI SDK that they have to build chatbots, or Chainlit, which is an open source Python, fully Python-based interface that you can use to build chatbots, even customize them. Or something that you can do locally, open web UIs and alternative as well. And then, of course, visualization. So I showed you Chatbot that can visualize, that can plot graphs, and maybe different types of plots. Depends on which, let's say, of these interfaces that you pick. But you also need to pick how do you visualize, and which, let's say, visualizations can it create. You can already see, yeah, well, there's a lot of different options, like different, let's say, multi-tenant setup, different tooling setups. And I want to guide you a little bit through how to deal with this a bit, let's say. And I think it all boils down to what is the situation that you have and how can you make that work for you. I'll explain a bit on how we did it on our end and why we made certain decisions when it came to this setup. In terms of tooling, so one of the givens that we had is we already were using Google Cloud Platform. And I think that makes a strong case to use certain tools in a managed version that makes life easier. If you use AWS, if you use Azure, it's probably similar. You use similar tools that already are in that, let's say, that already are available to you, but they will have probably a slightly different name. So in terms of vector store, in terms of examples, how you can save your knowledge base, I think Postgres makes life a lot easier if you are already using it. So in terms of vectors, in terms of saving, let's say, the questions and answers that are given in the chatbot interface, the embeddings, if you are using something like Postgres already, makes life easy to just use it. But we'll see later that it, of course, comes with different challenges. So you don't need to have a specific vector database like Weaviate or Pinecone. It's, let's say, Postgres can deal with it a lot, let's say, as well. It's open source as well. It comes in almost any of the cloud environments. AI, so I mentioned Google Cloud. Gemini was an easy pick for us. We experimented with OpenAI, with Cloud. They all had, let's say, some things that they were better at generating, some things they were not. But Gemini, looking at Flash, but also at 2.5 Pro, out of the box performs quite well when it comes to generating SQL, semantic layer codes. It's a visualization code as well. It performs quite well, and therefore was our choice. And in terms of AI framework, we were a little bit on the fence. I think we tried out a lot of different AI frameworks, but we figured out that, for example, with Langchain and Langraph, but also with Krew AI, the boilerplating that we, let's say, the boilerplating that it facilitated came at a certain expense that we wanted to simplify it as much as possible. And therefore, with Agno, with a pretty minimal setup there, trying to simplify it a lot. And perhaps even we wouldn't use an AI framework at all if we had, let's say, maybe more time to build something a bit more specific. Because we had a sense that we don't really need it. But yeah. Interface, I shared a bit around, let's say, the interface. It's a Python interface. It helps us iterate much quicker. Everything in Python, and we just iterate and go with it. And then deployment, GitHub Actions and Cloud Run for having a scalable deployment. Architecture-wise, so let's go through it a bit. I think I already explained it. So you saw the client. You saw the web browser. You want to authenticate a user. You would do that essentially, let's say, in a typical web interface already. And you would call your back end, and it would generate a token. So the auth token, you can also use, for example, to authenticate directly to the Chainlit interface for it to know this is this user, and this user should be authenticated against, for example, your own database. And it would say, all good. Or it would say, no, this user does not have, let's say, permission to do it, and it would not work. Another thing is chat history. For example, that you can also save in the database. And let's say Chainlit out of the box also supports doing that. The AI agent. So in terms of context, of course, you have, let's say, the questions being asked, the context of the chat history. You would give it as context to the AI agent. And the AI agent would, let's say, from there, in a certain, let's say, format, try to see which state it's in. And then from there, try to determine what to do. This is all in Cloud Run, by the way. I already said memory and examples. I think it would get, let's say, context from the database in vectors. Gemini, from the AI agent, it would be able to call Gemini at any step to generate code, to generate SQL, et cetera. And of course, it has tooling calls where it can call BigQuery, in this case, to retrieve data from our data warehouse, which has a standardized format with row-level security per customer, and it can visualize data by generating Gemini, let's say, Gemini generating Python code, which generate plotting. Very quickly, challenges. So I think high database loads is one of the things that we faced. I think you can think of different setups to try and facilitate that more. I think session management was one of the difficulties that we had. Cloud Run has a very scalable deployment. But if you do it all in Python and you have it in memory, then if a user suddenly goes to a different instance, then that can bring a lot of problems. And therefore, you need to really think about state management and session affinity. So having sessions go to the same instance and save everything in the database. And then agentic setup, I think already mentioned. We did not go for a free format agentic setup. We went more for a decision tree-based one, where we really go, these are the different states. And you always can traverse the decision tree to go back to the initial state. But in principle, if someone is already asking for visualizations, they typically won't really ask me for other things directly. So you probably want to iterate on the visualization. And therefore, we went a bit for a decision tree setup there, to not have it fully free format, but a bit fixed. All right. A lot of things. I, let's say, am a blog writer on Substack. So if you're interested, I'll also be sharing all of the learnings here in a blog on Substack with also more in-depth details on how to potentially do this yourself and also sharing a lot of other things on there. What is the future? very briefly. So I think with more standardization in API layers, MCPs, Model Context Protocol, you might have heard of it, coming more into play, it will become easier for SaaS platforms like Palm to be able to universally embed into AI agents. And then lastly, I think RenAI, generative BI tool, embeddable, embedded analytics tool combined, I think, are very interesting and are the future, I think, of analytics that is embedded to customers, really around embedding and generating dashboards from natural language. That's it.

Speaker 2 [24:26]

Thank you so much, Mr. Rodel. So we have a couple of questions, and we'd like you to answer them for us. So first of all, have you experienced any stability or security issues with long chain?

Speaker 1 [24:42]

Yeah, so one of the initial setups that we had was with Langchain. I don't think it was necessarily about security, I think, that we were, let's say, facing problems with, but it was more around the bloatedness of code that you had to write to make it work. I think when we switched to Agno, a lot of stuff made more sense. And also in terms of requirement dependencies, dependencies. I think there's a lot of requirement dependencies if you use something like LanChain or LanGraph. It still can work, I think, in a lot of cases, but, like, there's a lot of stuff you might not need, and that makes it a bit tricky to really minimize the setup that you need with that.

Speaker 2 [25:27]

Thank you. We have a second question. The data model exposed to the agent, did you use any data Modeling approach to facilitate generating the sql like data Vol, star schema, obt, et cetera?

Speaker 1 [25:42]

Yeah, so in terms of the data model, we tried to keep it, let's say, relatively simple and having a few big tables, so wide tables, which you generally also see in more like BI use cases. And that worked pretty well for us. And it also facilitated that you, whenever you generate, for example, an SQL, you can give it full context on the schema. While if you would do that, for example, maybe with star scheme with a lot of, let's say, different tables, you need to have a lot more context. And you might only be able to grab maybe a question and then from your vector store which tables are relevant for this. So, it's really tricky, but a few tables, very wide, works, in our case, worked a lot better than smaller tables and more of them.

Speaker 2 [26:41]

Thank you. Our next question comes, how do you evaluate quality of LLM responses beyond Synthetical corrections of queries and code?

Speaker 1 [26:53]

Yeah, so I think what we try to do is two things. So I didn't show it in the chat interface, but we generally have thumbs down or thumbs up buttons where you also can write in your feedback. That's the human part of it. We also, let's say when we ever make a change, we explicitly have in our chat interface buttons that users can use to give, let's say, feedback on what has been given. And what we also do is at every step, we try to get a bit of a sense of the sentiment. So if the sentiment, let's say, is negative, we will save, let's say, that data point and the full trace of the conversation history to have it as an example, positive as well. And based on that, we can use the knowledge base that we obtain to add in, and it's mostly positive examples that we do, but we can add in the positive examples to try and have, let's say, better performance. But evaluation is a tricky thing that we're definitely, let's say, we're struggling with. But it's all about, I think, also communication to the people using it. And having open conversation is about it.

Speaker 2 [28:06]

Thank you. Our next question is, what database schemas work best? Should you convert data to a star schema?

Speaker 1 [28:15]

I think that's probably the first question that I answered. So let's say not a lot of tables, but wide tables.

Speaker 2 [28:27]

The next question is was the aggregation you showed on the graph done on the fly or Was it pre calculated and the chat only showed a different angle of the data?

Speaker 1 [28:37]

So, when you interact with the chat interface, it will try to determine if the data that it had queried in the previous step is already sufficient to do the visualization. If it isn't, what it will then do is it will re-query it, otherwise it will try to use that same data to create the visualization. So I think in terms of data frames, you can think of, let's say, a data frame being queried, And then you can do, let's say, your transformation logic. It can do the transformation logic in Python and then generate the visualization. So it depends. It will try to see if it's not. If it needs to re-query, it will re-query.

Speaker 2 [29:19]

Thank you so much, Mr. Rodel. This was our session today. So please, a big round of applause for Mr. Rodel.