Langfuse, OpenLIT, and Phoenix: Observability for the GenAI Era

Speaker 1 [00:07]

Observability for the Gen AI era and let's start this session with the most fundamental question every session should answer. Why should I care? And the reason is pretty easy to understand. Managing Gen AI application is very hard. As some guy has learned the hard way. When ChatGPT launched there were answers like this one. So, what is the world record for crossing the English Channel in Tel Aviv on foot? Well, it's 12 hours and 10 minutes set by George Reif in 2016. Of course, that's clearly false. George Reif never set this record. Only one person walked on water, as far as I know, and he didn't do it for 12 hours and 10 minutes, and it was a much warmer place. Well, this guy is laughing for the mistakes of his competitor, but let's see how it goes for them. How many rocks should I eat? AI overview by Google, at least one small rock per day, according to a geologist from UC Berkeley. So of course, you have to trust them. But as an Italian, I have much more fundamental questions to ask, what if the cheese is not sticking to my pizza? Well, that's simple. Just add some glue. But be sure that it's not non-toxic glue. So at least you can survive this one to tell the tale. What about Jeff? Are you laughing, Jeff? Let's see what your AI assistant on Amazon can do. This is Rufus. It was on beta when it launched. And you could use it as a copilot if you were to run out of tokens for your course subscription. So this is laughable, of course. But remember, these are the best companies in the world. These are the best engineering teams in the world. So if they are not capable of shipping a product that works 100% of the time, what makes think that you can do it, bugs are inevitable. And therefore, what can I do? Well, the answer is pretty simple, just test in production. I mean, GenAI is based on language models, language models are stochastic in nature. There is no way you can test 100% of the path through your application before you ship it. Your users are going to find ways to trick your application into doing something that you didn't think about. And the only way is testing in production, shipping your application knowing that something bad will happen, preventing all the worst stuff from happening with proper gaveling, with proper testing beforehand, and then seeing what happens once the application is live. So you have basically two moments. Before you deploy, you can go with testing and evaluation, and these should take out 90% of the issues, most importantly, the most dangerous ones, the ones about discrimination, the ones about inappropriate content and so on and so forth. But then, and this should ensure that you are providing your users with real value in most of the situations, but then, after you deploy, you must have a way to understand what is happening in your application. And this is where observability and user feedback comes in. Observability means being able to reconstruct the internal system of your application without knowing exactly what is happening, only from the outside. How is this possible? Well, this is possible if your application emits signals, if it emits some hints about its internal state. And in traditional observability, these signals are spread into three main categories, metrics, traces, and logs. Metrics are quantitative data from your application that are emitted at a specific point in time. Latency is a very good metric because it's a quantitative attribute, it's a number, it It may be a number in seconds, which is emitted every time you interact with an LLM or every time your endpoint answers to a request. Matrix are usually useless unless you aggregate them. And here you see a very common example. You have the 50th, the 90th, and the 99th percentile of latency. That means what is the average latency, What is the mode of the latency? What is one of the worst-case scenarios? And what is the worst-case scenario in the worst 1% of the cases? Traces are the path followed by a request through your application. So a user is interacting with your application to do something. And remember, your application has not to be a chatbot. Even if you have some LLM-based processing in the back end hidden from the immediate interactions of your users, you should have this kind of observability in order to understand what's going on with the prompts that you're probably creating dynamically and with the answers that the LLM is providing to you. So a trace is the path of code execution throughout the lifecycle of a request. Maybe the first point of contact of your backend with a request is an authorization endpoint. Then you may have, as a service in your code or as a service in a distributed environment, another component that does the data fetching. Maybe you have another component that interacts with the LLM and so on and so forth. Tracing means you measure the time required to complete every round trip for every component in a hierarchical way. So A is the root span. A span is basically a subset of a trace. It's a unit of work, while the trace covers the entire lifecycle of the request. So A may be your entire request. B may be the, I don't know, the authorization unit of work. C may be a subcomponent of the authorization, and so on and so forth. Somewhere there, maybe in E, there is the LLM interaction. Logs, you probably know very well, are events happening in your application. Logs can be either unstructured or structured. When they're structured, you have a very cool dictionary with keys and values marking the attributes. When they are unstructured, usually you have just strings linked with some metadata, like the timestamp, the severity of the log, and maybe the line of code where it was emitted. Combining all these three categories has been a thing since some year. In 2019, OpenTelemetry was announced. In 2023, the last component of OpenTelemetry, the log, reaches stability for some languages. In Python, you have that traces and metrics are stable. Logs are unfortunately not stable yet in the OpenTelemetry ecosystem. But it is safe to say that OpenTelemetry has become the standard for observability. How does it work? Well, you have different servers, maybe different applications, maybe different microservices for the same application that send traces, logs, and metrics to a collector. There is an open source open telemetry collector free for everyone to use. The collector takes all the input data. It processes it. It filters out some content that you do not want to store. It can perform sampling and so on and so forth. It acts as middleware from your applications to a proper back end where you can store, visualize, and elaborate all the data. This back end can be any service that you want. It can be an open source one. It can be a proprietary one. It's for you to choose. But how does this whole stock that we have available fit into the GenAI ecosystem? Well, it's time to find out. And as showing is better than telling, let's head to some code, hoping that it will work. So we will see two tools today. First one is, yeah, that's already not good. First one is LangFuse, and the second one is Phoenix. I will zoom a little bit in a while. Both tools have multiple features. Today, we will only talk about features for observability. That is mainly trace logging and methods collection. We will not talk about data sets. We will not talk about prompt registry. We will not talk about evaluation, which are still features provided by these two tools. If anyone is curious, both tools have a cloud-based version. which has a free tire and then a paid plan. And both tools can be self-hosted on anyone's infrastructure. I will show the cloud-based version just for simplicity. Now, I have a very simple code here performing some lag. So we are starting from a data set of movies with their title, general release date, and a short description. We are embedding all the descriptions of the movies, and we are adding them to a table in Lansdb. Why using Lansdb? Well, because it's very easy to use, very easy to set up. It uses local storage, so no need for Docker containers or stuff like that. It's very simple, nice API for querying, nice API for creating data sets. All right. So as you can see, what I'm doing here is just creating a table, providing a schema. And your question may be, all right, where is embedding here? Lansdb can do embedding for you. So as long as your embedding function is supported by Lansdb, it can do it automatically on the database side without, basically, you having to pollute your code with this feature. Here we are using TextEmbeddingSmall by OpenAI. Of course, you need to provide an OpenAI key for that. I'm doing it with my environment variable. There is an OpenAI key here. And all right. And here, in the Pydantic schema of the table, we are specifying that the overview is the text source field for the embedding, and vector is the vector field for the same embedding. So from now on, every time we launch Semantic Search on this table, LensDB will know how to do it. All right. But let's assume this has been already done. Let's go for the rug code. So now what we need to do is to start with a query. I would like to watch a movie with dragons. And we would like, of course, to search relevant movies on our data set. Assume that this is not a script, but this is an API exposed by your application and communicating with your front end. You probably want to have a session ID and a user ID. Why a user ID? Of course, this should be not name and surname for obvious privacy concerns. This should be a unique identifier, which is unrecognizable with respect to the identity of the user. You want to have this in order to run some product analytics, in order to know which users are getting the most value out of your product, and in order to enable upselling and other actions like this. And you want a session ID in order to collect all the traces relative to this usage of the product. All right. So we are initializing the trace with the session ID and the user ID, which normally would come from your API. And then we go through the stages of the RAG pipeline. As the first step, we want to use hide, hypothetical document embedding. So we are not only embedding the query and compare the query against the documents that we have in our data set, but we will also ask an LLM to provide an imaginary answer to this query and embed the answer as well. Then we will perform two semantic searches in parallel, and we will combine the results. Hide query is implemented in this function, not very interesting. What is interesting is this atObserved decorator, this is how you tell LangFuse that it needs to trace the interaction within this function, right? So main was observed as well, so main is probably going to be our root span, the beginning of the trace, this is going to be a span which is a child of the root span from main. And here we have another function, buildHideQuery, where we have generateWithOpenAI, which is, again, observed, so that all the interactions with our loud language models are tracked by LangFuse. This client comes from here. And it's not imported from OpenAI, but it's imported from langfus.openai. So here, Langfus itself is injecting into the OpenAI SDK all the methods it needs in order to perform tracking. Very good. The last thing I want to mention is this update current observation with input query column query, Because by default, LangFuse tracks all the input arguments to the function with the atObserve decorator. And you do not want to track the client, because it has the API key within it. So if you track the client, you have your OpenAI key inside the traces. And that's not necessarily a good thing to have. Very good. So the same happens for the semantic queries run against Lansdb. The same happens for the re-ranking component that takes all the movies from the semantic search on the query and the semantic search on the hypothetical document and then combines them together, taking out the unnecessary results. And then the same happens from the generation part of the rug, where we take the context and we ask the model to answer the query with the context. As the final step, we track also an hypothetical customer feedback with a thumbs up score that was given to this generation. So if we check what happens, let's take this one, for example, we have this overall trace, which has three sub-traces, three spans, I should say. One, two, three. And we see that each of them has inside it a generation span, where we can see exactly the prompt that was given to the model and the answer of the model. Of course, some metrics are automatically tracked, such as the latency, the cost, the number of tokens, the model, the temperature, and so on and so forth. Can you do this with standard open telemetry? Yes. But just because you can, it doesn't mean you should, because you would have to rebuild from scratch many things that LangFuse gives you by default. So no need to do it. All right, what about doing the same thing with Phoenix? The code looks pretty much the same. What is different is that using Phoenix, Phoenix is natively built on top of OpenTelemetry. While LangFuse is compatible with OpenTelemetry, but it started with a different data model, a different protocol. So because it's built on top of OpenTelemetry, the instrumentation process is much easier with Phoenix. So all you have to do is to run this register function with auto instrument equals true. And all the calls to the LLMs within supported packages will be automatically tracked. So no need for atObserve, no need for importing the SDK from a different package. Here we are importing the SDK directly from the OpenAI client. All right, again, we need to give some information from the outside, user session, user ID. Usually this comes from your authentication and authorization part of the application. And then the structure is pretty much the same. The only difference is the set score, because unfortunately, Phoenix does not have yet a method in the SDK to track the user feedback. So you have to do it the old way by calling the APIs. What we get at the end is something like this. Again, we have a root span, which is called chain in the jargon of Phoenix, but it's the same thing. And we have here some more spans. We have the three that were already there with LangFuse, and we have two more. Because LangFuse does not instrument the embedding calls to OpenAI, while Phoenix does by default. So you also get the embedding calls. You have one for the query, and the other one, sure enough, is for the hypothetical document. And as with LangFuse, you get the tracking of all the text exchange from your application to the LLM with the indication of the tokens, the indication of the latency. You don't have the indication of the cost. You have to run the computation yourself. But still, once you have the number of tokens, it's pretty straightforward to do a multiplication. Now, if you would like to have a look to the code, of course, everything is on GitHub. This is the QR code to the repository. Feel free to scan it and to keep it, and feel free to share any feedback that you may have. And this is it. This was a very quick introduction to observability for the Gen AI era. I am Emanuele Fabiani. I'm a university professor in Milan. I teach AI in the Master of Science in Economics and Finance. I'm also the co-founder and the head of AI at Xtreme, where we build AI solutions for scale-ups and corporates. In case you're interested in having a chat, of course, feel free to reach out. This QR code is not phishing, it's going to point you to my LinkedIn profile.

Speaker 2 [22:28]

Thank you, Emanuele. It was a really nice mix with a fun start and then serious stuff. We have a few questions on Slido. So the first one, what is the simplest possible way to perform tracing in Python without using a web service?

Speaker 1 [22:46]

Without choosing it?

Speaker 2 [22:46]

Web service.

Speaker 1 [22:48]

All right. So it's basically using structured log and dumping your structured log to a local SQLite database. Then you can use DuckDB or Poll House to run some analytics on it. By the way, OpenTelemetry supports also local collectors. So you can set up the whole OpenTelemetry infrastructure with inside your Kubernetes cluster or whatever you want to use.

Speaker 2 [23:21]

Phoenix, because it's built on top of OpenTelemetry, supports most Gen AI agent frameworks like Langchain, Agno, etc., out of the box.

Speaker 1 [23:32]

so both Phoenix and Lanfuse are relatively new but not super new they have been around since a couple of years now so they both support basically all the frameworks that are out there do

Speaker 2 [23:50]

LensDB update embeddings in case of text update.

Speaker 1 [23:55]

not automatically meaning that I mean it really depends how you configure the the table so if you if you do it like I did with the embedding function defined in the schema of the table whenever you update the the source field in the table for any row then the vector field gets automatically updated if you don't do it which you can, I mean you can define a vector column without the indication of an embedding function, then you have to do it yourself. So it really depends on the configuration. With the configuration I use, everything is automatic, also the OpenAI builds. If you configure the table without the embedding function, you have to run the embedding function yourself.

Speaker 2 [24:51]

Can you combine the traces with OTL traces for a Java app?

Speaker 1 [24:57]

Can you combine?

Speaker 2 [24:59]

The traces with O-T-E-L traces.

Speaker 1 [25:03]

LORIS D' Not that I know, meaning that if you give, if you use LangFuse or Phoenix as your author collector, which you can do both with LangFuse and Phoenix, then everything you emit from your application ends up there. So if there are some application logs or some metrics that have nothing to do with LLMs, they get mixed with the LLM-based ones.

Speaker 2 [25:40]

Bye. Specific question, someone wanting to benefit from your expertise. Does tracing have to start at client level? We run a big API gateway that makes LLMs accessible and would like tracing without having our customers implement it.

Speaker 1 [26:00]

No, you don't have to, meaning that you can trace only the part of the request lifecycle that is under your control. The problem with that is you may miss some information. Because, I mean, depending on which part of the lifecycle you control, you may not be able to correlate different traces to make sure that they belong to the same session. You may lose the information of where a session starts and where a session ends unless the information is provided from the outside. So imagine you have an API gateway with an API key. you are like open AI, right? You receive two API calls, one after the other. How can you know from the same API key, so you know it's the same customer, how can you know if it is the same user performing both the requests or maybe different users performing different requests? If you only control that part of the request lifecycle and you have no information from the outside, that information is lost. Is it OK for you? That really depends on the use case. But I don't think you have any choice, honestly.

Speaker 2 [27:31]

I guess this is a satisfactory answer for the question. And the last questions combined on LogFire, challenging you, I guess. Any experience with the sponsor LogFire? Did you test out LogFire from Pydantic?

Speaker 1 [27:49]

No, I didn't. So the point is, when I decided which frameworks to bring to this talk, I only considered something that you can self-host on your infrastructure and that has open source code. And unfortunately, as far as I know, LogFire does not satisfy these two requirements. So I really didn't test it. And for our operations, we mainly use LangFuse for everything. So Phoenix was an experiment, and LogFire still has not been experimented by us.

Speaker 2 [28:23]

I mean that's a great reasoning anyways okay so we have one minute let's give a chance to the

Speaker 1 [28:24]

Yeah, I mean, that's

Speaker 2 [28:30]

last question why would I want to observe embedding calls is it just for latency purposes

Speaker 1 [28:36]

No, it's also because maybe you have some automatically generated text that you're embedding, like here for the hypothetical document. And if you do not observe it, you do not know what is the hypothetical document that you're embedding. You can say, yes, but here you had the output of your previous trace that you can use. Sure, it makes sense. But it's not always the case. So imagine there is a case where you are embedding some automatically generated text, and for some reason you do not have the complete text as the output of another trace. Then you want to know what the hell you are embedding and what you are running the semantic search with. So that's basically the reason. Again, all traces come with metrics. So you have the number of requests, the latency, the time to first token, the number of tokens, the cost, and so on. And this is another very good reason why you should probably observe everything you can. Then it's easy to filter out stuff that are irrelevant when you analyze the data.

Speaker 2 [29:48]

That will be a wrap for your presentation and for the day in this room. Thanks for sharing the last moments of the conference with us, and thanks for the mini-workshop with lots of valuable insights in observability in Gen AI. Thanks again for all of you.

Speaker 1 [30:09]

Thank you. This is the first time I bring this talk on the stage. Usually, I talk about much more research-related topics. So if you have any feedback, please feel free to reach out on LinkedIn and to send a message. It would be very, very helpful to me, also because I need to replicate at PyCon Italy in a month. Have a great day. Thank you.