Destructive Testing: 10 Practical Ways to Expose Hidden Application Risks

Speaker 1 [01:15]

And how we can make the life of the developers very hard. So, because of that, it's named Destructive Testing in my current project. We are testing real deployed software, and this breaks nearly every time we try something new. And so, I get this title that if I do a new test, then everything is broken. And I try to extract my knowledge of that to 10 things I can show you. As first, I want to say, what is a QA doing? So we all know this test pyramid, and somebody already does that. And so as I understand QA, mostly it's done like the QA is doing only the end-to-end tests. This is different from project to project, but in general, they really test the final thing that is really deployed, and then doing their complex test cases and different approaches of what they are doing. And all the other things, they are also somehow quality, but normally a software tester is not really testing unit tests or writing them. So as a developer, you're responsible for your unit tests, for your integration tests, and how you really specify these differences. And for the end-to-end tests, if you have a very big project. So I'm in big projects with very critical loads and very critical software. And there you have to be sure that this is working in a deployed version. So not it's running on my PC and then it should work or it's working in the pipeline, but really deployed with all services with full capacity. And for today, I bring you an example of what it could look like. So an end-to-end test. It's quite interesting what an end-to-end test is. So for me, it's more or less a fully deployed system with all external resources, if possible. And because I'm a technical guy without UI. So I'm testing mostly APIs, but the most concepts also are for the presentation layer. But you have to test every time your API, because on this level, you make it public to the Internet or something like that. So this is the point where attacks or the real requests are going against and not against your RP normally. So as example, imagine we have a web shop or we want to build a new web shop because all other web shops are not so good. And if we need a web shop, we normally start with something like an RP gateway if we are more or less in the microservice wave and cloud architecture things. So we have a gateway and every request is coming against the RP gateway. And then we have normalist services. Like our webshop service that is giving us a product and so on. And you have also a database that lives all the data. And then to a normal webshop, you have more things. Like a payment service. You want to be able to let people pay you for your things that they buy. You, of course, need a database for that. This is clear. But you also need a paying service. And now it's getting interesting. So how can we test that a paying service is working with my application? Because this is a very critical part. And if this is not working, this connection, then we have a problem that nothing is working. Nobody can order something. And for that, as a developer perspective, you normally mock this somehow. Okay, we have a paper, we have a special behavior, and so on. but as QA we want to have a real test environment that is fully separated from our productive environment that you can also break it also with pain and with force and for example PayPal is giving us exactly this possibility PayPal has a sandbox for us available we can get a PayPal account and spin up a full PayPal environment especially for us that we can configure. And here it's getting interesting because we're using Paypal maybe in the newest version and we can test our stuff against the newest version of Paypal. And here we are getting the complexity because we not only test our service but we test a full end-to-end customer experience. This is a big difference. And this could break because of Paypal just changes in the RP. It's not the responsibility of the developer to know that. But if it's not working, it's not working. And so, additionally to our normal services, we have something like a shipping service where the products are stored and how they are delivered to the customer, and for the communication between the services, we have queues normally or direct calls, but in Microsoft we have normally queues, mostly also a persistent version that we don't lose packages between the communication. And you see, this is a very complex structure. And this is also hard to test that this is working in general. So you can test the webshop. You can test the webshop in combination with the database. But in a fully scaled system, the webshop service has normally runs on Kubernetes. Then you have different pods in different places in the world. And so they have traffic between what happens if a data center is going down or is unreachable and so on. So you have a lot of problems that you get by scale and by complexity with external resources. And especially for this thing. So we assume as QA that every service is fully tested from the developers. And they give us good services that should work together normally. And now I show you how as QA, how I do that. And we have QAs that are on the good side that say, oh, I will now do a test and do a load test and maybe be sure that your system is working. Or the kind of my things is I do a load test at night and we will see what happens the next day. And the developers will learn how a QA thinks and how we can break systems. And I will show you this is very easy. At first, I normally test RPs, so I build in my test project, so normally I have a new Python project, and then I build a client to all our services that I have a class to communicate the service that my test cases looks easy. So in general, I build one client for the communication and creating a user, deleting and the user and so on, you can do some Python things with data classes and types or not, but this is more or less an abstraction layer that a QA can understand what the RP is doing. And then we are going to a long journey. So who is happy if the happy path is working? So who's happy if the application does what it does? So this is normally the first thing we test as QA or as tester. We are happy if this is working. And I understand all the developers, if you're able to, if you have a web shop and you can create an account, look in, order something, do it in your basket, get the payment done. If this workflow is really working, then the biggest things are mostly solved. And so for us, QA is only the start of our journey. So what we are testing, we are testing the functional correctness. We take test cases that are mostly defined from a product owner or from ourselves, depends a little bit on the project structure, to get the basics running. I'm not talking about very complex. I go to this setting in this deep nested thing and change something, but really the basics. So for the web shop, maybe around 50 to 100 test cases, I order something to the basket, I remove it, and so on. And so with that, this is our start. And if this is working, we are at the good point for an application. So we are talking about applications that are very big and complex. So my current project, we have 300 developers and a few QAs, and so you see this is very big. And this is the first step. So heavy path testing, everybody does. And then we are starting to add a little bit, be more creative of how to misuse an RP. No, we start with encode. So you see a typical test case, also like in unit tests for range act assert, you set up somehow your environment using a little bit of fixtures of PyTest. And then you create a user, get the resource back, and you get the user and just validate that this is working. So this is an example of how we are testing or how I test an application. Then, a second step, we start to get creative. We want to see all negative paths. Because here we have regular problems that if you have a complex service structure, that if one service raises an error, that we never see this error. Or a system crashes, or not reachable, and so on. Or data that you could put into that is never really good. So we try to force the limits of the RP, for example, using an empty string, misusing fields. But we also every time expect that this fails in a specified behavior so we're getting normally it's not a 500 that you want to have or you want to have a 500 as feedback if something is going wrong but normally you want to get a feedback later for your rp that this is an error like the input data is not correct or something like that so um this this is the second step so um with that we test somehow if we can crash the application, and this is surprisingly often possible if you're just at a none somewhere or at additional fields that the system completely crashes or something like that. And if you're a little bit creative of how you design your test cases, then this is a very easy way to do that. Here for our example, so we create a user, like before. We create a user just with a name and an email, and we're using an invalid mail, so we have not an add symbol, and so it's invalid mail, and in the test case we've PyTest. We just use PyTest races. So maybe it's a good point when we execute tests, just to talk about that here. I normally split my test cases in somehow an acceptance test, a very fast test execution setup that you can run after deployment. So half an hour like that from execution time. and then you have a test that took much more time, so hours or a weekend or something like that. And so that you have one time the situation that you can run this test suite after each deployment and one time that you can run them over a weekend or overnight or something like that with more complexity. Yeah, so this is the negative path. So now we are, like the developers, we have the positive, the negatives. And this gives us the first time, this is working, this gives us the first time the possibility to stimulate the application in a more sensitive way, like authorization. This is every time a problem. So, oh yeah, the image RAI generated, I hope you like them. You shall not pass. Who is aware of this sentence? You shall not pass. Oh, great. My wife said, nobody knows that. I said, maybe. You shall not pass. So if you have an RP, normally we have authorization and authentication. So authentication is who is this user, and authorization is what is the user allowed to do. And if we are in the authorization, then we have now our client. We have our test cases with positive and negative things. And now we can go through each resource. We are creating and look, okay, who is able to do this, an administrator, a user, and so on. So we are getting very big test cases, or not big, a lot of combinations with less test data. So all test cases have so much differences of what you can do, and can I get this, can I create here something, can I delete? So you get a mass of things, but normally it's a good idea to do that because developers normally forget this somehow. And unauthorized accesses are a daily business, if you are in the QA business. And so authorization tests look more like that. So I have a test for deactivate a user. So we're doing the same thing. We create a user, and then we try to do a delete. And with a client, so normally we inject over fixtures, our tokens, and each token has different possibilities with different rights. Depends on how you authenticate. But if you are on token level, then we just use fixtures to inject this in our test cases. and then they can use different clients with different class instances one time to delete it and assert that this is failing with an error and one time and delete that this is somehow deleted. And so the test cases looks like every time like that. So you can imagine testing is very easy from the technical point of view because all the test cases looks like that. I have the easiest code you have ever seen. so they are quite easy and not much complexity but you have every time the problems with requirements so is this user able to read this resource is this also for this field included so you have to really think about all the resources and so you hit every time the requirements so should he see this should he not see this and this is getting big discussions inside the requirements a product owner or developer who is responsible for that. And so we regularly have this problem that we don't really know against which expectation we test because we have our expectations and the product owner and this normally never matches. So as QA, we are discussing a lot of things. And so we are now on level three, I think. And until then, until this level three, This is the funny part, and now it's getting interesting. Load tests. It's every time the same in the project. It's working somehow, basically, and the most features are working. And then we try how fast it is, and then it's too slow. It's every time the same. Who knows that? Whoever had a too slow application? Nobody. Sure. Me also. Never. Yeah, so we evaluate performance and Sqa this test cases are very easy to implement because we just use our client we have built and Do a lot of parallel requests and measure the times of them but for the developers is a real pain if you have an Uncomplex architecture with multiple services and then you say okay I do a request against this service and this is slow in in one of 1,000 cases. And then they had a lot of fun to debug. So SQA, it's very easy for us to do that. So it's not a difference if I do 1,000 parallel requests or 100,000 parallel requests. But for the system, the background, this is a huge difference. And so also debugging for developers is very hard. And so SQA, we must be sure that generally it should work. So we have tested everything. We saw that this is working if we are alone. And now we are starting with performance. With performance, you have different definitions of how many requests per seconds you can process or how many requests in parallel you can do and how long they need or which loads under which conditions. So you have a lot of different definitions. So you have to be also sure that you talk about the same number. For example, if you're a ticket shop and then you have a Rammstein concert, for example, and then you have no load, and then the ticket shop is open for the tickets and they have two minutes of traffic and then nothing is working again and nothing is happening again. And exactly this leads to a lot of problems. Also with microservices, if you have an auto-scaling system behind, then it's scaling up, but maybe it's too slow scaling up. The question is if your application must be able to scale fastly up or not. And this is every time then the discussion, okay, so if we have a system, one hour is nothing running on that and then I start with a low test, then it's slow but if I do the load test after the load test then it's fast and then it's a discussion is this an expected behavior because we want to to to scale our resources as much as possible because of prices or if this is a not expected behavior and so we have every time this discussions of all the performance topics so this is every time a problem and also how much load application should hold. This is in the most problems not defined. So you say, okay, we want to handle a lot of requests, but then you ask, okay, how many? Because for the developers it's different if you have 100 requests in parallel or 1 million requests in parallel or per second. And so this is all the testing things. The test cases, you can do it a little bit in different ways. You can use special tools for that. This is for very complex load tests with different ramp up phase and ramp down phase and so on. Or how I do it mostly is with PyTest async. I just do an async call and just loop against, like here, just loop against 1,000 requests and set a max duration. So this is SQA. It's very easy for test case. So a functional test extended to a load test is very fast implemented. You can also do it different. So it's not about the code. It's more about the idea behind that. So with load, you can break applications, and this is very easy. Normally, if you are in the Kubernetes world, you have your pods, and if your pods, they have a health check normally that you know that a pod is going down, and if you make too much requests against the pod, then you have the problem that the health check is fading and then Kubernetes is killing your pod. And then you lose a pod that is doing requests and then the other pods are more overloaded and then Kubernetes is killing more or less all your pods. And then you have an application that is not working and it's not like it's limiting the requests, but the default Kubernetes behavior is that it's killing them and then it's getting worse and worse if you increase the load a little bit. So you are going up with your load and working, working, working, and if you go more up, then it's breaking and it's fully unavailable. And so the question is, what is the expected behavior, and what should we do in 4-hour accept, or for my acceptance test, I do normally like that. So this request took 5 seconds here, I do annotations, and you can see in the code, but this is only Python magic, that your 5 seconds is limit, And then you see if it's working to create this much users at once. Okay. Has anybody of you an application that will hold all the requirements until now? Yeah. So the most applications did not hold all the test cases until now. and now we are getting very, very it's getting more dirty. So we're doing this more or less a positive low test. Now we're doing this as negative low test. I used the short word chaos for that. And we try to break this application in a more sensitive way. So we said before, okay, we know all the inputs and we know that if I have an invalid input, that this raises an error. But we have a lot of combinations for that. But we are a code as SQA, and so we just implement methods that is doing for us all the random data generation normally. So we have a lot of things that you can add or remove, wrong chars, encoding problems, and so on. And so we are able to do that. And if we have the API specification, we can also generate it to the limits automatically. So if you spec or something like that, you see, okay, the string max length is like that and then you can extract this and Use exactly the max strength and the min strength and all the limits. So if you do it in a smart way within fast RP You can generate your test data easily without knowing what you're really doing And because we are sure that this should more or less work or fail every time we can create very easy test cases Then the random data is one thing. So with very big strings very short strings numbers, negative numbers, empty lists, very long lists, wrong data types, and so on. And this is normally breaking a lot. So if you run this for one hour, for example, then you execute half a million executions, then you will find something, and then you have 500 or something like that. But this is really for big projects, it's not for a five-person team. Then a very interesting thing is injections in general. So we have SQL injections. Who knows what SQL injections are? Oh, nice. SQL injections are more or less, I would say, the first big problem in the IT. And it's still there. And you have SQL injections as one topic. And you have all the different databases. I have also the same problems. Like if you have a graph database or what you use, they have normally also the same problems with injections. So we also try to do in our strings and our numbers some of our injections. And the newest thing, if you have an AI workflow inside your application, is to try prompt injections. Who knows what a prompt injection is? Oh, the most of them. For the rest, a large language model could not separate between input data and the command. and we can try to input a command into the LMM to do something different that it should not do. And with that, you can break nearly each LMM model with a behavior that is not really expected and it's hard to protect. So SQL injection is, I would say, a solved problem if the developers know what they do. But, yeah. And what I do with this test is I build a data generator for all my things, or also tools. There are sometimes also tools for that. I just added here a little bit of code, how it looks like, and then you generate your test data, a little bit focused on injections and all the things that could go wrong, also like Linux command executions or Python commands and so on here. And with that, you will find something normally. And you have some tools for that. you can reuse it or you do it with a random choice in Python. And if you're a little bit creative and with the support of AI, it's very easy to get you a long list of different data. Okay, so, and these are critical tests. So, with that, you can break applications or inject problems. And then, now we have the load thing. And what is a normal developer doing if you have load problems? A normal developer is doing something like scaling up. So we make our database bigger, or we start to cache things, we add more pods and so on, or we deploy over multiple instances, multiple data centers, and then we run directly into the next problem, and this is concurrency. What is concurrency? Concurrency is when I execute a command or two commands that are in indirect directions. For example, I will show you the code for delete a user and change the password of a user. So, if I have an RP with two endpoints, delete user and change password, I want to execute two commands in the same time. One time that deletes the user and one time that changes the password. And with that, you normally get a lot of mess inside your executions because they run in parallel. If you have a load balancer with different services, with a shared database, over different instances, then you have really a problem to get this running. And the concept behind this is normally locking in databases. So you have the lock state, and the lock state has a problem that you have pessimistic and optimistic locking and so on, that you cannot change a row or a column or what you are really locking. And this is going against the performance. So, if you improve your application to be very fast, that normally you forget that you have to manage also this kind of request, and so this is a little bit against each other because logging is taking a lot of resources and so on. And this is the funny thing of QA, because if you make one thing better as a developer, we find another place where we can find interesting things for you. So we do parallel requests that are not working to each other, and so we find race conditions and data inconsistencies, and depending on the locking behavior, how it looks like. And the funny thing is, as QA, we already have this code. We have the code tested for creating a user here, and for delete the user and for change the password and here change user to admin. We have this code already with our client and we just use these three commands and do it in a way to execute them in parallel. So we have a nice test case with very less work but with a lot of problems for the developers. Who thinks that your application is able to hold this kind of test? Two. So, you see, And you must exactly understand as developer how locking is working, how locking is working about shared databases or shared services, or with transactions. And as QA you don't have to need an idea of that, you just do that and then it's funny for the developers. And because we already have the test, we are getting to more funny things we can do. I name it the day after tomorrow, the 60 hour test. We start on Friday and look at the results on Monday. And we just do a constant load of requests to your server to find problems. Like, if you do performance increasement, you start with caching. And if you start caching, then caching needs memory. And if you do long tests, then normally your cache gets, it depends on how you develop it and where it's placed and so on, it's getting full. And so then you run out of memory, and then your application crashes again or is getting restarted. And with this test, you can do it one times per week without new deployments normally. And then you just run it. And this detects so much problems. Locks are getting full. You have disks on servers that are just full because of no lock rotation and no lock deletion. And all these problems are coming here from. And the funny thing is QA is it's very easy for us. Because of that, we are so less QAs and so many developers because we can show problems that are easy to detect but hard to fix, mostly. Because of that, you don't need so many QAs, and they don't need really to know what they're doing. I was a developer before. And then, backup and recovery tests. This is my personally favorite one for microservices, Because if you look at this one, you have a service, you have a database. You have a second service of a database. You have a third service of a database. You have an external service. And you have a queue. And sometimes you're getting a message, maybe to the shipping service, and this one is processing this and saving it to the database. Now my question is, how can you back up this? And this is not so easy. This is a technically complex problem. backup a system with multiple databases and with a queue between, with packages and processes. What we can do is we can stop all services and then do a backup, and this is working. But in general, this is a hard problem. The question is, why do we need backups, especially if we have messages and so on, and how we are doing it. This is not clear how a backup will work in a microservice architecture, and it's a very big pain for the developers. But, as QA, you say, okay, you need a good process for disaster recovery. Show me your code to backup and show me your code to recover this. And I will use these two commands, backup and recover, and then we will see if it's working. And what we are doing with backup and recovery tests is really doing backups and recoveries, including adding data before and getting in a state where the queue is full and there are right processes, depends on the definition, and really try to run data, back up it, recover it, and validate if the data is correct. That is still there. Which application from you is able to do that? It depends a little bit on the requirements and so on. Clear. But just an idea of what we can do. And then... Oh, we have so many funny things. I name it welcome at microservices dependency failure tests here we have the problems with microservices that everything can go wrong everywhere the startup order of the services could be different, a service could be unavailable maybe because it was killing itself because the request runs too long so we as QA we must test or we should test how the microservice architecture behaves with critical situations. And for that, it's different every project, but you have a few ideas what you can do. You can maybe shut down or kill and port or application, it depends on what you deploy, or restart it, and then you must make sure that data is somehow consistent before and after, that it's not doing crazy things. And you can also do maybe fireball rules, so that you just add dynamically a fireball rule inside your test case that blocks the traffic. And then you can recalibrate traffic and look how the application behaves. And if a request, like a create user, is going through and is processed correctly. If you are on this point with your QA guys, then you have a very good application. So normally, this is never working. And then the last one. I named it Every Test is Summoned Full. The resources are limited. You cannot scale up unlimited. Your disks are summoned full. Your memory is summoned full. Everything could be full summoned. Your fiber communication is maybe full with load. And how the service behaves here, it's getting more and more complicated to test for us. So here you have to know what you're doing and understand all the details, It's how to get a system in a state with a nearly full disk. You need also support of QAs. But this is a typical case. You have a not rotating logging entry, and then it's getting full, and then the service is shutting down, and the rest is also failing. And then you have a system. You have no log message because the disk is full, and you have no idea what happens and what the data consistency is. In test cases, this is funny. In production systems, if you are, for example, a shopping system for Rammstein, then you have a problem because you don't know which transaction belong to whom and in which state you are. So if you have something like that, you need something like that. Or buy big disks. And a lot of RAM. This is expensive. And in summary, we have somehow product quality and we have different points of what we can test against product quality. And if you pass all All these test cases, these are 10 test cases, and then you have a very good application. And what I want from you is to take one of the test cases and think if this is relevant for you and try to understand how you can improve your application for that.

Speaker 2 [36:18]

Thank you so much for that, Pascal. That leaves us with extra time, actually, for Q&A. That's always fun. The first one is, do you integrate malicious testing in your stack, like the QA trying to hack the app?

Speaker 1 [36:34]

The QA hack of the app? It depends on what is hacking, because hacking is mostly things I showed. So we can test against things we know. So our identification is going regularly wrong. But what we can hard test is test about millennium bugs and so on. So hacking application is a different world, I would say, because there are so many things. I try to do my best with all my knowledge I have to break it or to come inside that but there are so many nasty tricks that they can do to come into that so mmm most you need a special security guy for that

Speaker 2 [37:18]

Good second question. How do you keep the costs down for intensive tests? For example, I have an app running in GCP and I'm bombarding it with my with requests in my tests My cluster scales to meet demand and costs goes up

Speaker 1 [37:37]

No, this kind of test, especially low test, costs money. This is absolutely clear. But you also have to say, if this problem happens in production, the pain in the product is normally higher. But if you have an application that crashes in production and you have only $100 of loss, then you don't have to do something like that. This is only getting interesting if you have millions of customers or big applications, and then the costs are not a problem. If you look at the cost of the compute, then maybe the test case is not relevant for you. So test only what you really need because we also cost money and this costs everything. This costs money. This is only we don't write productive code. This that's it.

Speaker 2 [38:23]

Good. Aren't the example concurrency tests non-deterministic? If they are, is that a problem?

Speaker 1 [38:32]

For my surprise, they are mostly deterministic. This is surprisingly so. If you use one, I'm also surprised. But if you use one machine from where you're requesting, then your requests are really technically queued on the network level and so on. And then the requests are coming in the right order. And the systems of locking are so fast that they are in the right position that you normally have no running over them. No, the first one is faster than the other one. This is normally the case. If it's not the case, this could happen and depends a little bit on the requirements. Then you have to do your test case. For example, if I have three requests that are in different directions, then only one will pass and two will fail. So this is sometimes also the case. But surprisingly, not so often. So this is working very consistent, mostly.

Speaker 2 [39:34]

Here's a question that I think you will answer with oh that depends but maybe you can give like a concrete rule of thumb or so. The question is how large are common QA testing setups and how do you decide on scoping what to test when to stop implementing more tests?

Speaker 1 [39:53]

In the projects I am, we have 1 QA to 10 developers around that full-time equivalent, something like that. So if you have 10 developers, you can use 1 QA for that, or 6 developers, depends a little bit. But you see that the test cases are very easy to implement and very easy to maintain, but hard to fix. So one to ten.

Speaker 2 [40:23]

Is load test and stress test the same thing or do they have differences?

Speaker 1 [40:29]

It depends a little bit on the definition. If you look to ISTQB, this is the testing certificate thing, they have different definitions. I'm not really sure about the exact thing, but they measure different things. If a system has a base utilization of 60%, how long does a request take to run through? Or you request how many requests you can do in one second. And so everybody's measuring this a little bit different, and the definitions are spread and very unclear. So you have to be clear on your team what you're really testing. But normally you measure something like requests per second or request time per 1,000 requests or something like that. But this is doing everybody different, and you have to be aware of what you're talking about, because load test is for everybody something else.

Speaker 2 [41:27]

Next question is, what would be sufficient methods for testing issues of race conditions or T-O-C-T-O-U? That's time of testing, time of check, time of usage, right? And last part of the question is, and how would best testing with concurrent requests?

Speaker 1 [41:47]

Okay, we start with, I lost the beginning. Can we start with the first one? For sure.

Speaker 2 [41:52]

What would be sufficient methods for testing issues of race conditions or TLCs?

Speaker 1 [41:57]

So raise conditions, they hit you everywhere here normally. And this happens somewhere. If you have a lot of load or if you run things in parallel, or you can do the concurrency tests in parallel, so you can scale this all up, then you will hit raise conditions. The problem is with raise conditions is that it's very hard to stimulate and that they happen in a test case. So they just say, okay, run test, failed with raise condition. So normally this happens all over the process. And then you must hope that the developers see this easily in their logs or something like that. So it's hard to reproduce them normally. What else was the last part of the question?

Speaker 2 [42:42]

The last part of the question was, how would you best test concurrent requests? I guess that's very...

Speaker 1 [42:49]

Similar, right? Yes, I would say.

Speaker 2 [42:54]

Then there's a question about regression tests in general.

Speaker 1 [42:57]

Oh, maybe I missed this.

Speaker 2 [42:57]

Ah, regression.

Speaker 1 [43:01]

So a regression test is if I execute a test every time again. What we are doing here is, of course, everything under regression. So we test in a deployed system normally, in a test system, or in the highest level, we test in a deployed system. And if anybody deploys anything, then we run our fast test suite, acceptance test somehow, or smoke tests, from the naming like half an hour execution time and then we regularly execute all the other things like overnight or daily or something like that. But this is everything is executed every time. It depends a little bit on the strategy but everything is regression. Who is not using automated tests to do a regression?

Speaker 2 [43:48]

So we have one last question is also about infrastructure and setup How do you observe and track failures, especially in the more complex test cases? I assume there is automation around that as well

Speaker 1 [44:01]

The good thing is that it's not... Okay, tracking test cases. Maybe I'll give two answers. Tracking a test case, we use... There are tools for that, like test trails or something like that, that you can see your test cases when it was executed and which state it was. Also with the exception that was raised from the test cases. This is one thing, or you do it with pipelines. so that everything is done with pipelines. So the pipeline is started, and if the pipeline is read, then you have some test cases that fail if you don't have a test case or test result reporting system. But if you have a system like test rails, it's a very good one, I would say, then you have a very good overview how often this happens and with which version and so on. Also, if it's complex deployments, you have 10 or 15 different versions deployed for things. and the second question is monitor