Which kind of software tests do I really need?

Speaker 1 [00:06]

I'm coding since 18 years, started in school, and I had a lot of experience with different projects and I want to give you a summary of what, in my opinion, is the best way to find the best kind of software tests. To start with, the first big question is why do we test? Some say because of quality or because it's easier to implement, but in summary, there's One reason that is much bigger is money. So the big question is, in my opinion, testing should be 20 to 50 per cent of the budget on your implementation, so why we can save money with spending money? And this is a really interesting question, because what happens if you do not test? But this is the interesting thing. We will produce a lot of bugs and damages, and for that, the only way to save money is that we have risk and damages from events, and with testing, we can reduce them. And so in total, the target is that we have less money spent with our new risk and new damages with testing than without. And so this is the only way to save money. But maybe we need some examples for risk and damages. I searched it, and I found it three days ago. You can buy a couch for €3, and if you don't have money, you can spend it in 12 months. But this is quite an interesting bug. Because you have an employee who changed the price, so maybe it's his fault. But on the other side, you have a software that should be smart enough to say, okay, if you are able to spend it in 12 months, maybe €3 is a little bit less. Are you sure that you want to sell it for €3? Or you sell it before for €3,000 and now you have a discount from around 100%. Are you really sure to change the price to that? So this is somehow a bug, or a missing feature, or it's the user who does it wrong. But in total, I think the responsibility is on the programmer and product owner side. So I think this is a bug. And if you have that, you have contracts, and then you sell couches for €3 until you're out of stock. Okay. Maybe it can go a little bit crazier. I go back in time eight days ago, we have a gambling machine. The gambling machine has a software update, and it has a bug, and you can cash money in and cash more money out without winning, without playing. So this is maybe just a little bug, but it produces a lot of damage, and with testing, we want to focus on things that can go really wrong and focus on them. So with unit testing, you will have a problem to find this error. Maybe it's going a little bit deeper. We go much more back in time. One month ago. Same situation. We have ATM. And we made also software update and then you can go there, enter your card, get money and don't balance your account. So this is also not good for your company. And we have a lot of these examples. This is two months, three months ago. We have security issues with RPs, and then there's a data set released with 12 terabytes of data from big companies. And so there are a lot of problems that are more or much behind unit testing and clean code and things like that. And there are so much examples what could go wrong, not maintainability in five years. If you have a software architecture, it looks like that, yeah, have fun. The good engineers are leaving the company, and the new one will have a lot of fun with that. And so, okay, but how we can find a solution for that? And for that, I found, in my opinion, two good ways for that. The one thing is with risk analysis, where you can identify the risk and think a lot about what is going on and what could go wrong, or the second thing is we just use an already available list of quality attributes for software, and we take the most important ones. Okay. For the risks. Now you saw the risks, and I presented a lot of events that already happened, and so if you create a list what could go wrong, then you can priorise them by how big is the damage, how big is the risk that it could happen. And after that, you have something like this table. You can then you have a list that you can go to your manager and say, okay, maybe we have to test this thing more in detail in that, and I need money for that. And I need a lot of money for that, because if you want to have a software that is running in five years, we have to think about the software architecture. And we have to think about software tests, unit tests, and so on. And so this is the first way, it's a hard way because a lot of things can go wrong, but I think it's very good for your project and also for your risk management, for your manager, so you should be in contact with them. The second way is quality attributes. We have a lot of quality attributes, we have, there are a lot of attributes, I showed them to you, you don't have to read it, but you can. These are all software attributes we have, or quality attributes. This is much. This is very much. And they are all existent, for example, time behaviour. If you have a business application, then time is not often the problem. You can execute and this is running. But maybe if you are an airbag, it would be nice that you're not doing updates. So it strongly depends on your project which quality attributes you really need. You have also attributes that race against each other. For example, timing optimization has often problems against good code or maintainability and testable. So for that, you have to be sure which quality attributes you really need and which of them you want to focus on. And so I would prepare for you, take this thing, take the five most important for you and focus on that. So this would be a good idea, and also read it somewhere. I will share the slides. But then we have the quality attributes or the risks we have, and for that we need tests. I tried to create a list of all tests I know, and these are also a lot of tests. And yeah, we couldn't do every test of that. That cost a lot of money, and this is also not every time the best thing for your project. So this should be really based on the things you need, and it's maybe better to do two Two of them with 80 per cent, then one with 100 per cent. I have something I bought at some test, one of them I would present to you now. Okay. Customer as tester. This should never happen that a bug goes to the customer. But it will happen. And the problem for our customer is that he has often no chance to give you feedback that you have bugs. And for that, my proposal is if you have a UI and so on, that you create something like a crash report. Here is an example. That could be in your application, and if something goes really wrong, then you have this opportunity to send the data to the developers. And it's like an exception chain, the exception goes to the customer, and the customer re-raises it to the developer, and he has to fix it. And with this easy method, you have very good feedback from the customer if something goes really wrong, and that box that are produced there are in a production environment, and they are from a real customer with real data, and this leads to big improvement in the feedback if you have a bug until the customer reaches it. And so, for that, it's a good idea to do that, like that. Okay. The next thing for testing, manual testing. Manual testing is a good thing. If it's not the only thing. So manual testing should be done, and it should be done in a nice and efficient way. For example, for a developer, he should be able to create a development environment for himself on his laptop with one command line, with data, with valid data, and with more data than two examples. That he's able to set up his environment very fast to get very fast feedback what he has changed. Yeah, he should write unit tests, but not only, he should be also able to do it very, very fast. The second thing is product owner and manager should be able to see the current development process. For that, you have often something like staging branches or release candidate branches, and From then, from there, you should build with ICD environments that are usable with test data, with more test data, and with good test data, that you can try the software in the current version. And it's not easy to build something like that, especially if you have a database in the background, but it's needed for the product owner, for example. And this is the only efficient way to get this in a nice solution. Then for the customer. The customer needs also something of testing environments. Because if you deploy a new version, then it would be good for our customer, depends on the case, that he could test it before. If it's working with his business processes, and if it's working in general. And for that, you should also build something like custom environments. Windows has a nice solution for that. You can register your PC on the alpha channel, and then you get releases before they are deployed in production for all of the testers. And, yeah, this is the second thing that you are able to test. The third example is something like test data or random test data. If we develop an RP, we have fields where we could add data, for example, names field. And if you're a European developer, then you know, okay, names have chars in a space and one part before and one after, and we have normal chars, but in total, if your software is deploying on other regions than in Europe, then you have very fast problems that your software will not work, because you have charts you did not support, or, for example, if you have an address in Germany, it started that you have a street and a house number, they started with numbers behind the house numbers, and if you don't accept that one, you will have will have problems. And what I'm doing with random data generator is one thing, I use it for methods. I just generate data that I want to add into methods. For example, if you have street names that you want to process, and I assume that the streets are passed without raising errors. And if you use fakers or fake data, you will be surprised what could go everything wrong. And also, if you're testing RPs, if there are fields not filled, or if a string is empty, or it's a number and not a string, or it's a string with a number, and so you can really get a lot of information. What can go wrong in your system? And here's example faker that provides a lot of things, like emojis, if you have comments of emojis, they should be supported and should also be able to save the database. For that, I use unit tests, for example, that one, use the method and do a lot of iterations and validate that. So the next thing, unit tests. test is also very interesting, because it's from the functional tests on the lowest level. So you saw the slide before, and the functional tests are mostly the most important ones. And with unit tests, you can do a lot of things, but you couldn't do everything. And from a testing hierarchy, it should be something like I do unit tests, and unit tests classes, and after that I do integration tests, and the integration tests test the connection of the classes, but not mainly the internal behaviour of the classes you already tested with unit tests. So you need a testing hierarchy, and on the highest level you have end-to-end tests with real external systems, but on each testing level of functional tests, you should use you should focus on the integration problems and not on the problems that could be fixed from a lower level. The lowest level is unit testing. Unit testing is hard to do right. Here is an example. The biggest issue a lot of people do is that they test only the good case, and it's every time a good idea to test all of the bad cases, data you didn't expect, check that exceptions are raised, and so on. And the second thing for good unit tests, what I can say to you is that functional orientated code is a very good idea to use. What does this mean? I use a lot of data classes. For example, if you have an RP, then you create a data class with a command that is coming from the RP end point directly with all the information. And then this data class, this command, can be processed. And the process is also a data class and gives you the feedback, also with humans. Here if you have, I would, if you need a validator, for example, if the class is in total valid, then I would create a new class, human validator, that has a valid, that expects a human, and gives a Boolean value, so that you have a standalone validation class that is only responsibility for the validation of the human, and the human has maybe some methods like that. minor methods that are doing small things, but not big things. Is adult a nice example? Yeah, for now it's correct, but if you do it international, then it's getting fast complicated because every country has different rules to be an adult, and so when you need the country as input and so on, I would prefer to split it later in a separate class to the logic of the adult method that is only taking a human and giving a backup. And with that validation class or state classes, you can return and test it very easy because you have a class that is doing only one thing, maybe only one method, but it's working normally really great. So unit tests should be fine. The next thing is training data. We have here a lot of data talks, and I also created a few papers and also new test data for text-to-speech in German, and I also created papers with running them and training them and so on. And what I learned is that the test data should be also tested. Because your network could only learn things that your training data supports. And this is the only thing it could learn. And here you see, for example, we have words, we have an audio file and we have a text file of that that is representing the audio, and we have sentences with up to 60 words mainly. And you would expect that if you train that and you enter a sentence with 200 words, that the results will be very bad, because your training data does not fit for your propose of what you want to do and what you want to train. So I would say if you try something to learn with neural networks, generate charts, have a look at that and let you see if it's fitting to your requirements you have. Okay. And so in summary, what does this mean? Every project has different needs. You saw it with the quality attributes, you saw it with the risks, and you should really focus on that one and try to solve them. This is also a lot of communication with your management, because if your management says, yeah, the application maybe it's running three or four years, and then we shut it down, then you don't have to focus so much on architecture, and it's also burning money if you test things that are not needed, but you have to communicate it to management and it's hard discussions and they don't want to say, for example, how many users will be on the system or what the management will expect. And if they say, okay, you have 1,000 concurrent users, then you can design your software architecture for 1,000 users, but if there are 100,000, then you will have immediately a problem. So for that, you need a lot of commitments from the management, and they must also know If they leave these commitments, that this will lead to a lot of money investing before. On the other side, if you do too much, if you say, okay, the system should be designed for 100 trillion users at the same time, then your architecture is totally overkill, and you will have a lot of new problems, like with microservices. You have services, what happens if a service is going down, what is with updating, what with different update regions and partly updates and so on, and so for that you need a lot of communication with your management. And then the second thing is try it with your teams. Think about what could go wrong, you will be surprised what everything, what really could go wrong, for example, a developer is leaving the company, or the best developer, what is LAN, or with quality attributes, just have a look at them. I also find this list and I didn't know everything. At the beginning I didn't understand everything. But it's really a good idea for you as personnel and for your team that you really have a look at that. And so, yeah, just do it. And then test, test, test. And that's it. Thank you. Thank you.

Speaker 2 [21:52]

You actually nailed the timing. We have seven minutes for questions, and there's some great questions in here, so thank you, everyone, for putting them in. Also, can we just have one more big round of applause for everybody working, presenting, and talking in not their native language? Because, heck yeah, that's so frickin' impressive, and I want to look acknowledged. Okay, so lots of really good questions in here. First of all, where would you start mocking in unit tests, would you only test on method? What if the method calls multiple methods and methods from other modules, which is...

Speaker 1 [22:30]

Okay, mocking is really interesting, because in my opinion, mocking has often the problem of a wrong architecture. For example, the database. If you want to use a database, then you have a database class that is using. But for that, I would prefer to use something like an onion architecture, that the database is behind an interface, and you just use it if you really need it. But in most things, if you have an onion architecture and you have your logic in the core, you don't need to load the data from an interface or something like that, and then you just have only your logical methods, you can create objects directly, and for that, you can remove mocking a lot. For me, I don't use mocking, I try to avoid completely, because it makes so much work and more problems than solving.

Speaker 2 [23:27]

Yeah, we have a clear agreement from the audience.

Speaker 1 [23:30]

Yeah, the rest didn't allow me.

Speaker 2 [23:33]

whole conference on mocking. Let's keep going because we have so many good questions. Do you always unit test everything? When would you skip writing unit tests?

Speaker 1 [23:43]

Skipping unit tests in general for prototypes. If I do a prototype for new big functionality or something like that, I just write the code and get it somehow running if I use a new technology. And after that, I throw the prototype in garbage and write it new with unit tests and then I know what I'm doing. And I skip unit tests for, yeah, I prefer the onion architecture or everything that is far out of the centric, for example, with communications and so on. Because this is a socket communication, for example. It's really hard to unit test. Because then you have to fake it and then you get data that you don't know if it's really in this format. you have in total one.

Speaker 2 [24:35]

Yeah. Sorry, there are a lot of new questions coming in, and we're just not going to get through them all, so I'll just say also, please chat to Pascal after this talk and, you know, pick his brain further. Five minutes. Yeah, yeah, five minutes. How do you generate test, oh, sorry, yes, how do we generate test data that has realistic distributions, et cetera? Can you recommend a tool for those? So I guess this is where you want synthetic test data that sort of looks like what you might expect in production but has not got personally identifiable information and all that fun stuff in it.

Speaker 1 [25:06]

I try to generate it automatically with Faker, for example, but the problem is it's hard. So mostly semi-automatic, maybe with some scripts. So it's not so easy to generate good data for a local test environment. So the best thing, take some real data, take some fake created data from random generators and some manually created data, but you have a mix of different sources.

Speaker 2 [25:38]

Yeah. I'll add to that. If you have any folks in your team who are from other countries or other places in the world and they don't mind using their names as test data, I always find that useful. So use people who don't mind being used as test data. How do you ensure reproducibility and determinism of unit tests when using Faker? Which is, yeah, how do you know if your test data is generated every time? How do you know that you're testing the same qualities?

Speaker 1 [26:06]

No, it's not possible. Because the data is every time new generated, but this is not the idea behind it. The idea behind this is that you generate the data and expect that nothing goes wrong until you save it to the database, for example. And this is often a very good step if you can save a name of Chinese jars in your database when you are loaded again, then you have a good way inside and out, and if it's looked like before, then it's already a good test, so this is often a way, put it inside, load it again, and test if it's equal. And deterministic, it's not, because you have some edge cases, and sometimes that happens, and sometimes not, so big loads, or sometimes it crashes, yeah.

Speaker 2 [26:55]

In the customer as tester example, the user got a more extensive error report presented. Oh no, it just moved. Sorry. This is the problem with the voting is it changes order. Is this a good UX UI experience? So they're seeing this extensive error.

Speaker 1 [27:13]

UI UIs? I think no. I know I don't like UIs. Fair enough. But it's working, and it could be improved, yeah.

Speaker 2 [27:31]

Yeah, I mean, the developer needs the extensive error report, and the user needs to know what they can do, if anything.

Speaker 1 [27:36]

And they should also know what they send to you. Because if you have, for example, you can do it in an offline environment and you can create automatically an email with the attachment if you have no connection to the server. Then they'll send data to you and maybe they will have a look inside that and it's a technical report and it's hard to make it nice.

Speaker 2 [27:58]

Yeah.

Speaker 1 [27:59]

and the customer should never see this page. So, hopefully.

Speaker 2 [28:05]

Catastrophic error, X said Y, 12229. Okay, we have time for one more chat. Thoughts on TDD, test-driven development. In one minute.

Speaker 1 [28:16]

I love it. It's really good, because you have to think about, okay, what is it? You write your test, and after that, you write your method, what is doing that. I love it, because you have, at first, think about what you expect, and this is the really tough part, and then, in the second way, you see if it's really working with your code you produced, because sometimes Python does things that are not expected, especially if You are newer to that and you have some minor edge cases like default construction with lists and methods. It's reused and yeah.

Speaker 2 [28:53]

So do you think you should write the test first and then write the function like do the really pure TDD thing?

Speaker 1 [28:59]

In a bigger software with inside the core logic, yes. And if you have a prototype or in data science, it's also really hard to write unit tests in general because you call a command train and then it's doing things. When it's returning things, what should you test? Yeah, yeah, yeah. With unit tests. And so I prefer to write the test first. It's also nice for pair programming. One person implements the code, the testing code, and one person implements the logic, and then the tester who writes the test tries to crash the software of that one who is implementing that, and then you find a lot of edge cases how to destroy the other software. It's like a game. I have a nice idea to make your code not working. the empty string.

Speaker 2 [29:55]

Well, we're at time, so everyone say thank you again to Pascal for sharing his expertise.