What if I tell you that your specs are broken

Speaker 1 [00:04]

And I'll let him introduce himself on his own terms.

Speaker 2 [00:09]

So, welcome everybody. Thanks a lot for joining this talk. So, I'm Samuel and today I'm going to talk to you about network APIs, how changes might eventually break your production services and how these might influence your users. During this talk, we are going to check a bit more on what are eventually mitigation strategies that can be used to reduce this probability of these type of errors and eventually tools that helps you to detect them early on and to actually try to avoid to have these type of errors at all. So about myself, I'm Samuele, I'm a backend engineer at Yelp and I'm, well, mostly a Python developer. I do like a lot API design and I'm mostly working designing them within using this Wagger specification. I'm one of the maintainer of the most commonly Python library that supports Swagger, that is mostly Bravado Pyramid Swagger and Bravado Core. So maybe if you're using them, you're actually interacting with something that's front-end to time and touching. What about Yelp? So actually, Yelp is a company where the mission statement is pretty easy and is easy to digest. We want to connect people around the globe with great local businesses. What we want to do is to provide features to the users to make sure that they get access to the businesses that are around them. I mean, I do expect that most of you that are here in Berlin are actually not usually located here in Berlin. So maybe you're angry around and you want to find a restaurant, a bar or something. Actually, thanks to Yelp, you can actually find those restaurants and find reviews about them and actually decide if this type of restaurants might be good for you. But Yelp is not only about restaurants. It's also about, it's more for all the type of businesses. We are actually having support for local services. So maybe you are here with a car and you have a problem or your pipes broke at your place and you need a plumber you actually can find those services as well. The overall agenda for this talk is going to be split in three main areas. The first area is going to be Swagger 2.0. I will give you a brief introduction just to make sure that everybody is more or less in sync with what specifications are and how they look like. It's not going to be a super deep talk on Swagger so don't worry about that. Then I'm going to present a use case example is basically a bookstore API that is going to be used for the rest of the talk to make sure that we can actually have examples and see how breakages can actually influence our user experience. And of course, we need to introduce what backward incompatibility is from a more formal point of view and actually investigate more on the effects of them so moving on on the swagger specification swagger is the de facto standard for defining apis there are multiple standards around them but thanks to swagger or the evolution of swagger that is basically the open api 3.0 you do have access to a great amount of tooling so you do have swagger ui and swagger editor to simplify the life of developers and users of your api so your api will be automatically documented through a website that is accessible also for you know for people around the globe that are actually not technical people so they can just use them and read them and do whatever with them is actually language agnostic what that means is that these wager specification are using like a sort of neutral language to define them this means that the definition of your interfaces of your network interfaces are completely independent from what is your backend implementation it doesn't matter what languages are you using the specification of interface are defined with a different set of language and usually those are serialized using the json or yaml format this is done mostly to make sure that the content of your specification are eventually readable by humans but most importantly is to reduce the entry fee that your backend implementation will require almost all the most recent programming languages out there they do have kind of native support for json some of them they do have support for yaml Or anyway, chances that there are great libraries to deal with them are extremely high on all the languages. Swagger specifications are machine-readable as well and self-contained. What that means is that thanks to the swagger specification, you do have the capability to provide all the information to your user on how to interact with your APIs. Nothing is left behind. All the information that you need are going to be present there on this single place that are the specs. and most importantly is actually specification are extremely extendable on swagger 2.0 you do have vendor extension what this means is that is actually you can enhance the feature that your apis are producing by adding specific extensions for example in yelp we have two main extension that actually help our development life we do have gateway services that are services with a really big amount of endpoints and such conditions is not really realistic to think that there is a single point of contact a single team responsible for all the endpoints and you know developers wants to have information eventually users want to have information about the npi the endpoint so we do use the specific vendor extension to actually target a specific team on top of on top of what is the endpoint definition as well another really good example is related to slos that are service level objectives so you want this endpoint to give an answer within i don't know hundred milliseconds for the 99% of the time. And well, how could you actually communicate that to your users? Or actually, how could you monitor this? What we do in Yelp is actually encoding this information on the specs. So users are actually aware of this information, as well as we have monitoring systems that picks up the specs and automatically create alerts and monitors and dashboards that allows you to keep track of what's happening and be alerted if you are not upholding with the expectation that you to you gave to your users so how specs looks like i mean specs are mostly divided in three main area there are there is the metadata section where you of course you have to specify which version of this specification you're using and you are mandated to provide information about your your service so you have to provide some information for this you are required to provide a title you know just a super short description just something that explain what the service does and a version additional information that are usually good to provide to users is a bit longer description of what your service does you know from a code name users won't really understand what the service does so a description is usually beneficial as well as providing them licenses term of usage contact points and stuff like this so you know this is just a metadata around the service it allows people to know how to deal with this and to understand actually what it does. Then we have the server's information area. On this part, you know, I mentioned that bugger specifications are self-contained. In order to be self-contained, we need to make sure that our clients know how to talk to the service. So which protocol they are supposed to use, where they are supposed to send the request, and, you know, how the data are supposed to look like. In this case, you know, for example, you have JSON stuff. But as well as eventually you do have some security definition. Maybe you using tokens or you are using additional different type of authentications but of course specifications are meant to define endpoints i mean in the http word what is an entry point is actually this endpoint that has a path and a verb and in the specification you actually have a specific area that is named paths that allows you to define them in order to define them the minimum requirement is well providing the path the http verb that you're going to use and well you want to define what type of status code are possible as response in this case we are saying that there is a 200 and you have to provide a description but now what's actually the importance of providing this if you are actually not giving a schema to your user your user will not know how to use your response body and actually they might care about that so it's actually recommended to provide a definition of the schema within the response this is optional highly recommended to do though and well then you do actually you know your service might also reply with other status code or the responses are all the same with all the same schema so you actually can use the default in order to say you know all the status code that haven't been defined they will be answered with the schema and then there is an interesting attribute that is an operation id and this is mostly recommended to be used especially in the condition where this wager specification are going to be used to generate code think about you have mobile application like an android app or an ios app that wants to interact with the service they need to generate code and usually you do use something that is as wager code gen to generate this code directly from the specs and you want to provide a reliable naming for them so thanks to the operation id you can achieve this so how but at this point how do we write this specification we do have two main approaches to write them The first one is a sort of manual approach. This is what we actually use at Yelp. In this context, developers are supposed to know how Swagger is formatted, what is the logic behind it. And this actually enhance the probability of developers catching issues while they are writing specifications. An alternative approach is actually related to spec generation. And this is mostly common within Java frameworks, for example, in Spring. and there are as well a good amount of libraries on Python that do this. In this context, developers can actually be completely unaware about Swagger specification. They are writing the backend implementation and from the backend implementation, the specifications are derived. The downside in this case is that developers are not actually exposed to the spec, so they might not be aware of breaking changes that eventually they are doing, but the advantage is that they might actually decide to start to publish the specs with a different format because someone did ask for it. So with this example, with this bookstore example, what I want you to envision is that, you know, you should be assuming like you're working on a startup environment that, of course, is related to books somehow. And we want to expose a web service for them, for the users, and the web service should allow you to interact with those, you know, with books. So for sure, you will have something that allows you to search book, get more detailed information, sell books and do stuff around them. And of course, well, let's not forget that authentication is required. I'm not going to say much more about authentication because it's not the target. But let's remember that authentication potentially is something that you need to keep in mind, even if, you know, your project owner doesn't say that explicitly. And let's remember, we are in a startup environment. We want to move as fast as we can. We don't want to wait for having the perfect specification at first before starting the implementation. So we will leave with the fact that the definition of how a book looks like is not going to be defined from the beginning. So we will make some assumption and then eventually we will iterate over the model and we will make sure that, you know, we actually are accommodating the feature that are actually requested. So the end point that we will need in this case are mostly five end points. them at least those are the most important that i see is like a certain point an endpoint that allows you to get details of the book and then once again you know the buy borrow and sell end points those was more or less what was presented before but what about the book model so we'll get started with you know we put the model on the definition area we said that is named book and that's all good but at this point the book is actually a model it does have attributes so we will start to say that well first of all is an object and then we will start to define the attributes the first thing that came to mind is that you know you need like as something like a primary key for this and on the books environment well that is just the SBN then you will have a title and the authors what you can see over here is that actually for the authors we decided to delegate another area of the specs to define how the outer look like but in this case it's also reasonable to think that every book has a primary key as a title and at least does have one single outer so we can actually say as well that's all this parameters are required and this is actually how this book model will appear through the swagger UI you can see that is basically it doesn't tell anything anything related to swagger you know it's just something that you can digest and this is as well how the endpoint will look like now let's assume that we want to get more detail on the search endpoint just to get more detailed information here we will have a query parameter that is going to be required that allow is you know is this search term that you're looking for and the response let's say that our service is perfect is never going to have an error is always going to be at random response and will contain the list of books that you were looking for so now we do release we have released this this service and we started to have feature requests from the users i mean it makes sense users want to deal with your service and wants to get better information eventually and actually they came out with a request that is hey it would be so great if we can actually look for books that have been released after a specific date we know that this book was released last month so we want to search for it so the first thing that came to mind if you remember how the book was modeled is that we actually don't have information about a release date in the model so well as i said we need to iterate on it we will add this information on the model and all is good But as well, we need to expose the capability to filter for this release date. So now we have the discussion of should be this parameter required or optional? And of course, here the answer would be most probably has to be an optional parameter because otherwise the old clients won't be able to talk properly. And this is how this pack evolved in this case. You will see that there is a release date that has a format date. So you also get some validation for free thanks to the specification. And now we move on. we continue to evolve and users start to ask about genres you know there are so many different type of books but eventually i'm interested on a specific kind of books so actually i want to look for them once again you know do we have this information we don't have it yet so we actually have to define the difference in this in this case is actually this type of parameter is actually enumerable not all the strings on the planet are actually very generous so you will get the agreement of any having an enumeration for that and well again the same question as before you need to define the parameter and this will be once again an optional parameter and this is more or less how the specs might be looking and well we'll continue to evolve and well our initial set of genres wasn't actually so wide we are starting to have more books so we get the request of please add more and more because you actually have them on the store and so we just announced the list so what is the real topic of this talk at this point is understanding what is backward and compatibility understanding how we could have break it break the specification and so eventually have impacted the user experience before moving forward i want to actually provide a kind of an official definition i mean wikipedia is kind of the leader for this type of things and well what wikipedia wikipedia provides in this case is something like is a property of a system that allows interoperability with legacy systems the focus point is the interoperability and the fact that there is legacy. You cannot really expect that your clients are always going to use the most updating specification. Maybe you just updated them. How could you expect that a user does already have the latest one available? And so at this point, the question is, do we actually care about them? Should we care about them? And there are mainly two cases where you might eventually, I'm not saying that this list is going to be complete, but you eventually might decide to not care about it one is if we have full control on all the clients on the planet so basically saying you know all my users just send them down requesting hey stop using my service you restart your service and everybody gets the new specs oh this is good but actually this implies a downtime and i don't think that everybody wants to have a downtime on their services everybody is fighting for you know kind of reaching the under percent uptime goal so you cannot do that at least generally speaking you can't another approach might be related to have versions endpoints where all the definitions are fully independent what that means is that you have the initial version of your endpoint you evolve it you create a new endpoint with a new version but you cannot share anything that already exists on the spec because otherwise you get the risk of modifying things in this case though i mean even though it looks great actually there are there is a downside if you are using generated code for mobile application this leads to the burden on the mobile developers to actually create the mappings between the different representation of the same model and especially for small changes that is not something that is beneficial you are going to invest a lot of time on doing something like this and eventually you don't want to do that so let's try to understand what is actually the effect of these type of changes and how these could bleed on production issues on you know outages or stuff like this so on the web board you can see here on the graph this is a representation of the traffic that our backend services are receiving according to the version of the survey of the web service that we have released you can see that the different color represent a single version and you can see that all of those are split are separated almost vertically so what that means is that when the color is the same actually the same version is replying to the requests that are coming to the service. And when there is actually the transition that is not a straight line, it's like, you know, it just moves a bit over time, this is actually a real deployment that the service was having. In the case of web, what usually happens is that a user opens the web page, it loads some HTML that contains links to JavaScript code, it downloads the JavaScript source code, and eventually there will be additional requests that will be sent to the backend. In this condition, what might be happening is that the request to download the source code so the javascript code and the request that the javascript code is doing are eventually hitting two independent versions of the service and this might lead to errors if the apis are not really compatible but the benefit of having this version that are separated vertically so over time you will tend to have a single version you actually will have information of you know there will be no issues all is good but this actually changes over the mobile development whenever mobile apps at the receipt release cycle is way different if you can see here in this graph actually there is a color that gets to have a bigger amount of traffic received and this actually you can also see that the color the versions are not split vertically any longer they are horizontally separated this means that there will be an old version there are old versions that are still hitting our services this means that in case of a backward incompatible change this error will not fade away over time but it will stay on your service maybe will reduce a bit in percentage but it will stay there so at this point the question is for mobile applications specifically are you are you ready to give up on those percentage of users and eventually especially if it's a small change maybe you don't want to do that so backward compatible changes you know those are not something that you would like to have these those impacts user experience maybe the application starts to crash and that's not great as well as are hard to identify especially if you publish the specs before and then you do implement the backend change as well as those are usually related to change on the specs that were weren't really intentional from the developer. So how could we reduce the probability of this to happen? Well, first of all, let's try to define guidelines. Let's try to make sure that, you know, you are limiting a bit the freedom of the developers, but you are increasing the stability and consistency of the spec within your company. And as well as try to use tool. You know, humans are really good at verifying this, but starts to be boring and time-consuming, you know, So it's not really great usage of human time. So let's try to get support from machine. Try to use the tools that are already available. If they are not, try to build them and make sure to integrate them within your pipelines. At Yelp, we did actually develop a tool that is named Swagger Spec Compatibility. This is freely available on GitHub. And basically, this tool is a tool that helps you to determine if spec changes are safe. This tool has been built as a static analyzer. so no runtime penalties on production and has been built with uneducational purposes in mind we want to provide to developers not only a red flag of well this is actually is not a great change but we wanted actually to make sure that we are providing them explanation why this is bad at the same time is extremely modular and configurable so you can actually integrate it and do some magic with that if you are required to the tool presents itself on the command line more or less like this i'm not going to spend much time because we are about to be at the end so now we made some changes on the specs we changed it three times did we ever broke them now the question is we need to be sure that we have done that or not and just to recap you know we added the release date the genre and then we updated the list of the genres so this was the initial specs and then we change them we added this release date either on the parameters and on the response body eventually all looks good as for now so let's try to run the tool we do run the tool and the tool tells us oops actually there is a narrow report you know we added some required properties some on the request contract and this is weird so we'll try to get some more information thanks to explain command and we get a detailed information and a link to a web page that eventually provides us with mitigation strategies if we try to read more carefully what was the report actually you see that on the reference up there there is a reference to the selling point and it wasn't the session point the one that we did modified so actually how was that possible if we check this waggery eye we actually see that the book was used on the body of the request so eventually these might make sense it wasn't intentional but we were actually breaking the specs what what has to be done here is actually try to understand what were the changes and you see the evolution here is pretty simple we added the release date we added it as required because of course a book was released so it will always be there and this is actually the final specs and this justified the issue that we created on production what we need to do here is taking some action of course if you use the tool you do care about that but you need to decide eventually is is interesting to revert the change or eventually this endpoint wasn't actually really public so it's good enough to say okay we go to production it's fine that is not really backward compatible but this was at least an intentional change it wasn't up it didn't happen by mistake or just because you weren't aware of that about the other changes well you know here in this case we added a generalist and in this case if we try to run the report actually the report tells us that all was good so actually how was different this case from the one before and well we added the generalist and that basically was it we never touched the required so all was great at this point but now do you think that actually adding the enhancing and enlarging the list of genres did actually change anything i mean we moved from something that looked like this to this did it actually break let's try to run the tool and the tool actually starts to complain about multiple endpoints saying that actually adding enum values to response to responses is not a good idea and on the documentation online it actually tells you that what's going to happen there is that all clients are going to receive a new enum value that they were not ready to support and if the application is like doing python you know wrapping of enums is actually going to crash because it doesn't know the value so what you you can do there is actually well there is not much that you can do from a compatibility point of view but you can try to ensure compatibility through some business logic making sure that eventually you are aware of the version of something that allows you to not get into this specific issue so to recap this is something that i want you to remember at the end of this talk there are four important points the first one is work aspect changes i mean they're inevitable if your company is moving and you're releasing new features chances that you are modifying the specs are extremely extremely high and you need to leave them and specs eventually services eventually tend to be larger and larger with the amount of feature that you release and it's not really credible that you actually know all the details by heart of the specification so try to make sure that you know there are way too many moving pieces around so try to make sure that you rely on tooling and deeper analysis instead of yeah well that eventually should work because this is my memory you know memory of humans is not necessarily great all the time. Backward compatibility is important, especially if you are supporting mobile applications. You know, you have to have a really good reason to actually decide to give up on a percentage of users. You don't want to lose them. I mean, those is what keeps your business running. And most importantly, backward compatibility should always be verified. If you are not ready to use tools, try to train humans to actually do that but please make sure to use tools as soon as you can because boring tasks are never great for you for humans so that's all for my talk if you have questions and we have time please just send them to me otherwise just poke me around and specs and everything will be available on KitHub as soon as I'm out of the room

Speaker 1 [26:19]

So, we actually have time for plenty of questions. That's a great response. And let's start from the first hand that I've seen, Rezan. Thanks. Thanks for the good talk. While having the Swagger documentation is super useful, I find it always problematic. How do you keep the Swagger spec, which is in a completely different file and also tends to get very big, actually in sync with the code? I mean, no one really forces you if I update my Flask application to also update the Swaggy documentation so it's very easy to actually do backwards incompatible changes without actually noticing it.

Speaker 2 [26:55]

So, around this, actually, my recommendation would be make sure that your backend is aware of Swagger.

Speaker 1 [26:56]

So,

Speaker 2 [27:03]

For example, at Yelp, we mostly use Pyramid, and that was one of the reasons why we do have Pyramid Swagger. That is a tool that actually verifies all the requests that are incoming to the service to make sure that they are compatible with the specification. So, they validate the requests as well as we validate the responses to make sure that we are not sending something that is not compliant with the specification to our clients. so my recommendation there would be try to make sure that you're using tooling that actually is aware of swagger and that you perform validation on your production traffic in this way you can be relatively sure that specification and what the backend implementation expects are in sync

Speaker 1 [27:40]

So, next question. Yes, thanks for the talk. I wanted to ask if you actually use Swagger CodeGan for your clients?

Speaker 2 [27:49]

Yes. This was actually a talk that a colleague of mine gave a few months ago. We actually are using this Swagger code that has been released by the OpenAPI community, but we did some minor modification to this generator because we wanted to reduce freedom. As I mentioned before, we want to have clear guidelines. We want to provide... This is the way that we want to do things. We are free to change if it makes sense but you are not really free to you know you're using java you use react or coroutine no you need to decide one

Speaker 1 [28:24]

More questions? Also for me, thanks for the talk. It was really, really good. My question is, why haven't you moved to OpenAPI version 3 yet? Is there a specific reason, or are you planning to switch? And then the follow-up question would be, what do you think about the connection package that Zalando released that basically takes a lot of overhead from you? Do you have any opinion on that?

Speaker 2 [28:51]

Um, so I'll try to give an answer to the first part. Um, so we haven't, we don't have a real plan to move out of Swagger 2 yet. This is mostly related to the fact that OpenAPI 3.0, it does provide some benefits respect Swagger 2.0, but those benefits, at least in our opinion, are not necessarily good enough to move yet. Of course, at a certain time, eventually we need to move, but as for now, there is no pressure, especially because we are also considering different wave ways of publishing apis so it's more about we are not ready to pay this cost yet about the connector i don't actually have experience with that so maybe it would be nice to talk about it later

Speaker 1 [29:30]

I'll join that talk as well. We use Connection. Anyone else for the questions?

Speaker 2 [29:34]

There's a person there.

Speaker 1 [29:38]

Toot toot toot toot toot toot.

Speaker 2 [29:39]

Toot toot toot.

Speaker 1 [29:41]

Thank you for the talk. I actually used Bravado before, so thank you for the library as well. My question is related to the first question, which is, when you say the backend needs to know about the Swagger specification, what do you mean by that? Do you mean that you also have integration tests that runs against the Swagger spec with your server to make sure that those endpoints exist and that the requests and the response are compliant with the Swagger specification? So,

Speaker 2 [30:11]

So, something that we do is making sure that on all the development environments, so either on dev or on pre-production environments, we do have validation enabled on all the cases. For all the requests, we do have validation on. This means that especially while testing our application before releasing that, we actually see that if we actually got a response back from the backend, this mostly means that the request was valid. At least it wasn't inconsistent with what the spec says. as well as the response was valid. This is something that we do. As well as we do have validation of production traffic, but this is usually done on percentage of traffic because we don't want to add a lot of overhead for production traffic. Something that we don't do though is related to the endpoints that we defined. So you were mentioning the case of what if the endpoint is actually not defined on the backend implementation? We don't do that mostly because the APIs that we do release are actually not completely public. Those APIs, the Swagger specification for the gateway services, as for now, those are meant to be used by applications and the specs are not supposed to be public. For public APIs, we do have integration tests for all the endpoints as well.

Speaker 1 [31:25]

Thank you. Any other questions? I see a hand there. This will be the final question for today. So, shoot. As you said, you have no reason to start writing the APIs you have right now. I'm from an environment where we use GraphQL for some stuff. For a ground zero project, it would be better to start with GraphQL or open API 3 in your opinion, given the tools we have today.

Speaker 2 [32:04]

So, around that, actually, this GraphQL is something that at Yale we are investigating and trying to understand if it makes sense for our use case. There is not a real answer to this question. It is more about it depends on your use case and what you want to achieve. The most important thing there is actually that OpenAPIs or Swagger are defining what are basically REST APIs. So, you do have access to resources. While GraphQL is more about I'm sending you something that is a query, like an SQL query, and I will receive a model back. so the interaction model is very different this doesn't change the fact that backward compatibility it might be present on graphql so please take care of that as well but there is not a real answer on is mall so open api is better than graphql it really depends

Speaker 1 [32:50]

Thank you, people, for your attention. Let's give Samuel a big round of applause.