It is all about files and HTTP

Speaker 1 [00:03]

Hello friends, good morning. I'm Efe Öge and I'm from Turkey and I'm going to talk about HTTP protocol and files today. So let me introduce myself before. I've been working at Hippo for six years up to now. My current role is back-end and infrastructure development. Hippo is used to be a consultation company for mobile or web applications. Now for a two or one year we're more focused on decentralized blockchain applications. I am also a co-organizer of PyCon Turkey and by Istanbul user group we successfully had our first PyCon Turkey in 2020 as online due to COVID. I'm happy to see familiar faces today from two years ago. Hopefully, we will have our second PyCon Turkey in person soon. Finally, I have a blog under EFA.me. Okay, enough from me. What I'm going to talk about today, it will be a 30-minute long talk. We will have five minutes Q&A at the end. Slides are available in my blog right now, so you don't need to take any snapshots or something like this. There won't be any code snippets during the presentation so it will be more like an architectural explanation of HTTP and how the modern web applications work currently. Also I am not going to cover network level stuff like TCP IP or any kind of port mappings or other stuff. So at the end there is no need to be a backend or API developer for listening to this talk. Okay, it seems nice, but what is the file on HTTP? Before answering this question, I'd like to ask you a kind of funny question. I'd like to find an analogy to explain the topic which come up for all of us at this room. So what is the common thing for all? Are we all backend developers? I think, no. We have a lot of data scientists or data engineers currently. Are we all in Berlin? Also, this is no. We have online attendees from all over the world. Hello, folks, by the way. So what's the common part of us? I think all of us visited this website, pycon.de. So I thought that since we visited this website for once, maybe it might be a good case study to work on it and explain the HTTP basics. So I'm going to explain the HTTP principles on PyCon.de. For this, we need to learn to use a tool almost all browsers have, which is a network manager or network monitoring tool. I don't know why, but each browser gives it a different name. I use Firefox. You can use whatever you want, of course. In Firefox, you can open Network Manager by pressing Command Option and I if you are a Mac user. And if you succeed to pressing the K combination, it should be something like this when you open it. This is a real screenshot from Network Manager of PyCon DE. It seems a bit scary at first, but no need to be afraid of. Let's try to make it clear together. We have status at the left-hand side, as you can see. Most of them are in green color. It should be something good. So the first two rows are in neutral colors. Also, at the end of the rows, we have a 404. I think all of us know this. It means like 404 does not exist. Also, at the second row, we have a method. At the second column, we have a method column. All of them are get requests. OK. So at the third column, we have domain, actually. But as you can see, there is a difference between the first row and the others. There is a lock on each one, but the first one has something like no sign top of it, so I'm going to come with soon. Also, on the fifth column, we have type, like HTML, CSS, JavaScript, VOFF2, something less. It's something weird, it seems. But actually, it's a new format for web fonts, specified in 2018. So we can think them as a font. As both two in the type column, there should be something which sets this type, right? So where this information comes from? Does it come from the extension of the file? OK, we might think like this. But as you can see on this line or this row, the file itself ends with both two. But the type is something different, which is octatestream. So also, we should keep this in our mind. I'm going to come to this later, too. So let's move on. We have size and duration columns, too. Here, we will not attend these. Also, at the right-hand side, the right column, We have a timeline, which I'm going to explain later. So I think we should first iterate the first three requests. When I type to PyConn DE to my browser's address bar, at the first, browser checks that is there a well-known prefix at the address bar, like HTTP or HTTPS. If not, it automatically adds it to this. Now we have HTTP PyCon D, which is added by browser. But as you know, HTTP is unencrypted protocol. Almost all of the modern web applications use HTTPS, the encrypted version of HTTP. Therefore, there is a redirection from HTTP of PyCon D to HTTPS version. But this is done by web server of PyCon DE, not from the browser itself. But as you can see, this is a HTTP 301 redirection. So it's something permanent. So we think that web developers of PyCon DE thinks that it will be always HTTPF. Then we see that there is a second redirection from PyCon DE to 2022 PyCon DE website. Also, the creator of this website is real smart people, in my opinion. As a quick solution, they could just use subdomain as next year, and they just used root domain for this year. But what would be the problem? As you know, we currently share a lot of URLs in Discord, Twitter, and other platforms for social event link, for schedules of talks, and other stuff. After a year, we would have a new website for 2023. And all of the existing content of this year moved to the subdomain. But the shared existing links will be dead. So it is important to remember that URLs are shareable, and we should support their lifetime too. At last item, we got 200 OK response from 2020 PyCon DE, and it is the HTML file. Then what should we go next? OK. Maybe we should come again to the timeline. At the right-hand side of the network manager, we can see that this is the timeline. But as you can see, until the first HTML is loaded, you can see from here, I guess, it's a bit small. But the images, like these shape images and other images, aren't loaded yet. Because browser itself couldn't know what kind of images a website has. So it first needs to fetch the HTML and check the image text and other stuff inside of it, then request their source. At the end, after all content is loaded, there is a 404 request for favicon, as you can see at the bottom. Favicon is a small icon of a website which helps to identify on the browser window. But I'd like to mention that this request is initiated from the browser, not from the HTML of the website. So we should keep in mind that, in some cases, browsers or browser extensions can initiate requests for you. OK, we talked about HTTP codes, redirections, orders of requests. So now we can come back to the file types. In general, we can summarize file types, like this. We have our index HTML or other HTML files. Then we have static or asset files. They are mostly CSS or JavaScript files. They are like end products of front-end development. By the way, do not be confused static and immutable. Content of these files can change over time by developing new features and other stuff to the website. Then we have media files. They can be a content provided by a user. For example, it might be an uploaded avatar for any kind of social media applications. But I like to come back to the interesting thing about the font stuff, like the VOFF2 on the network manager. So what was it? As you can see, we have two different fonts. As you know, websites can have multiple fonts. The first one has the same extension, .v of f2, but the first one is the type of first one is v of 2, but the second one is octat stream. So big news, browser actually do not care about extensions of the files. They are interested in something else, which is basically HTTP headers or HTTP messages. So in HTTP, there is not only the name of the file or the content of the file. Each request or response have some messages attached to it. You can think these like written addresses or attached barcodes to the envelopes when you send to your friend. Okay, let's start with the request messages. This is actually a real request messages from PyCon.de. The first line is actually a special one. The first word which is get refers to HTTP method. It can be get, post, patch, output, delete. You can have all the HTTP methods. There is a slash near to get is the path. In this example, the request goes to the root domain domain. And there is a HTTP2, which states the HTTP specification number of the request. Other lines except the first one are actually the HTTP headers, like host, user agent, accept, accept language. And each of them can be settled by browser or by fetcher or like JavaScript, who makes the request. Actually, these headers are part of a process called content negotiation. As you know, each client or browser has different capabilities. I can use my phone, you can use your Mac, And we can even have terminal-based browsers, too. So all of their capabilities are different. So some of them may support, for example, WebP image format to display. Or some of them may support GZIP compression. Or it can be a social thing, too. We may have an API which supports multi-language. Then clients should get responses in their languages. So we need to have a system for organizing this kind of needs. For this, client sends all the capabilities to the server. Then server itself will decide the optimal ones and sends the response back. As you can see, for example, in here, we have accept, and we have a list of items, like text HTML, application XML, or something like this. And for accept language, I can accept English. But I bought my computer from Turkish, and I have Turkish language settings. So Turkish is added here. Also, accept encoding is added. So I can accept GZIP, deflate, or other things too. Now you can see the response messages or headers of the same request. On previous slide, you saw that we sent a list of items for accept and encoding, then server picked gzip, as you can see here. Because it is the most compressed version, and sent the data in gzip encoding. Also, we have content type, which is text HTML. Where is it? It's here. Also, there is an encoding standard near to this. This is actual value of how to handle file type in the browser. So when we encountered the font issue, actually the content type in the response messages, content type was different, actually. Also, I like to mention some notable HTTP headers. The first one is content type is already described a bit. It is sometimes confused by meme types. But actually, a meme type is a different notation. It is multipurpose internet mail extensions. It is funny, but actually, email is invented before the World Wide Web or HTTP. The meme type usage comes from that age, actually. And most of the case, the meme type doesn't have a chart set value. It just has like image or text HTML. The second one is content disposition. As the value, it can be inline or attachment. Maybe you noticed that sometimes you copy PDF URL and paste to the browser. But sometimes it opens in a browser, but sometimes you have a pop-up and the browser asks you to where to download the file. Actually, this is controlled by this header. Also, we have a lot of cache headers. I don't want to get into the details of this, because it's too much topic for this talk. But we can think that pragma is the old version of cache control. And the first five of them mostly controls the cache's lifetime. But I like to mention about the very header. I think most of time, we have bugs from there. It states that your cache vary on which attributes. It's good practice to have very on accept encoding, accept language, or cookie. Otherwise, once user starts your application in Dutch language, for example, as a default maybe, it will cache, and it couldn't possible to switch it to English, even if you send the accept language header as an English. So it always overwrites this cache. Also, someone may think that I don't want to get performance gain from caching, so I don't need to implement these cache headers. I think it's a bit wrong decision, because if you don't set cache headers, browser itself set this cache address for you. For example, each 200 response is automatically cached by the browser. And if you don't set this like a no cache, it automatically caches in your browser. Then if you make any changes, any user won't be getting these new changes. OK, up to now, we went over the HTTP headers. And we saw that clients or browsers can make lots of requests for loading a single web page. But how does the lifetime of a single request look like? Let's deep dive into this. So most of the monolithic backend applications look like this. We have a n number of nodes. In this case, our nodes run Django with, for example, Unicorn or UVGi. And there is Nginx in front of this. And we have a database connection between our Django. And there is a load balancer in front of our nodes, which distributes the user load from the user load. The interface between Django and Nginx is actually specified in the PEP. PEP 3332-33. Actually, this comes from Perl language. Maybe if you are familiar with the early age of World Wide Web, Perl was very popular for web development. And in this PEP, Python Web Service Gateway Interface is implemented or specified. And we have applications of this, like UVGi or Unicorn and other stuff. I don't want to get the details of this, but Ryan Wilson Perkin has awesome talk about UVGi for web developers. The talk was in the PyCon Canada, I guess. And it really explains all the things about how to implement a UVGi application. OK, let's imagine that user 1 is going to upload a new avatar for itself, because our topic is more about the files. So if, for example, load balancer distribute this user 1's request to node A, and if we store our files in local storage of our nodes, When user 1 make another request, the load balancer, for example, let's say that it goes to node B, but there is no this file on their node B. So for files, we couldn't have, we cannot store our files in the application nodes. If we can do, we can make it by mounting same volumes for this. But it's a bit bad practice. So instead of this, modern projects use object storage. So object storage is like a solution for web backends, well-known ones like AWS S3, Google Cloud Storage, or DigitalOcean Spaces. And they provide you durability, accessibility, too. Also, they have some gains about permissions, like pre-signed URLs. When you put your object to object storage, you can get a pre-signed URL which has lifetime, like 30 days, 30 minutes, or something like this. Also, you can have versioning on your objects in the object storage. Or you can lock the files, then there won't be any updates on your objects, too. But I like to mention that all of them are actually solutions. So while we are making a web application, we shouldn't depend on a single application or single cloud provider. So because of this, we shouldn't use, for example, AWS switch file URLs directly for emails or for any kind of HTML. Instead of, you should implement a routing in our custom domain and point to the related buckets. Also, we can use content delivery network, which is commonly known as CDN. We can think that CDN is like a queue which has a limited space for files. For example, our file lives in this space. But if users like to get the file, it first tries to get from the SuperPOP. And if SuperPOP has this file, it automatically gives the file. But there is no file. It fetched the file from origin and saved the disk. and return to the user. So up to now, I mostly talked about the general perp, general usages. So in Hippo, in my company, we need to create a custom solution for files. In one of our projects, we have critical data, which we are storing in the S3. But we don't like people to share the URL and get the access from the file content. And we created a package, which is basically a Django package, which supports for Django REST framework. It is called yedemin. And I'm going to explain how it works in a single term. For example, a user make a request to API documents, an API request. And in this response, you have list of items, list of URLs of files. But this package creates a key for each file and appends this key to the end of file. For example, in this example, this is file's secret file. And we append the key at the end. And the key is specific for secret file PDF and the current user. If you ask me how I create this key, basically, it is like a Django signing package. We use Django signing package for creating this key. Then we have a basic router view, which lives on the files path. And when you get this route, Django just validates the key for the file and the user. OK, this is the correct user, and this is the right file. Then Django just returns empty response with the Nginx custom response headers. What does this response headers? You can think these like an internal redirects of Nginx. For example, in this value, you can get the pre-assigned URL of the S3 object. Then Django just validates. So it just makes the easy job. And Nginx does the heavy job. And our solutions can be used in the more private object storages. And that's it from my side. If you have any questions, I'd like to answer them.

Speaker 2 [26:48]

I'm not seeing any questions on the slider at the moment. So quickly press them in if you are halfway through writing one. But I have it up here. Maybe folks can also come and find you because we have a coffee break now. Or maybe if you're prepared to be at the side of the stage just for a couple of minutes, folks can come up and ask. I was wondering in the request what role the user agent plays. and can you speak anything to this user agent value you showed in the HTTP request?

Speaker 1 [27:22]

What's the use case of this? What kind of... By the way, user agent can be... You can set anything by... If you like to set user agent to your name, you can do this. So we shouldn't rely on this kind of things, which user can abuse the values of this.

Speaker 2 [27:43]

Yeah, there's no standard, right? Yeah. Okay, so it's a bit of a useless value or a non predictable value. Yeah Yeah, cool. Okay, let me check Slido one more time. I think I've figured it out now.

Speaker 1 [27:56]

Thank you very much for your time.

Speaker 2 [27:59]

Thank you so much for your time, I think.