HTTP headers are a topic that can improve the fundamental understanding of your technical SEO. It can improve the technical SEO of your website. By using HTTP headers, you can reduce the size of your website. In this article, we will learn simple ways to understand and implement HTTP headers.
What are HTTP headers?
When you type a website URL in your browser and press the Enter button, your browser sends a request to the server of that website and in return for this request, the website server sends a reply to your browser. So when your browser sends a request to the server, there are headers in it. And when the server replies to your browser, there are headers in it. So every page visit has more than two headers. That’s why we call them HTTP headers.
These headers contain a lot of information, like the URL of the web page, how long the content is, the method of fetching, status code, encoding, cache, and a lot of advanced things. Let’s see how these headers can be seen and how they are displayed. You have opened the page, but how will you see the headers? Let’s go to the browser.
- First, let’s open the developer tool.
- If you are on Mac, press the command option I. If you are on Windows, there is a very simple way, F12.
- Let’s open the network tab.
- Now you have to type your URL in the address bar.
- For the demo purpose, we will use dailytechbyte.co.in and press the enter button.
- You have to wait for the page to load. Once the page is loaded in this list, scroll to the top and click on the first entry.
- Now you will see headers in the pop-up on the side.
- Now this list is very long, so let’s manage it a little. Click on these arrows and twirl up. And now you will see three major parts. General, response headers, and request headers.
So request headers are those that your browser has sent to the server, and response headers are those that the server has sent to your browser. Request, response. It’s important to understand the difference. First, let’s look at the request header. This will tell us what your browser has said to the server behind your website.
One thing to remember is that the entries you see in the headers can be different for different websites. Maybe you will see this article after a few months. And even on dailytechbyte.co.in, if you see different entries, don’t panic.
So if we look at all these entries carefully, we find out which site our browser has tried to visit. What is the path? What is the scheme? That is, it has tried to visit this page on HTTP or HTTPS. What type of file can the browser accept? Here the browser is ready to accept HTML, XML, images, any type of image, video and any format of video..
Which type of encoding or compression can it accept, and which language can it accept? It is also telling user agent. Our browser is telling the website’s server which user agent it is. So next time, if the boss in the office tells you to impress that Google collects so many things about your website’s visit, device, model, browser, etc., then tell him that Google is not tracking it. Your browser is sending all this information by itself.
We start giving credit to Google for everything. Google sometimes just collects information and we send it ourselves. Anyway, these are the things that your browser is sending to the website’s server. But for SEOs, the most useful thing is what the website’s server is sending to the browser.
So let’s go to the response headers and see what the server is saying to the browser. The first thing the server gives is the type of compression. This is very important because the browser is ready to handle that type of compression.
What is Compression?
There are a lot of files, compress them and send them in small size. Remember, compression affects the speed of the website. So if this compression information doesn’t send the server of the website to your browser or sends it wrong, then it has a bad impact on the website’s performance. And this has happened in a lot of real-world cases where the website’s server is sending the wrong information about compression.
Apart from this, what type of content will this URL provide? HTML, PDF, and image, we get all this information. Yes, this is true. We are looking at the headers of the HTML page, but if you click on the image link, you will see the header of the image, PDF, video, and all types of files. CSSJS has its headers for all types of files. The content of this web page is cached. What is the policy of caching? Has the web page changed recently or not? The server of the website, Which is it?.
All this information is automatically available to the browser in the headers. Apart from all this, the status code is basic but very important information in the headers. The status code only tells you that a page has been loaded successfully; that is, it gives a 200 status code, a redirect of 301/302, or a client error of 404/410.
All this information You can find it in the topmost general section. You don’t have to go here and there. You can also find the IP address of the server in this general section.
So far we have seen the basic headers. Now let’s look at some advanced headers and their use cases. If we scroll down the response headers, we can see a header named Vary at the bottom.
Vary header is a very useful header that can be used in different ways on big corporate sites and e-commerce websites. Its use cases can be very creative.
Let me give you an example. Your website has a lot of features and functions. And your website’s loading speed is not fast at all. It may take a long time for Googlebot to load your website, or it is a JavaScript-based website, and you don’t want to use server-side rendering for all users. It could be for any reason. Budget is a problem. The technical stack may be a problem. But you don’t want to give Google a JavaScript-heavy website because that could confuse Googlebot. So developers and SEOs develop a simple solution to serve all human users on a normal website.
When a request comes from Googlebot, we show it a static page that is already rendered, which can be loaded quickly, Googlebot can digest it easily. This solution sounds easy to hear, but it has a problem. The problem is that Google’s systems can consider it as cloaking. That is, they may feel that you are showing different content to different users. You are showing different content to humans and different content to Googlebot and this can create confusion. Serving different types of content
It’s called cloaking and it can be considered as a spam policy. Google can punish your website. So, in such a situation, we use the vary header.
The headers that we have covered so far are in this website’s header. But let’s see what else we can add to these headers from our side. Because it’s not just that. We can add a lot of things from our side. Are you ready? The list is quite exciting. read carefully.
If you want to noindex a specific URL, you can use HTTP headers to noindex it. If you want to add a nofollow with noindex, you can use HTTP headers. If you want to noindex an image of a specific page, you can use noimageindex in HTTP headers.
You can serve all tags like no-snippet, max-snippet, max-image-preview, max-video-preview, no-translate, unavailable-after through HTTP headers. In fact, not only headers, but canonical tags can also be served through headers. It will look something like this. By the way, can we serve any other tag besides canonical through HTTP headers?
As I said, the list is very long. You can also implement the hreflang tag through HTTP headers. It will look something like this. If your website uses AMP, you can also serve the amp-html tag through HTTP headers. This code will also look something like this. And if you want to inform Google about the feed URL of a page, of course, you can use this code to tell Googlebot about the feed version of your URL through HTTP headers.
Now, if you are thinking that after all, I can edit all this information in the website and put it in the head section of a normal web page, then what is the need to learn one more trick? Why should I get involved in HTTP headers? Why should I get involved in development coding? Well, there is a need. There are many such scenarios where it is useful to give all this information through HTTP headers and it is easier to implement HTTP headers, to be honest.
Reasons for using HTTP headers
1. Convenience
Well, maybe in many cases, the head part of your website cannot be edited. This can happen for many reasons. Maybe the organization is big where it is not possible to change anything so quickly with the developers. Maybe the website’s infrastructure is old or complicated where no information can be added by editing in the head. Maybe there is a tech stack where the head has to be
It can’t be changed for any specific page. Client is stubborn, developer is rude, anything can happen. But you have to do your work. So without touching the page, you can send all this information through the server, through HTTP headers. You don’t have to change the website header at all. All your work will be done through the header. Now, how to do it? Let’s come to this. We will cover the execution part very soon. It is much easier.
2. Non-HTML pages.
This means the URLs you want to put this tag on, they don’t have a head section. Don’t get confused. There are a lot of content types in which there is no header. For example, there is no header in the image.
Have you ever seen a head section in an image? No? Have you ever seen a head in a PDF? It’s not in a doc file. Google indexes, ranks, and gets traffic from all these files. So, if your website has a lot of PDF files, like a test series, study material, journal publication, documentation, report, websites where there are a lot of PDF documents or doc files,
In all these websites, you have to put a canonical or hreflang tag on a similar pdf or doc file. And how do you do all this? Because these are not head sections or html files. So, you can use HTTP headers to do SEO on this kind of websites. Because through HTTP headers, you can put a canonical, hreflang or noindex on any type of file. You can do anything or Google Search Console.
3. Crawl budget.
Just imagine that one page of your website is noindex. Now, if this noindex tag is in the header of the page, then Googlebot will have to download the entire HTML first. Along with that, it will download CSS, JavaScript, etc., everything. And after that, it will find out that this page does not have to be indexed. So, all the hard work is wasted.
Right? But if you send this information through HTTP headers, then Googlebot will know before downloading the content that it doesn’t want to index it. And Googlebot won’t smartly crawl your entire page. It won’t download it. If it doesn’t download it, then your server’s labor will be saved. Your bandwidth will be saved, your CPU will be saved, etc. Whatever your hosting is charging you on, and your crawl budget will be saved.
4. Speed.
Less code in HTML means less time. Your competitor will copy your page and you will be able to play through headers. So you can increase the speed of the page through HTTP headers. Now the question is, if there are so many benefits and types, then how do I implement it? Because no one wants to develop it. We are not developers or coders.
The good thing is that the way to implement it is very simple. You don’t have to do anything in your website. Where do you have to implement all these things? In your website’s .htaccess file. If your server doesn’t support .htaccess file, then there is httpcon file or config file. Whatever kind of server it is, whatever configuration file is used instead of htaccess, you can implement these codes there and the way to make these codes is very simple. You can use ChatGPT. It’s free.
You don’t need to sign up or log in. You can go to chatgpt.com and generate this code according to your requirements. For example, let’s say you want to noindex a XYZ URL and you want to serve this noindex code through HTTP header, then you have to go to chatgpt.com and say.
“This is my URL. It’s example.com and create HTTP header code for noindex this page”.
You can make rules for individual URLs, you can make rules for individual categories. You can generate code by giving a bunch of URLs. You don’t have to do anything.
If you want to test, copy the old htaccess file and save it. And make all the changes in the new htaccess file. If there is a break, then you only have to replace the htaccess file. You already have hosting access. So don’t touch your website, don’t log in, there is wp-admin, there is javascript file, there is any type of Shopify, Drupal, anything. It doesn’t matter. As long as you have hosting access.
