How WordPress 5.3 will change Google Indexing

How WordPress 5.3 will change Google Indexing

Posted 17th September, 2019 by Nadejda

Google indexing

WordPress has just announced a significant change concerning the way WordPress is going to handle Google bots blocking and website indexing in general. A new approach will be used in the new 5.3 release – Robots Meta Tag will be replacing Robots.txt.

How-Google-indexes-your-website-using-Googlebot



How does it work?

There is currently a certain setting in WordPress - discourage search engines from indexing (found in Settings→Reading), using the robots.txt and adding a Disallow line in the code. This prevents search engines from crawling the site and in turn indexing it in Google.

It’s a standard practice using Robots.txt in order to block the indexing of a website.

But what exactly does indexing mean? The crawling of a certain site by Google Bot is what indexing consists of. It is how Google knows about your website and shows it on its search results. You can stop Google downloading a webpage using Robots.txt and it’s blocking mechanism. This should stop Google from showing your pages in its Search results.

However, the robots.txt mechanism only stops Google from crawling a certain page and it could still add the rest of your site it to its index if it could discover any URL of the site, which makes this option a bit inefficient.


WordPress 5.3 will indeed prevent Google Indexing

WordPress 5.3 will abandon the Robots.txt approach, meaning that when a user chooses the option " discourage search engines from indexing this site” it will prohibit Google from crawling this site.

With the new WordPress 5.3 the far more reliable Robots Meta Tag approach will be adopted in order to prevent indexing of a website. The change is a great improvement and will assure users that your blocked pages will not be shown in Google Search.


Why stop your site from being indexed?

A lot of people may start to wonder – why would I stop my site from being visible on Google, isn’t this the main purpose of writing content, creating new designs and making it user friendly for visitors?

The answer is yes and no. If you have a clone of your website – a beta version or staging site where you first make the changes and you later implement in your live website, then you certainly do not want that content for be visible. It would make the Search results on Google look strange or duplicate your content, which in turn leads to penalty from Google.


Robots.txt

Using Robots.txt for blocking pages from Google was the way to do it until now. However, despite of its popularity, this method was not very reliable. As of September 2019, Google no longer supports the robots.txt which prompted for the change in WordPress.

Robots.txt was never “official”, despite being honored by most search engines, but as things have evolved, it’s no longer fit for this purpose.



Robots.txt Vs. Robots Meta Tag

Robots-meta-FI-600x314


The idea behind Robots.txt was to keep a page out of Google’s Index, this is now Robots Meta Tag’s job. The Robots Meta Tag is a specific tag which tells the Google Search engines which pages and links to follow on your website and which not to follow. Keep in mind that when you have a lot of links going out of your site this lowers your page rank. Keep some of the Google juice with you by preserving some links.

The Meta Robots Tag has 4 main functions:

Follow the command tells the search engine crawler to follow the links in the webpage.

Indexthis command tells the search engine crawler to index the webpage.

Nofollow – the command is used NOT to follow the links in the webpage.

Noindex– the command tell the crawler NOT to index that specific webpage.

If you are relying solely on robots.txt to prevent indexing certain pages from your website now is the time to take action! WordPress will handle this for you but make sure to contact our Fixed team should you encounter any issues!



Categories: Wordpress, Optimisation, Monitoring, Business