Your content cannot rank on search engine results pages (SERPs) if it is not included in Google’s index.
Think of the internet as a library and Google is the librarian. When you aren’t sure where to find an answer to your question, you normally would ask the librarian for help.
But before the librarian can help people with answers, they need to find books, understand them, and index or categorise them in terms of what they are about.
So when Google indexes your web pages it is to gather data about them and categorise them. This helps Google to define what queries or searches your webpages could potentially answer.
The following is how indexing works according to Google.
| A page is indexed by Google if it has been visited by the Google crawler (“Googlebot”), analysed for content and meaning, and stored in the Google index. Indexed pages can be shown in Google Search results (if they follow Google’s webmaster guidelines). While most pages are crawled before indexing, Google may also index pages without access to their content (for example, if a page is blocked by a robots.txt directive). |
And of course, with almost 227 million Google searches happening in an hour and about 5.4 billion per day, you don’t want to miss the opportunity of your web content being indexed and shown to people.
Especially if you’re selling something.
So as an eCommerce owner or marketer, you need to ensure that Google is indexing your content. If not, you’re missing out on potential revenue driven by organic or search traffic.
And yes, there are instances where Google isn’t able to index a page. But you can fix that when it happens to you.
In this section of the KAMG SEO content hub, we will guide you through what you can do to ensure that Google effectively indexes your eCommerce site content.
Crucial Terms You Need to Know
Before we dive in, there are a few words that you have to properly define.
Crawling
Crawling is Google’s process of finding new content. It is mostly done by following hyperlinks.
Web Spider
Googlebot is Google’s branded web spider or crawler. This software carries out the crawling process.
Indexing
Indexing is Google’s process of storing web page contents into a vast categorised database.
One vital fact to remember is that indexing is not ranking.
Indexing is showing up in the competition while ranking is winning it.
When Google indexes your site, it knows what your pages are about and can show your content to answer relevant queries. But Google will ensure to show first content that is highly relevant and authoritative, whether it is from you or someone else.
So if your content is poor, it won’t rank highly on SERPs.
But of course, you can’t win if you don’t show up. So you need to ensure indexing happens.
How to Know if Your Website is Indexed
Google, in essence, should be able to index all internet content. So you probably don’t have to worry about not showing up on SERPs.
But it never hurts to double check.
Go to Google and type in and search site:yourwebsite.com.
The results number shows an approximation of how many of your pages have been indexed by Google.
You can use the same step to check for the indexing of a certain web page as shown in the example image below.
If there’s at least one result, then your page is indexed.
You can also get a more accurate insight using the Google Search Console’s Coverage report.
Simply go to Google Search Console > Index > Coverage.
Put your attention to the number of valid pages, with warnings and without.
If the total of these two numbers is anything greater than zero, then Google has indexed at least some of your website pages.
Otherwise, you can definitely say that you have a severe problem because not one of your site pages have been indexed.
You can also use the URL Inspection tool of the Search Console to check if a certain web page is indexed.
If the page you’re checking is indexed, the tool will display “URL is on Google” like in the image below.
How to Ensure That Your Pages are Indexed
If you have found out that one or more of your web pages aren’t indexed by Google, you may try fixing the problem by doing the following:
- Ensure that your page follows Google’s general guidelines.
- Go to Google Search Console
- Open the URL inspection tool
- Paste the URL of the page you’d like to index into the search bar.
- Wait for Google to finish checking the URL
- Click the “Request indexing” button
When you have a newly published post or page, you should follow the above steps. By doing so, you effectively tell Google that you’ve added something new to your site and that they should check it out.
Still, there might be underlying issues that disallow Google from indexing your page, most especially old ones.
Also Google can take anywhere from a few days to a few weeks to index a website.
Fortunately, there are steps you can take to improve indexing efficiency.
To assess and solve the problem, simply go through the following steps.
Other Things You Can do to Ensure Indexing
Optimise Your robot.txt File
Robots.txt files are recognised by Googlebot as an indication that a web page should not be crawled and thus not indexed. Robots.txt is also recognized by Bing and Yahoo! search engine spiders.
You would use robots.txt files to assist crawlers in prioritising more important pages so that your own site is not overburdened with requests.
Although this may appear to be a bit technical, it all boils down to making sure your page is crawlable, which you can check with SemRush’s On Page SEO Checker. It provides optimisation feedback, including technical edits such as whether a page is crawlable as shown in the image below.
The tool is intuitive and will give you information on how to fix issues when they arise.
Ensure That Your SEO Tags Are Free of Errors
SEO tags are another method for directing search engine spiders such as Googlebot.
There are two types of SEO tags that should be optimised.
- Rogue Noindex Tags
These tags instruct search engines not to index pages. If certain pages aren’t being indexed, it’s possible that they have noindex tags.
Look for the following two types:
- Meta Tags
Meta elements are HTML and XHTML tags that provide structured metadata about a Web page.
Look for noindex page warnings to see which pages on your website may have noindex meta tags.
To index a page that has been marked as noindex, simply remove the meta tag.
- X-Robots Tag
The X-Robots-Tag can be used as an HTTP header response element for a given URL. Any directive that can be specified as an X-Robots-Tag can also be specified as a robots meta tag.
To find out which pages have an X-Robots-Tag in their HTML header, use Google’s Search Console’s URL Inspection Tool.
Look for the response to “Indexing allowed?” after entering a page.
If you see the words No: ‘noindex’ detected in ‘XRobots-Tag’ http header, you know you need to remove an X-Robots-Tag.
- Canonical Tags
Canonical tags inform crawlers whether a particular version of a page is favoured.
If a page lacks a canonical tag, Googlebot acknowledges it as the preferred and only variant of that page, and will index it.
If a page has a canonical tag, Googlebot supposes there is another preferred version of that page and will not index it, even if that other variant does not exist.
To inspect for canonical tags, use Google’s URL Inspection Tool. You’ll see a warning that says Alternate page with canonical tag in this case.
Optimise Your Entire Site’s Architecture
To see to it that Google crawls and indexes your website and its pages more efficiently, you have to ensure seamless internal linking and excellent backlinking.
Internal linking assists crawlers in finding your web pages. Non-linked pages are referred to as “orphan pages,” and they are rarely indexed.
Proper internal linking is ensured by proper site architecture, as laid out in a sitemap.
Your XML sitemap organises all of the content on your website, helping to identify non-linked pages.
Here are a few more best practices for internal linking:
- Remove nofollow internal links.
When Googlebot encounters nofollow tags, it alerts Google that the tagged target link should be removed from its index.
Nofollow tags should be removed from links.
- Include high-ranking internal links.
Spiders discover fresh content by crawling your website, as previously stated. And internal links help to speed up the process.
Streamline indexing by internally linking to new pages from high-ranking pages.
- Produce high-quality backlinks.
Google recognises that pages are relevant and important if authority sites consistently link to them.
Backlinks alert Google to the fact that a page should be indexed.
Make High-Quality Content a Priority
All indexing and ranking rely heavily on high-quality content. Consider removing poor and underperforming pages from your website to ensure its content is high-performing.
This enables Googlebot to focus on more important pages on your website, allowing you to make better use of your “crawl budget.”
Additionally, you want each page on your site to be valuable to users. Your content should be one-of-a-kind as Google Analytics may otherwise flag it as duplicate content.
You need to get indexed if you are to rank.
So clearly, you need to ensure that your website and its pages are indexed by Google if it is going to rank at all on SERPs.
Ranking higher means that more people will see your content and probably go to your eCommerce site to do business with you. This can be good for your revenue.
And so, there’s a lot more to do after ensuring indexing. To rank higher on SERPs, you need to optimise your content and web pages.
And that’s what you will learn in the next sections of this KAMG SEO content hub.
