In the second part of our video series, “SEO For Beginners”, we talk about how search engines like Google crawl, index, and rank websites.
Timestamps:
00:00 Introduction
00:28 How Search Engines Work
1:01 Crawling
2:58 Indexing
4:22 Ranking
5:52 Important Ranking Factors
This can help you optimize your website according to Google’s best practices and improve your visibility in search results.
Crawling
Crawling refers to the discovery of all the pages on the web. Google has little programs called GoogleBots which are also known as crawlers or spiders. Crawlers start off by discovering only a few web pages and then follow the links on those webpages to find new pages, and so on and so forth. This way they discover billions of webpages on the internet to add them to Google’s database.
With a Robots.txt file in a website’s root directory, webmasters can give directives to crawlers about which parts of their site the crawlers should and shouldn't crawl or index. For webpages that don’t need to be crawled or indexed, like test pages, a Disallow directive can be used. You can learn more about Robots.txt here:
https://www.linkbuildinghq.com/beginners-guide-robots-txt/ A Crawl Stats report in the Google Search Console shows how many pages of a website have been crawled in the last 90 days. The Coverage report shows if crawlers are encountering any errors while crawling a website.
Crawlers have a Crawl Budget which refers to the average number of URLs Googlebot will crawl on a site before leaving. Therefore, it is important to optimize the Robots.txt file to ensure Google isn’t crawling unimportant or junk pages. With a sitemap, GoogleBots make sense of the site architecture and understand which pages the webmasters think are more important.
Indexing
While crawling is the discovery of pages, indexing handles the storing of the discovered pages in Google’s database or index.
By typing in “site:yourdomain.com” in the search bar, you can know which pages of your website have currently been indexed. There can be a few reasons for webpages not being indexed which are discussed in this video.
Some common Robots.txt instructions for indexing include:
• A NoIndex tag which can be used to tell Google which pages to crawl but not index.
• A NoArchive tag to remove pages with outdated pricing. This is useful for ecommerce sites.
• An X-Robots-Tag to exclude specific folders or file types from being indexed.
Ranking
Ranking refers to showing webpages in a certain order based on a search query. Google uses hundreds of mini-algorithms, or ranking factors, to assess where and when to show pages in its search results.
These mini-algorithms add up to form Google’s ranking algorithm which keeps updating to improve Google’s ability to show the most relevant and high on quality search results.
Depending on the search intent, the searcher’s location, their search history, and other metrics, Google displays what it thinks are the most relevant results. Google’s ranking factors help the search engine in this process. There are over 200 ranking factors, but the most important ones include:
• Content Relevance
Relevance is among the most important ranking factors. Your content including your keywords needs to be relevant to the searcher’s query. Conducting keyword research will help you find the right keywords to target.
• Content Quality
The quality of the content helps you stand out from your competition. Assessing the content in terms of its EAT (Expertise, Authority, and Trustworthiness) helps you with this factor.
• Quality Backlinks
Backlinks act like “votes” from one site to another. Having backlinks from high authority websites improves this ranking for this factor.
• Mobile-Friendly Websites
With mobile-first indexing, your website’s mobile version is now even more important than the desktop version. Therefore, ensure your website is fully optimized for mobile. If you want to learn more about mobile-first indexing, read this blog:
https://www.linkbuildinghq.com/mobile-first-indexing-guide/ • Technical On-Page SEO
This includes having the right meta tags and headings, making sure your page can be crawled and indexed if you want it to rank, making it easier for search crawlers to understand your page, having a fast load time, using alt text for images, internal linking, having a secure site, deploying schema markup, and more.
Now that you have a solid grasp of how Google crawls, indexes, and ranks websites, it’s time to move on to developing a winning SEO strategy. And it all starts with a solid keyword research plan, which is the main topic of part 3 of this series. Stay tuned!
Link Building HQ
At Link Building HQ, our goal is to provide the most relevant and up-to-date information to search engine marketers.