#seo #robotstxt #aem
Configure robots.txt file in AEM
Robots.txt is a text file webmasters create to instruct web robots on how to crawl pages on their website. The robots.txt file is part of the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
In practice, robots.txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website. These crawl instructions are specified by “disallowing” or “allowing” the behavior of certain (or all) user agents.
The robots.txt file must be placed in a website’s top-level directory.
Some user agents (robots) may choose to ignore your robots.txt file. This is especially common with crawlers like malware robots or email address scrapers.
Each subdomain on a root domain uses separate robots.txt files. This means that both blog.example.com and example.com should have their own robots.txt files
t’s generally a best practice to indicate the location of any sitemaps associated with this domain at the bottom of the robots.txt file
Robots.txt files control crawler access to certain areas of your site
Let see the robots.txt elements
User-agent: The specific web crawler to which you’re giving crawl instructions
Disallow - The command used to tell a user-agent not to crawl a particular URL. Only one "Disallow:" line is allowed for each URL.
Allow (Only applicable for Googlebot): The command to tell Googlebot it can access a page or subfolder even though its parent page or subfolder may be disallowed.
Crawl-delay: How many seconds a crawler should wait before loading and crawling page content. Note that Googlebot does not acknowledge this command, but the crawl rate can be set in Google Search Console.
Sitemap: Used to call out the location of any XML sitemap(s) associated with this URL. Note this command is only supported by Google, Ask, Bing, and Yahoo.
Some common use of Robots.txt
Preventing duplicate content from appearing in Search Result Pages
Keeping entire sections of a website private
Keeping internal search results pages from showing up on a public SERP
Preventing search engines from indexing certain files on your website
Specifying a crawl delay in order to prevent your servers from being overloaded when crawlers load multiple pieces of content at once (in text document)
If there are no areas on your site to which you want to control user-agent access, you may not need a robots.txt file at all.
Please note robots.txt will not guarantee the pages will get excluded from the indexing, the pages are indexed even if disallowed in robots.txt in case the pages are linked from other sites . in that case use meta robots tag to specify noindex for the required pages.
Let see now how to enable robots.txt in AEM
As a simple step you can just create a robots.txt file in the root path of the website and enable the required rules, this is simple but managing the file for the larger number of domains will be challenging.
In case of any common changes to the robots.txt file, all the individual website robots.txt file should be modified individually to reflect the changes
This can be avoided by creating global and site-specific robots.txt files and serving the actual file through a servlet.