Robots TXT is one of the smallest files on your WordPress site - but it can have massive SEO consequences.
In this video, you’ll learn what Robots TXT actually does, how to edit it safely in WordPress, and the 14 most common Robots TXT mistakes that can quietly tank your rankings (or block your entire site).
Also learn how to test your rules, recover fast if something’s already broken, and how WP Rocket helps bots and visitors load your pages faster — without risky Robots TXT hacks.
Video Chapters:
01:09 What does Robots TXT do?
01:33 Robots TXT Directives You Should Know
02:12 How to Edit Robots TXT in WordPress (Rank Math)
04:01 Do You Need Robots TXT?
04:18 Benefits of Robots TXT
04:46 Mistake #1: “Discourage Search Engines” Still Enabled
05:01 Mistake #2: Blocking CSS or JavaScript
06:28 Mistake #3: Robots TXT in the Wrong Location
06:45 Mistake #4: Outdated Directives (noindex, crawl-delay)
07:32 Mistake #5: Copying Staging Rules to Production
07:47: Mistake #6: Forgetting Your Sitemap Line
08:14: Mistake #7: Conflicting Rules
08:52 Mistake #8: Using Robots TXT as “Security”
09:33 Mistake #9: Misusing Wildcards (* and $)
10:01 Mistake #10: Absolute URLs in Disallow
10:25 Mistake #11: Case Sensitivity Problems
10:40 Mistake #12: Trailing Slash Confusion
11:02 Mistake #13: Forgetting Subdomains
11:25 Mistake #14: Never Testing Changes
11:53 How to Recover From Robots TXT Mistakes
13:14: Robots TXT + Performance: Why Speed is So Important
Robots TXT Examples Mentioned in the Video:
• A solid default Robots TXT for many WordPress sites
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
• Blocking your entire site (don’t do this on production)
User-agent: *
Disallow: /
• Conflicting rules (unpredictable crawling behavior)
User-agent: *
Disallow: /blog/
Allow: /blog/
• Blocking PDF files (use carefully)
Disallow: /*.pdf$
• Use relative paths in Disallow rules
Correct:
Disallow: /private/
Incorrect:
Tools & Plugins Mentioned:
Robots TXT Editing Plugin (Demo):
Performance Plugin:
Crawling + Testing Tool:
Subscribe here to learn more about web performance and WP Rocket:
Connect with WP Rocket:
WP Rocket is the all-in-one performance plugin trusted by over 5 million websites. With features like page caching (including mobile caching), cache preloading, GZIP compression, and file optimization options like minification, removing unused CSS, deferring JavaScript, and delaying JavaScript execution, WP Rocket helps your pages load fast for both visitors and crawlers.
Ready to improve crawl efficiency and load times? Boost your site speed with WP Rocket today! 🚀