Two lines in robots.txt quietly removed an entire online store from search results — and no monitoring system caught it, because technically nothing was broken.
What's covered:
— What robots.txt actually does (and why it's not a security tool)
— The biggest misconception: Disallow ≠removing pages from search
— How Disallow and noindex conflict with each other
— Syntax breakdown: wildcards, Allow rules, and how conflicts resolve
— Yandex-specific directives (Clean-param, Crawl-delay)
— The most common robots.txt mistakes during migrations
— How AI crawlers (GPTBot, ClaudeBot, CCBot) use robots.txt
— The new llms.txt format and what it means for AI agents
— How to check your own robots.txt in Search Console and Yandex Webmaster
If you've ever migrated a site or wondered why traffic suddenly dropped, this is the file to check first.
#robotstxt #SEO #TechnicalSEO #SearchEngineOptimization #WebDev #SiteMigration #GoogleSearchConsole #Noindex #CrawlBudget #AICrawlers
Created with an AI avatar. Visuals are illustrative. The content is for informational purposes.