#robotstxt #onescales
This video tutorial teaches you about robots.txt:
Why you need it, What's it purpose and shows you a testing tool with real examples to understand it better.
We discuss the following topics:
1. What is Robots.Txt
2. Guidelines and How to Write
3. Terms
- User Agent
- Allow
- Disallow
- Sitemap
- Comments
- Crawl-Delay
- Wildcards
4. Limitations
5. Testing Tool