About the Robots.txt Generator
The Robots.txt Generator creates a valid robots.txt file to control how search engine crawlers access your site. Allow or disallow specific paths, set crawl rules per user-agent, and add your sitemap location. A correct robots.txt prevents crawlers from wasting budget on pages you don’t want indexed.
How to use the Robots.txt Generator
- Choose to allow or disallow crawlers and add any disallowed paths.
- Add your sitemap URL.
- Copy the generated robots.txt to your site root.
Frequently asked questions
Where does robots.txt go?
It must live at the root of your domain, e.g. https://example.com/robots.txt.
Does disallow remove a page from Google?
No — disallow only blocks crawling. To remove a page from the index, use a noindex meta tag.