Robots.txt Generator
Generate robots.txt files to control how search engines crawl your website. Add allow, disallow, and crawl-delay rules.
Advertisement
Ad
Rule
robots.txt
User-agent: * Disallow: /admin/
Advertisement
Ad
Frequently Asked Questions
What is robots.txt?
robots.txt is a text file that tells search engine crawlers which pages or files they can or cannot request from your site. It is placed in the root directory of your website.
Does robots.txt block pages from search results?
Not directly. robots.txt only tells crawlers not to visit pages. For true blocking from search results, use a noindex meta tag or password protection.
Where do I place the robots.txt file?
Place it in the root directory of your website at /robots.txt. It must be accessible at example.com/robots.txt to be found by crawlers.