A request, not a lock
Well-behaved crawlers follow robots.txt, but it does not stop anyone from opening a page. Protect private pages with a password instead.
FOR WEB DEVELOPERS
Code stays on your device
Robots.txt Generator
Create a robots.txt file with blocked paths, a sitemap link and rules for AI crawlers such as GPTBot, ClaudeBot and CCBot.
1 Rules
Processed in your browser. Nothing is uploaded.
THREE SMALL STEPS
How to create a robots.txt file
-
01
Set the default rule
Allow crawlers on the whole site except chosen paths, or block the site entirely, for example on a staging server.
-
02
Add paths and your sitemap
List folders to block, one per line, add exceptions and paste the full address of your XML sitemap.
-
03
Choose AI crawlers to block
Tick the AI crawlers you do not want collecting your pages, then copy the file or download robots.txt.
Good to know
Blocked pages can still be indexed
A blocked URL can appear in search results when other sites link to it. To keep a page out of Google, allow crawling and add a noindex tag.
Crawl-delay support varies
Bing and some other crawlers respect Crawl-delay, but Google ignores it.
QUESTIONS
Frequently asked questions
Does robots.txt keep a private page out of search results?
Not reliably. Robots.txt controls crawling, not access to a page, and a blocked URL can still be discovered through links. Protect private content with authentication. For public pages that should not be indexed, use an appropriate noindex directive rather than treating robots.txt as a privacy control.
Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended controls whether your content is used for Gemini models. Google Search still crawls your site with Googlebot.
Where do I put the robots.txt file?
At the root of the host, such as https://example.com/robots.txt. Each subdomain needs its own file.
How do I allow one file inside a blocked folder?
Add the folder to the blocked paths and the file to the allowed paths. Google and Bing apply the most specific matching rule.