robots.txt Generator
Build a robots.txt: allow or block crawlers, add per-bot rules for search engines and dataset crawlers like GPTBot and CCBot, crawl-delay and sitemaps. RFC 9309.
Place this file at the root of your site (https://yourdomain/robots.txt). It follows RFC 9309. Note that Google ignores Crawl-delay; set the crawl rate in Search Console instead. robots.txt is a request, not a security control — it does not stop access to a page.
A robots.txt file tells web crawlers which parts of your site they may request. This generator builds a valid file following the Robots Exclusion Protocol (RFC 9309): set a default allow or block policy, block particular search engines or the crawlers that collect pages for datasets and chat assistants, and add a crawl-delay and your sitemap URLs. It runs in your browser.
How to use this tool
- Choose whether to allow or block everything by default, or set custom paths.
- Optionally block specific search engines, dataset-collection crawlers or chat-assistant crawlers.
- Add a crawl-delay and your sitemap URLs.
- Copy the file to your site root as /robots.txt.
Formula
Each group starts with a User-agent line followed by Allow and Disallow rules. "Disallow: /" blocks everything; an empty "Disallow:" allows everything. Sitemap lines are global and use absolute URLs.
Example
A block-everything policy produces exactly:
**User-agent: *
Disallow: /**
which tells every crawler to stay out of the whole site.
Frequently asked questions
Does robots.txt keep a page private?
No. It only asks well-behaved crawlers not to fetch a path; it does not stop access and the URL may still appear in results if linked elsewhere. Use authentication or noindex for real privacy.
Why does Google ignore Crawl-delay?
Googlebot does not support the Crawl-delay directive. Set your crawl rate in Google Search Console instead. Bing and some others do honour it.
How do I keep my pages out of training datasets?
Turn on the dataset option to add Disallow groups for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and similar crawlers. The separate assistant option also blocks crawlers such as OAI-SearchBot and PerplexityBot, which may remove your pages from those services’ answers. Obeying robots.txt is voluntary, so it is a request, not a lock.
Sources
Last updated:
Page address with these inputs: https://kakao.mobile/tools/robots-txt-generator/
Cite or embed this tool
kakaoCalc. (2026, November 8). robots.txt Generator. https://kakao.mobile/tools/robots-txt-generator/
“robots.txt Generator.” kakaoCalc, 8 Nov. 2026, kakao.mobile/tools/robots-txt-generator/.
Paste this code into your page. The widget runs in your visitors' browsers, shows no ads and keeps a link back to this page. Embedding terms