robots.txt + llms.txt generator
Selectively allow or block AI crawlers, set disallow paths and sitemap, structure llms.txt — fully in your browser.
AI crawlers
Checked = block (Disallow: /). Unchecked = allow; the disallow paths below also apply to allowed AI bots.
When checked, all other crawlers may read the site except the paths below. Unchecked, they get Disallow: / and the whole site is blocked.
Optional. One path per line, starting with /, for example /admin/. These paths stay blocked for allowed crawlers too.
Optional. Full address of your sitemap, for example https://www.example.com/sitemap.xml.
robots.txt
User-agent: * Disallow: /admin/ Disallow: /cart Disallow: /checkout Disallow: /*? User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: Google-Extended User-agent: PerplexityBot User-agent: Perplexity-User User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: Diffbot User-agent: cohere-ai User-agent: Omgilibot User-agent: Timpibot Disallow: / Sitemap: https://www.example.com/sitemap.xml
robots.txt + llms.txt generator — control AI crawlers
This generator builds a matching robots.txt and an llms.txt right in your browser — with no server transfer. For each AI crawler (GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, CCBot, Bytespider …) you choose whether it should be allowed or blocked, optionally add a sitemap reference and describe your key content in the llms.txt. One click allows or blocks all known AI crawlers together. Copy the result as a ready-to-use robots.txt into your domain root and place the llms.txt next to it.
Typical use cases
- Operator wants to block all AI crawlers in one click and needs the ready-made robots.txt to copy
- Editorial team selectively allows only the reading AI crawlers and blocks training crawlers
- Agency creates an llms.txt for a client with the most important pages and descriptions
- Developer needs a syntactically correct robots.txt including a sitemap reference without manual typing
Frequently asked questions
What is the difference between robots.txt and llms.txt?
robots.txt controls which crawlers may fetch which paths — an established standard that reputable crawlers honour. llms.txt is a voluntary proposal that structures your key content for an AI model; it is not an official standard and not a ranking factor, and its effect is disputed.
Do my inputs leave the browser?
No. The generator runs entirely locally in your browser. There is no server transfer, no logging and no storage — suitable also for internal paths or unpublished structures.
Will an AI crawler really respect my robots.txt?
robots.txt is a request, not a technical lock. Reputable crawlers like GPTBot, PerplexityBot or ClaudeBot evaluate it; less reputable actors may ignore it. A full technical block is only achievable via server or firewall rules.
Does an llms.txt give me more visibility in AI answers?
This is not proven. llms.txt is not a ranking factor, its effect is currently disputed and not all providers evaluate the file. It does no harm, but we promise no measurable benefit.
Where do I place the generated files?
Both belong in the root of your domain: robots.txt at https://your-domain.invalid/robots.txt and llms.txt at https://your-domain.invalid/llms.txt. You can then verify the result with our AI readiness check.
Can I allow some crawlers and block others?
Yes. You set allow or block per AI crawler individually — for example allow the reading search crawlers and exclude pure training crawlers. The generator produces the matching user-agent blocks.
Related tools
Continuous monitoring vs. one-shot?
XICTRAQ monitors uptime, SSL, security headers, GDPR and performance 24/7. Free plan, no credit card.