AI Crawler robots.txt Generator
Every major AI vendor runs at least two separate crawlers under different user-agent names: one that trains models on your content, and one that fetches pages live to answer or cite them in a specific AI response. Pick which to allow per vendor and get a robots.txt block to paste in — most generic robots.txt tools lump both bot types together as "AI bots."
Generated robots.txt block — append this to your existing robots.txt (don't replace your User-agent: * rules):
What's the difference between an AI training crawler and an AI citation crawler?
A training crawler (like GPTBot, ClaudeBot,
or Google-Extended) crawls broadly and stores content
to help build or improve an AI model. A citation/answer-engine
crawler (like OAI-SearchBot, Claude-SearchBot,
or PerplexityBot) fetches pages live, at the moment
someone asks a question, specifically to answer or cite that page
in an AI response. Vendors run these as separate bots with
separate user-agent names specifically so site owners can allow
one without allowing the other.
Can I let AI answer engines cite my site without letting AI companies train on it?
Yes — that's exactly what this tool generates. Block each
vendor's training bot (GPTBot, ClaudeBot,
Google-Extended, etc.) while leaving their
citation/search bot (OAI-SearchBot,
Claude-SearchBot, PerplexityBot)
allowed, and you'll still be eligible to show up in AI answers
and citations without your pages being used as training data.
Does this tool send my robots.txt anywhere or check my site?
No. It only generates text based on the checkboxes you select — nothing about your site or domain is fetched, uploaded, or stored. Paste the output directly into your own robots.txt file.