Tool
robots.txt agent checker
Paste a URL. We fetch its /robots.txt, parse it, and tell you exactly which AI agents are allowed or blocked. Twelve known agents covered.
Check a site
Why this matters
Visibility
Agents follow robots.txt
GPTBot, PerplexityBot, ClaudeBot and others honor the same protocol Googlebot does. If you block them, you don't appear in their answers.
Granularity
Agent-specific rules win
A specific User-agent: GPTBot block overrides a wildcard allow. Match what you want declared, not just inherited from *.
Next-gen
Content-Signals are coming
The Content-Signal: directive lets you grant fine-grained consent (train / summarize / search). We surface it when present.