Directory
‹ Back to apps

Crawl Readiness

Check whether a website is set up for AI the way its owner intends, from Cursor: which of 50+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google's and more) may read it and what its access policy allows, validate JSON-LD and robots.txt, compare what people and AI crawlers receive, generate llms.txt, an AI-crawler robots.txt with its Content-Signal and TDM lines, and JSON-LD schema, read your Answer Analysis, and ask whether a page may be used for search, answers, training or agent action, with a signed receipt. Nine tools; five work without an account.

What’s inside

1 MCP server · 1 skill
  • MCP servercrawl-readinessNine tools that call the live Crawl Readiness service. Five work without an account: score a site 0-100 against 50+ AI crawlers with a prioritized fix list, validate JSON-LD structured data, audit robots.txt line by line, compare what people see with what AI crawlers receive, and check whether a page's published signals (robots.txt, Content-Signal, Content-Usage, RSL, TDM, ai.txt, noai) permit search, answers, training or agent action, with a signed receipt. A free API key unlocks three generators (llms.txt, an AI-crawler-aware robots.txt, JSON-LD schema) and lifts the permission check's daily limit; reading your Answer Analysis needs a key on a plan that includes one (Monitor, Agency or the audit).
  • Skillai-readinessCheck whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google's and 50+ more) can read a website and what its access policy allows, generate the files that fix what is blocked, read the Answer Analysis, and check whether a page permits an AI use. Use when a user asks whether AI can read their site, how to make it readable, or whether a page may be used for search, answers, training or an agent.