AI Crawler Checker - GPTBot, ClaudeBot, robots.txt & llms.txt
Check if websites let AI crawlers in: robots.txt rules for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 15 more, llms.txt and noai tags. $1 per 1,000 sites.
Publisher description, verbatimLast 7 dayson this actor
- 1other change
Each one reaches followers by email the morning after.Follow this actor
Price / 1k$1.00per 1,000 site-checked
Users / 30d1→ 0 over 9 readings
Users, all time2→ 0 over 9 readings
Runs / 30d8↑ 7 over 9 readings
Runs / user / mo8.00runs per monthly user
1top-10 keywords
best #9 for “scraper claude”
- scraper claude#9
- 2026-10-24contentREADME updated+485 characters+485characters
- 2026-10-16searchKeyword rank moveentered top-10 for “scraper claude” (outside top 100 → #4)—on record
- 2026-10-16changeListed—on record
Follow this actor, compare it against its rivals, and get every overnight change in one morning report.
Publish it yourself? Claim your publisher page and its full history is yours free, forever.
Numbers on this page are measured only by an independent census of the Apify Store — not self-reported.
Not indexed: too little has happened to this actor for us to have anything worth saying yet.