AI Crawler Access Audit: robots.txt, llms.txt, AI Opt-Out

Check which AI crawlers each website allows or blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 19 more, with the deciding robots.txt line. Plus llms.txt and AI training opt-out signals (TDMRep, noai). Bulk domains, change alerts, $2 per 1,000 domains. No login.

Publisher description, verbatim

Last 7 dayson this actor

3rank moves

Each one reaches followers by email the morning after.Follow this actor

Key figures22 readings

Price / 1k$2.00per 1,000 domain-audited

Users / 30d1→ 0 over 22 readings

Users, all time2→ 0 over 22 readings

Runs / 30d20↑ 19 over 22 readings

Runs / user / mo20.0runs per monthly user

Recorded changes

2026-10-22searchKeyword rank moveleft top-10 for “scraper user agent” (#10 → #20)—on record
2026-10-21searchKeyword rank moveentered top-10 for “scraper user agent” (#12 → #10)—on record
2026-10-20searchKeyword rank moveleft top-10 for “scraper user agent” (#10 → #12)—on record

showing 3 of 7 in the last 30 days · followers get all 7 by email

History

22 readings since 2026-10-03

Users / 30d22 readings

Users, all time22 readings

Runs / 30d22 readings

Follow this actor, compare it against its rivals, and get every overnight change in one morning report.

Publish it yourself? Claim your publisher page and its full history is yours free, forever.

Open in the app Claim this page

Numbers on this page are measured only by an independent census of the Apify Store — not self-reported.