Robots.txt & XML Sitemap Deep Analyzer

Parses robots.txt crawl rules and XML sitemaps. Detects disallowed paths, sitemap locations, AI web scraper permissions (GPTBot, ClaudeBot, CCBot), and index URL counts.

Publisher description, verbatim

Last 7 dayson this actor

2other changes

Each one reaches followers by email the morning after.Follow this actor

Key figures23 readings

Price / 1k$1.80per 1,000 results

Users / 30d1↑ 1 over 23 readings

Users, all time2↑ 1 over 23 readings

Runs / 30d8↑ 2 over 21 readings

Runs / user / mo8.00runs per monthly user

Recorded changes

2026-10-22schemaInput schema changed—on record
2026-10-22contentREADME updated+4,132 characters+4,132characters
2026-10-17searchKeyword rank moveleft top-10 for “scraper claude” (#8 → #13)—on record

showing 3 of 9 in the last 30 days · followers get all 9 by email

History

23 readings since 2026-09-26

Users / 30d23 readings

Users, all time23 readings

Runs / 30d21 readings

Follow this actor, compare it against its rivals, and get every overnight change in one morning report.

Publish it yourself? Claim your publisher page and its full history is yours free, forever.

Open in the app Claim this page

Numbers on this page are measured only by an independent census of the Apify Store — not self-reported.