robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks

Parse and audit any site's robots.txt in one call: user-agent groups, allow/disallow rules, declared sitemaps, and crawl-block issues like an accidental site-wide Disallow. $0.02 per site analyzed. No API key.

Publisher description, verbatim

Last 7 dayson this actor

No changes seen in the last 7 days.

The morning one happens, followers hear first.Follow this actor

Key figures30 readings

Price / 1k$20.00per 1,000 robots-analyzed

Users / 30d2↑ 1 over 30 readings

Users, all time3↑ 1 over 30 readings

Runs / 30d27↓ 1 over 30 readings

Runs / user / mo13.5runs per monthly user

Recorded changes

2026-09-25schemaInput schema changed—on record
2026-09-25contentREADME updated+1,540 characters+1,540characters

showing 2 of 2 in the last 30 days · followers get the next one by email

History

30 readings since 2026-09-25

Users / 30d30 readings

Users, all time30 readings

Runs / 30d30 readings

Follow this actor, compare it against its rivals, and get every overnight change in one morning report.

Publish it yourself? Claim your publisher page and its full history is yours free, forever.

Open in the app Claim this page

Numbers on this page are measured only by an independent census of the Apify Store — not self-reported.