robots.txt Analyzer — Rules, Sitemaps & Crawl Blocks

Parse and audit any site's robots.txt in one call: user-agent groups, allow/disallow rules, declared sitemaps, and crawl-block issues like an accidental site-wide Disallow. $0.02 per site analyzed. No API key.

Publisher description, verbatim

Last 7 dayson this actor

No changes seen in the last 7 days.

The morning one happens, followers hear first.Follow this actor

Key figures30 readings

Price / 1k$20.00per 1,000 robots-analyzed

Users / 30d1→ 0 over 30 readings

Users, all time2→ 0 over 30 readings

Runs / 30d29→ 0 over 30 readings

Runs / user / mo29.0runs per monthly user

Users / 30d30 readings

Recorded changes

showing 1 of 1 in the last 30 days · followers get the next one by email

2026-08-24contentREADME updated+1,505 characters+1,505characters

Follow this actor, compare it against its rivals, and get every overnight change in one morning report. Publish it yourself? Claim your publisher page and its full history is yours free, forever.

Open in the app Publish this actor? Claim the page

Numbers on this page are measured only by an independent census of the Apify Store — not self-reported.