apifystats
Log in Start free
Actor/kelp_forest_labs

AI Crawler Readiness: robots.txt & llms.txt

Monitoring

See which AI crawlers a site's robots.txt allows or blocks (18 known tokens plus your own), with the deciding rule for each, and whether llms.txt exists and is well-formed. Identified checker, three requests per domain, refusals and errors are free.

Publisher description, verbatim
Last 7 dayson this actor

No changes seen in the last 7 days.

The morning one happens, followers hear first.Follow this actor
Key figures16 readings
Price / 1k$2.00per 1,000 domain
Users / 30d1→ 0 over 16 readings
Users, all time2→ 0 over 16 readings
Runs / 30d14↑ 13 over 15 readings
Runs / user / mo14.0runs per monthly user
Store search presence
1top-10 keywords

best #2 for “scraper user agent”

  • scraper user agent#2
re-crawled daily
Recorded changes
  • 2026-10-17searchKeyword rank moveentered top-3 for “scraper user agent” (outside top 100 → #2)—on record
  • 2026-10-17changeListed—on record
showing 2 of 2 in the last 30 days · followers get the next one by email

History

16 readings since 2026-10-09
Users / 30d16 readings
10-09→ 010-24
Users, all time16 readings
10-09→ 010-24
Runs / 30d15 readings
10-09↑ 1310-24

Follow this actor, compare it against its rivals, and get every overnight change in one morning report.

Publish it yourself? Claim your publisher page and its full history is yours free, forever.

Open in the app Claim this page

Numbers on this page are measured only by an independent census of the Apify Store — not self-reported. Not indexed: too little has happened to this actor for us to have anything worth saying yet.