Robots.txt Tester

hipersoft/robots-txt-checker

Test whether URLs are allowed or blocked by a site's robots.txt for any crawler user-agent. Returns allowed/blocked, the matched rule, crawl-delay and the site's sitemaps. Great for SEO audits and crawler compliance. One row per URL as JSON, CSV or Excel.

Run this scraper →See the API call

Total users

Monthly users

Total runs

Per url checked

$0.0003

Last rebuilt

Oct 26, 2026

Overview

Check whether URLs are allowed or blocked by a site's robots.txt with the Robots.txt Tester. Pass a list of URLs and a crawler user-agent, and get back — for each URL — whether it's crawlable, the exact rule that matched, the crawl-delay and the site's sitemaps. Export as JSON, CSV or Excel.

Built for SEO audits, crawler and scraper compliance, indexing checks and site migrations, it applies real robots.txt matching rules so you know exactly what a bot is allowed to fetch.

What it does

✅ Allowed / blocked — a clear verdict for each URL and user-agent.
🎯 Matched rule — the exact Allow/Disallow line that decided it (longest-match, Allow-wins precedence).
🤖 Any user-agent — test as *, Googlebot, Bingbot, GPTBot or any bot.
⏱️ Crawl-delay — the crawl-delay declared for that agent, if any.
🗺️ Sitemaps — the sitemap URLs listed in the site's robots.txt.
⚡ Bulk & cached — tests many URLs and caches each site's robots.txt for speed.
📤 Export anywhere — JSON, CSV or Excel, or straight into an SEO tool or report.

Example output

{
  "url": "https://en.wikipedia.org/wiki/Special:Search",
  "userAgent": "*",
  "allowed": false,
  "matchedRule": "disallow: /wiki/Special:",
  "robotsFound": true,
  "sitemaps": ["https://en.wikipedia.org/sitemap.xml"]
}

How to use it

Add the URLs you want to test.
Set the crawler user-agent (e.g. Googlebot).
Run, and download the results as JSON, CSV or Excel.

Input fields

Field	Description
URLs to test	Full URLs (or bare domains) to test.
Crawler user-agent	The bot to test as (`*`, `Googlebot`, `GPTBot`…).
Max URLs	Maximum URLs to test (0 = no limit).

Output fields

url, userAgent, allowed, matchedRule, crawlDelay, robotsUrl, robotsFound, sitemaps, error.

Popular use cases

SEO audits — confirm important pages aren't accidentally blocked.
Crawler compliance — respect robots.txt before scraping a site.
Indexing checks — verify what search bots can and can't fetch.
Migrations — catch robots.txt mistakes after a site move.
AI crawler policy — see whether bots like GPTBot are allowed.

FAQ

Do I need an account or key? No. Add URLs and run.

How is the verdict decided? By standard robots.txt matching — the most specific (longest) matching rule wins, and Allow beats Disallow on a tie, for the selected user-agent.

Can I test different bots? Yes — set any user-agent (Googlebot, Bingbot, GPTBot, etc.) to see rules that apply to it.

Can I export to Excel or Google Sheets? Yes — results download as JSON, CSV or Excel and integrate with SEO tools and reports.

Test robots.txt allow/block rules for any URL and crawler — with the matched rule, crawl-delay and sitemaps — clean and export-ready for SEO and compliance.

Input

Field	Type	Description
`urls` URLs to test	array	Full URLs to test against each site's robots.txt — e.g. 'https://example.com/page', 'example.com/admin'. Bare domains are treated as HTTPS.
`userAgent` Crawler user-agent	string	The crawler user-agent to test as — e.g. '', 'Googlebot', 'Bingbot', 'GPTBot'. Default: `""`
`maxItems` Max URLs	integer	Maximum number of URLs to test (0 = no limit). Default: `0`

API

curl -X POST "https://api.apify.com/v2/acts/hipersoft~robots-txt-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://en.wikipedia.org/wiki/Special:Search",
    "https://example.com/"
  ]
}'

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("hipersoft/robots-txt-checker").call(run_input={
    "urls": [
        "https://en.wikipedia.org/wiki/Special:Search",
        "https://example.com/"
    ]
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('hipersoft/robots-txt-checker').call({
  "urls": [
    "https://en.wikipedia.org/wiki/Special:Search",
    "https://example.com/"
  ]
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Pricing

Event	Price	Paid plans from
URL checked Charged for each URL tested against robots.txt.	$0.0003	$0.0001

You pay only for what the run returns. Set a maximum cost per run and it stops cleanly at that limit.

Related scrapers

Bulk Image Downloader

Download thousands of images from a list of URLs and store each in the key-value store with a public link. Returns content type, size, dimensions and SHA-256 per image. Bulk-ready. Great for datasets, backups and media pipelines.

8,102 runs · 103 users · $0.056 / url processed

Text to Speech

Convert any text to natural-sounding speech as an MP3 download. 300+ premium neural voices across 70+ languages, adjustable rate, pitch and volume. Great for voiceovers, videos, accessibility and audio content. No setup.

278 runs · 22 users · $0.01 / audio generated

Google Maps Email Extractor

Search Google Maps for businesses, then crawl each business website to extract emails, social profiles and extra phone numbers. Turns any Maps search into a ready lead list. No setup.

159 runs · 8 users · $0.006 / business lead scraped

Website Screenshot

Capture full-page or viewport screenshots of any URL as PNG or JPEG, at any device size. For monitoring, previews, thumbnails and automation workflows.

156 runs · 3 users · $0.004 / screenshot captured

Bulk URL Status Checker

Check thousands of URLs over plain HTTP: status codes, broken links (404/410/5xx), full redirect chains, response time, content type, page title and SSL certificate expiry.

154 runs · 13 users · $0.002 / url checked

Google Search Scraper

Scrape Google Search results (organic listings) for any query: title, URL, displayed URL, snippet and position. Multi-query, per-country/language, pagination. Uses Google SERP proxy — reliable, no captcha, no setup.

121 runs · 8 users · $0.001 / result scraped

Need a source that isn't here?

Send the site and the fields. We build it and you get it first.

Request a build