Website Content Crawler Pro

Turn any website into clean, LLM-ready Markdown (or text) in one run. Built for feeding documentation, blogs and knowledge bases into RAG pipelines, vector databases and AI agents. Crawls with a fast HTTP engine or a full JavaScript browser, extracts just the main content (no nav, ads or cookie banners), and follows internal links with precise glob and sitemap control. No account, no API key.

Features

• 📝 Markdown output — clean, structured Markdown per page, ideal for LLMs, RAG and vector DBs (plus plain text and optional HTML)
• 🌐 HTTP or Browser mode — fast HTTP crawling for static sites, or full Chromium rendering for JavaScript/SPA sites
• 🧹 Main-content extraction — Readability strips navigation, ads, footers and cookie banners, keeping the real article
• 🗺️ Sitemap + glob crawling — seed from sitemap.xml, and include/exclude URLs with glob patterns
• 🔗 Smart link following — configurable depth, same-domain-only, and total page cap
• 🎯 CSS control — keep-only or remove specific elements; wait for selectors and dynamic content in browser mode
• ⚡ Fast & parallel — tunable concurrency, retries and proxy support

What you get

One record per page:

{
  "url": "https://docs.example.com/guide",
  "title": "Getting Started",
  "description": "A short guide to …",
  "author": null,
  "siteName": "Example Docs",
  "lang": "en",
  "wordCount": 812,
  "markdown": "# Getting Started\n\nThis guide walks you through …",
  "text": "Getting Started This guide walks you through …"
}

Output schema

Input

{
  "urls": ["https://docs.apify.com/academy"],
  "crawlerType": "http",
  "crawl": true,
  "maxCrawlPages": 200,
  "includeUrlGlobs": ["https://docs.apify.com/academy/**"],
  "saveMarkdown": true
}

Use cases

• RAG & AI knowledge bases — turn docs sites into Markdown for retrieval-augmented generation
• LLM fine-tuning datasets — collect clean article text at scale
• Docs & blog migration — export a site's content to Markdown
• Competitive/content research — pull structured page content across a domain

Pricing

Pay-per-event: a small amount per page crawled. See the Pricing tab for current rates.

FAQ

What's the difference from a basic content scraper? This Pro version outputs clean Markdown (not just text), renders JavaScript sites with a real browser, extracts the main content with Readability, and supports sitemaps + glob filters — everything a RAG/LLM pipeline needs.

Do I need an API key? No. Just give it URLs and run — no account or key required.

Can it handle JavaScript / single-page apps? Yes. Set crawlerType to browser and, if needed, waitForSelector or dynamicContentWaitSecs so client-rendered content loads before extraction.

What output format is best for AI/RAG? Markdown. It preserves headings, lists and links in a compact form that LLMs and chunkers handle well. Plain text and HTML are also available.

How many pages can it crawl? Up to tens of thousands per run via maxCrawlPages, with sitemap seeding and glob filters to target exactly the pages you want.

Can I connect this to other apps and services? The Website Content Crawler Pro can be connected with almost any cloud service or web app thanks to integrations on the Apify platform. It works with Make, Zapier, Slack, Airbyte, GitHub, Google Drive and many more, plus the Apify API, JavaScript/Python clients and MCP. Or use webhooks to trigger an action whenever a run finishes.

Notes

Original clean-room implementation using open-source libraries (Crawlee, Mozilla Readability, Turndown). Respects the target site's content; use responsibly and within each site's terms. This is an independent tool, not affiliated with any site it crawls.

You pay only for what the run returns. Set a maximum cost per run and it stops cleanly at that limit.