Firecrawl
API and open-source project that crawls a site and returns each page as clean markdown or structured data for language models.
- Developer API
- Open source
- Free plan or free tier
About Common Crawl
Nonprofit that crawls the web and publishes the raw pages, metadata and text as free open datasets.
Common Crawl is a nonprofit foundation that crawls the public web and releases the results as free, open datasets. Each crawl contains billions of pages, stored as WARC files with the raw responses, plus WAT files with metadata and WET files with extracted plain text.
The data is hosted on Amazon Web Services as a public dataset, and an index lets you look up which crawls contain a given URL or domain. Researchers use it to study the web, and many language model training sets start from filtered Common Crawl text.
Its crawler identifies itself as CCBot and follows robots.txt, so sites that block it are missing. The data is raw and needs filtering for duplicates, spam and quality before use, and copyright and privacy questions about the pages remain yours to judge.
Details
Listing written by SpiderPages, checked 2026-11-04. Report a correction or request removal. Back to crawlers for llm data.