Spider

Crawlers for LLM data

About Spider

Crawler written in Rust, offered as an API and open-source library, that returns pages as markdown, text or HTML.

Details

  • Developer API

    Yes

  • Returns markdown for LLMs

    Yes

Not enough info yet on open source, works without code, browser extension, free plan or free tier. Check the tool's own site.

Listing written by SpiderPages, checked 2026-11-04. Report a correction or request removal. Back to crawlers for llm data.

More tools in crawlers for llm data

Firecrawl

API and open-source project that crawls a site and returns each page as clean markdown or structured data for language models.

  • Developer API
  • Open source
  • Free plan or free tier

Jina Reader

by Jina AI

Reader API that takes a URL and returns the main content as markdown text that a language model can read.

  • Developer API
  • Returns markdown for LLMs

Tavily

Search, extract and crawl API built for AI agents, returning cleaned page content alongside search results.

  • Developer API

Exa

Search engine API for AI applications that finds pages by meaning and can return their full text.

  • Developer API

Crawl4AI

Open-source Python crawler built on Playwright that outputs markdown and structured fields for retrieval and agent pipelines.

  • Open source
  • Returns markdown for LLMs

Common Crawl

by Common Crawl Foundation

Nonprofit that crawls the web and publishes the raw pages, metadata and text as free open datasets.