Spider
Crawler written in Rust, offered as an API and open-source library, that returns pages as markdown, text or HTML.
- Developer API
- Returns markdown for LLMs
About Firecrawl
API and open-source project that crawls a site and returns each page as clean markdown or structured data for language models.
Firecrawl turns websites into clean data for language models. Give it a URL and it can scrape one page, crawl a whole site or map its links, returning each page as markdown, HTML or structured fields that match a schema you describe. It handles pages that need JavaScript rendering.
It is offered as a hosted API with SDKs for Python and Node.js, and the core project is open source for teams that want to host it. Developers use it to feed retrieval systems, build agents that read the web and collect documentation or articles for analysis.
Before a large crawl, check how it treats robots.txt and rate limits for your plan, and read the target site terms on automated access. Markdown output removes navigation and boilerplate, so spot check a few pages to confirm the parts you need survive.
Details
Developer API
YesOpen source
YesFree plan or free tier
YesReturns markdown for LLMs
Yes
Not enough info yet on works without code, browser extension. Check the tool's own site.
Listing written by SpiderPages, checked 2026-11-04. Report a correction or request removal. Back to crawlers for llm data.