Heritrix

By Internet Archive ยท Open-source crawler frameworks

About Heritrix

Archival-quality web crawler from the Internet Archive that writes WARC files for preserving sites.

Details

  • Open source

    Yes
  • Works without codeNo

Not enough info yet on developer API, browser extension, free plan or free tier, returns markdown for LLMs. Check the tool's own site.

Listing written by SpiderPages, checked 2026-11-04. Report a correction or request removal. Back to open-source crawler frameworks.

More tools in open-source crawler frameworks

Scrapy

by Scrapy project

Python framework for writing crawlers as spiders, with request scheduling, item pipelines, throttling and export built in.

  • Open source

Crawlee

by Apify

Open-source crawling library for Node.js and Python that switches between plain HTTP and headless browsers with one API.

  • Open source

Beautiful Soup

by Leonard Richardson

Python library for parsing HTML and XML and searching the tree, usually paired with a separate HTTP client.

  • Open source

Colly

by Colly project

Scraping framework for Go with callbacks for each element, request limits per domain and parallel collectors.

  • Open source

Apache Nutch

by Apache Software Foundation

Long-running Java web crawler from the Apache project, built for large crawls on Hadoop and plugin-based parsing.

  • Open source