Search AI web scraping tools

Desktop app where you click the elements you want, and it follows pagination, dropdowns and logins to build the dataset.

Works without code
Free plan or free tier

Browse AI

Train a robot by recording clicks on a page, then run it on a schedule to extract lists or watch the page for changes.

Developer API
Works without code
Free plan or free tier

Web Scraper

No-code scrapers · by webscraper.io

Browser extension for building sitemaps of selectors and scraping multi-page sites, with an optional cloud service for larger runs.

Works without code
Browser extension
Free plan or free tier

Bardeen

Browser extension that scrapes the page you are on and sends the rows to sheets, CRMs and other apps as part of an automation.

Works without code
Browser extension

Data Miner

Chrome and Edge extension that pulls tables and lists from pages into CSV or Excel using shared or custom extraction recipes.

Works without code
Browser extension

Thunderbit

Chrome extension that suggests columns for the current page with AI, then scrapes it and any subpages into a table.

Works without code
Browser extension

Simplescraper

Extension and cloud service for selecting page data by clicking, then running the recipe on a schedule or calling it as an API.

Works without code
Browser extension

Axiom.ai

No-code browser bots built in a Chrome extension that click, type, scrape and pass data to Google Sheets or other tools.

Works without code
Browser extension

Apify

Cloud platform for running scrapers and browser automations, called Actors, with a store of ready-made ones, scheduling, storage and an API.

Developer API
Free plan or free tier

ScrapingBee

Web scraping API that fetches a URL with a headless browser and proxies, returning HTML, screenshots or extracted fields.

Developer API

ScraperAPI

Scraping API that handles proxies, retries and rendering for each request, with structured endpoints for some large sites.

Developer API

Zyte API

Scraping APIs and platforms · by Zyte

Scraping API from the company behind Scrapy, with browser rendering and AI extraction of products, articles and job posts.

Developer API

Scrapfly

Scraping API with browser rendering, screenshots and an extraction endpoint, plus SDKs for Python and TypeScript.

Developer API

Crawlbase

Crawling and scraping API with a queued crawler for large URL lists and storage for the results.

Developer API

Diffbot

Extraction APIs that classify a page and return articles, products or discussions as fields, plus a large knowledge graph built from the web.

Developer API

Kadoa

AI extraction service that builds and maintains scrapers from a description of the data you want, then delivers it on a schedule.

Developer API

ScrapeGraphAI

Open-source Python library and hosted API that use language models to extract fields from a page based on a plain prompt.

Developer API
Open source

Import.io

Web data extraction platform and managed service for teams that need recurring datasets such as product and price data.

Nimble

Web data platform with browser-based collection and AI parsing that returns structured results from pages and search engines.

Developer API

Firecrawl

API and open-source project that crawls a site and returns each page as clean markdown or structured data for language models.

Developer API
Open source
Free plan or free tier

Crawl4AI

Open-source Python crawler built on Playwright that outputs markdown and structured fields for retrieval and agent pipelines.

Open source
Returns markdown for LLMs

Spider

Crawler written in Rust, offered as an API and open-source library, that returns pages as markdown, text or HTML.

Developer API
Returns markdown for LLMs

Jina Reader

Crawlers for LLM data · by Jina AI

Reader API that takes a URL and returns the main content as markdown text that a language model can read.

Developer API
Returns markdown for LLMs

Tavily

Search, extract and crawl API built for AI agents, returning cleaned page content alongside search results.

Developer API

Exa

Search engine API for AI applications that finds pages by meaning and can return their full text.

Developer API

Common Crawl

Crawlers for LLM data · by Common Crawl Foundation

Nonprofit that crawls the web and publishes the raw pages, metadata and text as free open datasets.

Scrapy

Open-source crawler frameworks · by Scrapy project

Python framework for writing crawlers as spiders, with request scheduling, item pipelines, throttling and export built in.

Open source

Crawlee

Open-source crawler frameworks · by Apify

Open-source crawling library for Node.js and Python that switches between plain HTTP and headless browsers with one API.

Open source

Beautiful Soup

Open-source crawler frameworks · by Leonard Richardson

Python library for parsing HTML and XML and searching the tree, usually paired with a separate HTTP client.

Open source

Colly

Open-source crawler frameworks · by Colly project

Scraping framework for Go with callbacks for each element, request limits per domain and parallel collectors.

Open source

Apache Nutch

Open-source crawler frameworks · by Apache Software Foundation

Long-running Java web crawler from the Apache project, built for large crawls on Hadoop and plugin-based parsing.

Open source

Heritrix

Open-source crawler frameworks · by Internet Archive

Archival-quality web crawler from the Internet Archive that writes WARC files for preserving sites.

Open source

Playwright

Headless browsers · by Microsoft

Browser automation library for Chromium, Firefox and WebKit, with APIs for Node.js, Python, Java and .NET.

Open source

Puppeteer

Headless browsers · by Google Chrome team

Node.js library that controls Chrome or Firefox through a high-level API, often used for rendering and screenshots.

Open source

Selenium

Headless browsers · by Selenium project

Long-standing browser automation project with WebDriver bindings in many languages and Grid for running browsers at scale.

Open source

Browserless

Headless browsers ·

Hosted headless Chrome you connect to from Puppeteer or Playwright, with REST endpoints for content, PDFs and screenshots.

Developer API

Browserbase

Headless browsers ·

Cloud headless browsers for automation and AI agents, with session recordings and debugging tools.

Developer API

Steel

Headless browsers ·

Browser API for AI agents and automations, available as a hosted service and an open-source browser server.

Developer API
Open source

Stagehand

AI browser agents · by Browserbase

Open-source framework that adds act, extract and observe steps written in plain language to Playwright scripts.

Open source

Browser Use

AI browser agents ·

Open-source Python library that lets a language model operate a browser to finish tasks and collect data.

Open source

Skyvern

AI browser agents ·

Browser automation that uses vision and language models to work through forms and multi-step flows, open source or hosted.

Developer API
Open source

AgentQL

AI browser agents · by TinyFish

Query language and SDK that locate page elements and data from a described shape, used inside Playwright scripts.

Developer API

Bright Data

Large proxy network with residential, datacenter and mobile IPs, plus scraping APIs and ready-made datasets.

Developer API

Oxylabs

Proxy provider with residential, datacenter and mobile pools and a web scraper API for search and ecommerce pages.

Developer API

Decodo

Proxy and scraping API provider formerly known as Smartproxy, with residential, mobile and datacenter options.

SOAX

Residential, mobile and datacenter proxies with targeting by country, city and network provider.

NetNut

Proxy network that routes residential traffic through ISP connections, with static and rotating options.

IPRoyal

Proxy provider selling residential, ISP, datacenter and mobile proxies, including traffic that does not expire.

Webshare

Self-serve proxy service focused on datacenter and residential proxies with a dashboard and API for managing lists.

Developer API
Free plan or free tier

Rayobyte

US-based proxy provider with datacenter, ISP and residential proxies and a published ethics policy on sourcing.

Visualping

Watches pages or parts of pages on a schedule and emails or messages you when text or visuals change, with AI summaries of the change.

Works without code
Browser extension
Free plan or free tier

Distill.io

Browser extension and cloud monitor that checks chosen page elements and alerts by email, push or webhook.

Works without code
Browser extension
Free plan or free tier

ChangeTower

Website change monitoring with keyword alerts, archives of each version and reports for compliance and research teams.

Works without code

Wachete

Monitors pages, including ones behind logins, and records history with alerts when the watched content changes.

Works without code

Fluxguard

Change detection for whole sites, comparing text, DOM and screenshots across crawls and flagging differences.

changedetection.io