Scrapy

Scrapy is an open-source Python framework for writing web crawlers. You define spiders that say which URLs to fetch and how to parse each response, and Scrapy handles the request queue, concurrency, retries, cookies and export to JSON, CSV or a database through item pipelines.

It has settings for polite crawling, such as obeying robots.txt, delaying requests and limiting concurrency per domain, which is one reason it suits long-running jobs. Middleware lets teams add proxies, custom headers or a headless browser for pages that need JavaScript.

Scrapy is a good fit for engineers who need full control and repeatable crawls of many pages. It has a learning curve, and it does not render JavaScript by default, so plan for a browser add-on when the target pages build their content in the browser.