Playwright

Playwright is an open-source browser automation library from Microsoft. It drives Chromium, Firefox and WebKit through one API, with official versions for Node.js, Python, Java and .NET. It was built for end-to-end testing and is widely used for scraping pages that need a real browser.

Its auto-waiting means scripts wait for elements to be ready before clicking or reading, which makes them less flaky on script-heavy pages. It supports multiple isolated browser contexts, network interception, file downloads, screenshots and PDFs, and a code generator that records your clicks as a starting script.

Use it when plain HTTP requests return empty pages because content loads in the browser. Real browsers use more memory and CPU than HTTP clients, so test how many sessions your machines can run, and keep request rates within what the target site permits.