How to Scrape Dynamic Websites With a Headless Browser
TL;DR
Use a headless browser only when the required data appears after JavaScript runs or after a user-like action. Direct HTTP requests are faster and simpler when the page or an authorized data endpoint already contains the data.
Wait for a business condition, not a generic delay. A visible result container, completed network response, or application state is more reliable than sleep(5000).
Extract stable identifiers and validate the result before saving it. A successful navigation does not prove that the expected records rendered.
Playwright is a strong default for headless web scraping of dynamic websites. Its locators re-resolve DOM elements and its actionability checks reduce common timing failures.
Browser processes are expensive compared with HTTP clients. Reuse browsers, isolate contexts, cap concurrency, and close every resource.
Managed crawling can replace browser operations when infrastructure is the bottleneck. Nstproxy Crawl provides bounded crawling and rendered outputs while the application remains responsible for domain-specific validation.
What headless web scraping does
Headless web scraping loads a page in a browser engine without displaying a normal window, executes client-side JavaScript, and extracts the resulting DOM or network data. It is appropriate when a basic HTTP response contains only an application shell and the records appear later. Nstproxy Crawl is a managed alternative when teams need rendered pages and bounded site discovery without operating browser workers directly.
Dynamic does not always mean “requires a browser.” Many applications call a JSON or GraphQL endpoint after load. If that endpoint is public, documented, and permitted for the intended use, calling it directly is usually more reliable. Use a browser when rendering, session state, interaction, or browser-only code is actually necessary.
A static scraper sees the response body returned by the server. A client-rendered application may then execute JavaScript, fetch data, create DOM nodes, hydrate components, or replace the page after a route transition. Extracting immediately after the initial response can therefore return an empty container.
Common timing boundaries include:
the initial DOMContentLoaded event;
a specific API response completing;
a loading attribute changing state;
the first stable result becoming visible;
an infinite-scroll batch being appended;
a route transition updating the view without a full navigation.
The browser's load event is not a universal completion signal. Long-polling and analytics can keep the network active, while visible results may appear before all resources finish.
Prerequisites
The worked example uses Node.js, playwright-core, and an installed Chrome executable. The Playwright library guide documents installation and the browser-launch model. Using the full playwright package with managed browser binaries is often more portable; playwright-core is useful when an approved browser is already provisioned.
Only scrape public or otherwise authorized content. Check site terms, robots policies where applicable, privacy requirements, and collection frequency. Do not automate login bypasses, paywalls, or access-control evasion.
Detailed tutorial
The tutorial uses a purpose-built page that inserts two catalog records after a short client-side delay. The exact script was run with Node.js and playwright-core against local Chrome; it returned two validated records.
Method 1: render and extract with Playwright
Step 1: install the dependency
Create an empty Node project and install the browser library:
npminstall playwright-core
For a portable environment, follow Playwright's maintained installation instructions and install its supported browser binaries. Pin the dependency version in production and test browser upgrades before rollout.
Step 2: launch a browser and navigate
Create scrape.mjs:
import{ chromium }from"playwright-core";const targetUrl = process.env.TARGET_URL;if(!targetUrl)thrownewError("TARGET_URL is required");const browser =await chromium.launch({headless:true,executablePath: process.env.CHROME_PATH});try{const page =await browser.newPage();await page.goto(targetUrl,{waitUntil:"domcontentloaded",timeout:30_000});// Extraction continues in the next step.}finally{await browser.close();}
Keep the executable path in deployment configuration rather than hard-coding a developer's machine path. The outer try/finally ensures Chrome closes after both success and failure.
The Playwright locator guidance explains that locators resolve against the current DOM, which helps when a framework re-renders nodes. Prefer accessible roles, text, labels, stable data attributes, or a documented DOM contract. Avoid long CSS paths based on generated class names and element positions.
Do not use a fixed delay as the primary strategy. A delay that works locally may fail under load and wastes time when data arrives quickly.
Step 4: extract a stable schema
Extract only the fields the pipeline needs:
const records =await cards.evaluateAll((elements)=> elements.map((element)=>({id: element.getAttribute("data-id"),name: element.querySelector("h2")?.textContent?.trim()??null,status: element.querySelector('[data-field="status"]')?.textContent?.trim()??null})));if( records.length===0|| records.some((record)=>!record.id||!record.name)){thrownewError("Rendered page did not produce valid records");}console.log(JSON.stringify(records));
Validation is load-bearing. Without it, an access-denied template, consent page, or changed selector can be stored as a successful empty result.
Step 5: run the scraper
Set the approved target and browser path in the environment, then run:
Waiting for a specific response works better when the rendered list has unstable markup but the page calls a recognizable data endpoint. Register the wait before triggering the action that starts the request:
This approach observes a request made by the authorized page; it does not grant permission to call or reverse-engineer private endpoints. Validate content type and schema before using the payload.
Method 3: detect repeated DOM changes
For interfaces without a stable response URL, a page-side MutationObserver can detect when an expected element appears. The MDN MutationObserver reference defines the API for observing DOM changes.
Playwright locators already cover many appearance waits. Use a custom observer only when the application requires a more specific condition, such as a result count stabilizing across several mutations. Always include a timeout and remove the observer when complete.
Handle pagination and infinite scroll
Infinite scroll is a state machine, not an instruction to keep scrolling. Track a stable record identifier and stop when one of these conditions occurs:
the application reports no next cursor;
no new IDs appear after a bounded number of attempts;
an explicit maximum page or record count is reached;
a rate, permission, or validation error occurs.
Persist the last accepted cursor or record ID so a retry does not restart from the beginning. Deduplicate by domain identity, not by the position of an element on the page. The automated data collection guide covers scheduling and refresh behavior beyond one browser run.
Control browser cost and concurrency
Launching a browser for every URL consumes memory and startup time. Reuse one browser process, create isolated contexts for unrelated sessions, and cap the number of concurrent pages. Close contexts and pages even after timeouts.
Playwright performs actionability checks before interactions, as described in its auto-waiting documentation. Those checks reduce flakiness for clicks and inputs, but they do not validate business results. A button can be clickable while the requested dataset is empty.
Track navigation time, wait time, extracted record count, validation failures, target status, and browser crashes. Avoid logging credentials, session cookies, or sensitive response bodies.
When managed crawling is the better choice
A self-managed browser is useful when the workflow requires custom interactions or exact page-level debugging. A managed crawling API is often better when the real burden is maintaining browser workers, retries, site discovery, rendering, and output conversion.
Nstproxy Crawl can process single pages or bounded sites, render JavaScript, apply page and path controls, and return selected outputs such as Markdown, HTML, JSON, links, screenshots, or PDF. The application must still validate that the correct content was returned. Teams building an AI web retrieval pipeline should compare cost per accepted document rather than cost per request.
The web index guide explains the next boundary: canonicalization, stable IDs, freshness, and chunking remain downstream responsibilities after a page is fetched.
Troubleshooting headless dynamic scraping
Symptom
Likely cause
Correct response
Empty result
Extraction ran before rendering
Wait for a visible result or data response
Timeout despite visible page
Completion condition is too broad
Wait for the specific business element
Duplicate records
Infinite scroll repeats items
Deduplicate by stable source ID
Works locally, fails in production
Browser or fonts differ; resources are constrained
Use roles, labels, text, or stable data attributes
Navigation succeeds but data is wrong
Soft error, consent, or denial page
Validate title, expected fields, and record count
Final verdict: wait for data, then validate it
Headless web scraping is appropriate when JavaScript rendering or interaction is essential. Playwright provides a reliable control surface, but the scraper must still use explicit completion conditions, stable selectors, bounded pagination, resource cleanup, and semantic validation.
The next step is to reproduce one target page with a ten-record acceptance sample and measure render completeness, extraction accuracy, latency, and failure categories. If browser fleet maintenance is the bottleneck, test Nstproxy Crawl on the same authorized URLs and compare accepted outputs.
Run dynamic extraction without managing browser workers
Use Nstproxy Crawl to collect bounded, JavaScript-rendered pages in formats your extraction or RAG pipeline can validate and store.
Headless web scraping uses a browser engine without a visible window to execute JavaScript and extract the resulting page state.
Q: Is Playwright better than Selenium for dynamic scraping?
Playwright is a strong default for modern browser automation, but the better choice depends on language support, existing infrastructure, browser requirements, and team expertise. Evaluate both on the target workflow.
Q: Should a scraper wait for network idle?
Usually not as its only condition. Analytics, streaming, and polling can prevent true network idleness, so wait for the specific element, response, or application state that proves the data is ready.
Q: Why does a headless scraper return empty HTML?
The scraper often reads the page before client-side rendering finishes or uses a selector that no longer matches. Record a DOM snapshot and validate the expected container after an explicit wait.
Q: Is headless web scraping legal?
Legality depends on the data, access method, contract terms, jurisdiction, and intended use. Collect only public or authorized content and obtain legal guidance for sensitive or high-risk workflows.
Ivy Lin
Sep. 3rd 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
