You open the URL in a browser and the page renders fully. Your scraper hits the same URL and gets back an HTML shell. The gap between what your browser shows and what your HTTP client returns is a core problem of scraping dynamic content. The same 3 failures keep appearing: empty HTML, anti-bot blocks, breaking selectors.
The 2026 Imperva Bad Bot Report found that automated traffic is now more than 53% of all web traffic, up from 51% one year before, while human traffic has dropped to 47%. Detection becomes stricter every year. To scrape dynamic content reliably in 2026, you now need to match a real browser at every layer. For many modern JavaScript-heavy or well-protected websites, a single requests.get() call is often not sufficient.
TL;DR
- Use Requests + Beautiful Soup when the data is in the page source (hidden JSON, JSON-LD, or a framework state blob).
- Call the JSON endpoint directly with httpx when you find one in the DevTools Network tab, or curl_cffi if the TLS fingerprint matters.
- Use Playwright when the page needs a real browser to render.
- Add residential proxies plus patchright (a patched Chromium that bypasses Runtime.Enable detection) when you get blocked.
- Wrap the scraper in TLS-fingerprinted retries, Pydantic validation, and idempotent writes when you ship on a schedule.
What is web scraping of dynamic content?
Web scraping of dynamic content means extracting data from pages that render their content with JavaScript after the initial HTML loads, so what you see in the browser is not in the raw HTML your HTTP client receives. eCommerce listings, social feeds, dashboards, search results, and almost every single-page app fit this pattern; if you've ever shipped a React, Vue, Svelte, or HTMX site, you've shipped dynamic content.
A static page returns its full content in the first HTTP response: marketing sites, blogs, and documentation pages still work this way, and a single requests.get() call is enough. Dynamic pages don't.
Many modern web applications send a minimal HTML document and rely on JavaScript to fetch or hydrate content, although rendering strategies vary across frameworks and deployments. The JavaScript may then call one or more APIs, receive JSON, and write the data into the DOM. Once rendering finishes, the DOM and the original HTML can be two different documents.
How to know if a website has dynamic content when web scraping
Compare the initial HTML response against the rendered page. If your target text shows in the browser but not in the raw HTML, the page is dynamic.
Quick test in the browser
Open the page in Chrome or Firefox. Right-click anywhere and pick View Page Source (not Inspect, which shows the rendered DOM). Use Cmd+F (Ctrl+F on Windows or Linux) to search the source for a word you can see on the page, like a product name or a heading. If the word is missing from the source, the data arrives later through JavaScript. Here's that source-vs-rendered split on a JS-rendered demo:
View Page Source on the left shows JavaScript that calls /api/quotes; the rendered page on the right shows the Albert Einstein quote that doesn't appear anywhere in the source.
There are 3 more checks that make this concrete:
- Disable JavaScript in DevTools (Cmd+Shift+P, or Ctrl+Shift+P on Windows or Linux, then "Disable JavaScript") and reload. Whatever disappears is the dynamic part.
- Open the Network tab, filter by Fetch/XHR, and reload. Each row is an API call that the page makes. Click through them and look for one that returns JSON containing your target data.
- Compare response sizes. Static pages send full content first; dynamic SPAs send a small shell, then fetch data through XHR/Fetch calls.
How to scrape dynamic content without a browser
You can get dynamic content from many sites without a browser at all. The 4 techniques below escalate in cost, so try them in order and stop when one returns your data. Use a real browser only as the last option, because running one for every request is slow, memory-heavy, and easier to detect.
Before running the code below, set up your environment. You'll need Python 3.11. Then, in a fresh project directory, run:
uv init && uv add requests httpx[http2] curl_cffi beautifulsoup4 pydantic tenacity playwright patchright
uv run playwright install chromium
uv run patchright install chromium
If you're not on uv yet, swap uv add for pip install and drop the uv run prefix.
Scrape hidden JSON in the page
Many JS-rendered pages embed their data set inside a script tag at first load. The HTML wrapper looks empty, but the raw response contains a window._ _ INITIAL_STATE _ _ assignment, a Next.js _ _ NEXT_DATA _ _ JSON payload, or a plain var data = […] assignment (as on the JS-rendered demo at quotes.toscrape.com/js/. The JavaScript bundle reads that block and renders the page from it. You can read the same block. Here's how that looks in the page source:
Page source of quotes.toscrape.com/js/ showing the var data array literal that holds every quote.
The regex below captures that array, and json.loads turns it into Python dicts:
import json
import re
import requests
resp = requests.get(
"https://quotes.toscrape.com/js/",
headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36"},
timeout=15,
)
resp.raise_for_status()
# The page embeds quote data as: var data = [{...}, {...}, ...]
match = re.search(r"var data = (\[.*?\]);", resp.text, re.DOTALL)
quotes = json.loads(match.group(1))
print(f"Got {len(quotes)} quotes")
print(quotes[0]["author"]["name"], "->", quotes[0]["text"][:60])
Run that and you skip the browser entirely:
Got 10 quotes
Albert Einstein -> "The world as we have created it is a process of o
The regex you need depends on the target: _ _ NEXT_DATA _ _ for Next.js pages router, _ _next_f for Next.js app router (RSC payloads pushed via self.__next_f.push([…])), _ _ NUXT _ _ for Nuxt, window. INITIAL_STATE _ _ for Redux-based React apps. Wrap the parse in try/except and log the raw text on failure so a shape change tells you what broke.
Note that the demo above runs against quotes.toscrape.com, a public site built for scraping practice, so the code is runnable without burning proxies or violating anyone's terms; the same regex patterns apply on real production sites.
Scrape structured data from JSON-LD blocks
Another reliable inline source is the JSON-LD block that sites embed for SEO and AI search. Product pages, recipes, articles, reviews, events, and local-business listings ship a script type="application/ld+json" tag at the bottom of the HTML containing a Schema.org-typed object. The data is in the initial HTML response (before any JS executes) and follows a public schema, so the shape is predictable across sites of the same category. Wikipedia's Python page is a clean example:
Wikipedia's Python page ships a Schema.org Article object in JSON-LD; the name and headline fields are marked here.
The parser below matches @type against a WANTED set and prints obj.get("name") or obj.get("headline"):
import json
import requests
from bs4 import BeautifulSoup
WANTED = {"Article", "Recipe", "Product", "NewsArticle"}
resp = requests.get(
"https://en.wikipedia.org/wiki/Python_(programming_language)",
headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36"}, timeout=15,
)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
for tag in soup.find_all("script", type="application/ld+json"):
# tag.get_text() handles multi-child <script> bodies; tag.string is None for those.
try:
payload = json.loads(tag.get_text())
except json.JSONDecodeError:
continue
# JSON-LD ships as a single object, a list, or a {"@graph": [...]} wrapper.
if isinstance(payload, dict) and "@graph" in payload:
objects = payload["@graph"]
elif isinstance(payload, list):
objects = payload
else:
objects = [payload]
for obj in objects:
# Schema.org allows @type as a string OR a list of strings.
types = obj.get("@type", [])
if isinstance(types, str):
types = [types]
if WANTED & set(types):
print(types[0], "->", obj.get("name") or obj.get("headline"))
Run that and Wikipedia hands you a clean Schema.org Article object without a single CSS selector:
Article -> Python (programming language)
The same parser returns Recipe objects on food sites, Product objects on eCommerce, and NewsArticle objects on news. The JSON-LD path is faster and more durable than DOM selectors because the publisher actively maintains it for Google and AI Overviews; a redesign that breaks every CSS class often leaves the JSON-LD untouched. One catch: on some sites the JSON-LD lags the rendered DOM (stale price, stale stock), so check a few records against the visible page when you start.
Scrape the backend API directly
If the data isn't embedded, it's usually coming from an API call that the page makes after first load. Open the Network tab, find the request that returns the JSON, and copy it as cURL (right-click in Chrome, Copy → Copy as cURL). Paste that cURL into Curl Converter to get a working Python Requests or httpx snippet, then keep the auth-relevant headers (cookies, Authorization, X-CSRF-Token) and drop the sec-ch-ua- / sec-fetch- noise. Here's that DevTools view on the quotes.toscrape.com/scroll demo:
Chrome DevTools showing the /api/quotes XHR for quotes.toscrape.com/scroll: the Fetch/XHR filter narrows 7 requests down to 1, and the Preview pane shows the JSON the page renders from.
The Hacker News public API is a runnable example: no auth, JSON throughout, and the same shape (list endpoint plus per-item endpoint) that most real APIs use.
import httpx
with httpx.Client(http2=True, headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36"}) as client:
top_ids = client.get(
"https://hacker-news.firebaseio.com/v0/topstories.json", timeout=15,
).json()
for sid in top_ids[:5]:
story = client.get(
f"https://hacker-news.firebaseio.com/v0/item/{sid}.json", timeout=15,
).json()
print(story.get("score", 0), "|", story.get("title", ""))
Run that against the live API and you'll see real stories scrolling past (yours will differ, since HN's top stories rotate hourly):
233 | Show HN: Files.md – Open-source alternative to Obsidian
77 | We stopped AI bot spam in our GitHub repo using Git's –author flag
54 | 1024000^2 Blocks, 2B2T Minecraft Server World Download Project
On a target you don't own, check what authentication the call needs (session cookie, X-API-Key header, CSRF token taken from the HTML) and test with realistic pacing, since the API is usually rate-limited per IP or per session. These techniques work with Requests, but at any real volume you'll outgrow it.
Scrape behind TLS fingerprint checks with httpx and curl_cffi
httpx replaces Requests once your scraper runs at any real volume. It supports HTTP/2, async, and clean connection reuse with a near-identical API, and Requests supports none of the three. The basic synchronous pattern looks like this:
import httpx
with httpx.Client(http2=True, headers={"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36"}) as client:
resp = client.get("https://hacker-news.firebaseio.com/v0/topstories.json", timeout=15)
resp.raise_for_status()
print(f"HTTP {resp.status_code}, {len(resp.json())} story IDs")
Run that and you'll see the call succeed over HTTP/2:
HTTP 200, 500 story IDs
For long-running async scrapers, set limits=httpx.Limits(max_connections=100, max_keepalive_connections=20) on the AsyncClient and raise the process ulimit -n. Long-running applications should monitor connection reuse and resource usage. Depending on implementation and workload, stale connections or resource exhaustion can contribute to timeout issues.
If a target rejects httpx even with realistic headers, the next step before reaching for a browser is curl_cffi. It's a Python binding around curl-impersonate that mimics a real Chrome version's TLS fingerprint and HTTP/2 fingerprint, neither of which Requests nor httpx can match. You'll see the hash referenced as JA3 in older guides and JA4 in Cloudflare's and Akamai's current bot-management docs. curl_cffi covers both, since the underlying TLS stack is from a real Chrome build rather than a Python-side hash being patched.
Hitting tls.browserleaks.com/json directly in Chrome returns something like this, trimmed to the fields that matter:
{
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"ja3_hash": "b3723bbf47fc357a1875cd2ac66d7974",
"ja4": "t13d1516h2_8daaf6152771_d8a2da3f94cd",
"tls": {
"selected_version": {"name": "TLS 1.3"},
"cipher_suite": {"name": "TLS_AES_128_GCM_SHA256"}
}
}
That ja3_hash is the value r1["ja3_hash"] reads in the snippet below. Run the same endpoint from Python and you'll see the field is identical; only the hash inside changes per client.
from curl_cffi import requests as curl_requests
import requests
# Same URL, two clients, two different fingerprints
r1 = requests.get("https://tls.browserleaks.com/json", timeout=20).json()
print("requests ja3:", r1["ja3_hash"])
r2 = curl_requests.get("https://tls.browserleaks.com/json",
impersonate="chrome148", timeout=20).json()
print("curl_cffi ja3:", r2["ja3_hash"])
Run this and the 2 JA3 hashes differ, confirming the fingerprint impersonation is real:
requests ja3: a48c0d5f95b1ef98f560f324fd275da1
curl_cffi ja3: f67527d08afa6838d8e453b4316d873c
Pin impersonate to a recent versioned label like chrome146 or chrome136 in production rather than the bare chrome alias. The alias has rotated meaning across releases and silently changes the fingerprint your scraper sends when curl_cffi is upgraded. The exact hash varies across curl_cffi releases as Chrome's TLS profile changes anyway; what matters is that the 2 lines differ, which is what a fingerprint-checking server tests for. Sessions, proxies, and async all work the same way they do in Requests.
How to scrape dynamic content with Playwright in Python
Playwright is a strong default for new browser-based projects. It supports Chromium, Firefox, and WebKit from one API, ships with auto-waiting that removes most timing-related failures, and exposes network interception cleanly. Install with uv add playwright (or pip install playwright if your project isn't on uv yet), then run playwright install chromium. The rest of the article's code works the same way with either.
For protected targets, swap the playwright import for patchright, a drop-in replacement. Its patched Chromium doesn't emit the Runtime.Enable Chrome DevTools Protocol (CDP) signal that some bot-detection stacks rely on. Install it with uv add patchright (or pip install patchright) and then run patchright install chromium; patchright ships its own bundled Chromium build and won't fall back to the one Playwright installed. After that, change one import line and most of your Playwright code keeps working.
Waiting for the right element
Most unreliable scrapers read the DOM before the data renders. Playwright's locators wait by default, but you still want to wait on the specific element that signals "data is ready", not on a fixed delay:
from playwright.sync_api import sync_playwright
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=True)
try:
page = browser.new_page()
page.goto("https://quotes.toscrape.com/js/", wait_until="domcontentloaded")
page.wait_for_selector("div.quote", state="attached", timeout=15_000)
# Below is JavaScript run in the browser tab, not Python.
quotes = page.evaluate("""
() => Array.from(document.querySelectorAll('div.quote')).map(q => ({
text: q.querySelector('span.text')?.innerText,
author: q.querySelector('small.author')?.innerText,
}))
""")
print(f"Got {len(quotes)} quotes")
finally:
browser.close()
Run that and Playwright intercepts the JSON that the front end uses:
page 1, 10 quotes, has_next=True
Intercepting the response this way gives you the same JSON that the front end gets, so you skip both the rendering and the parsing layer. For broader capture (every API call, including pagination and filter changes), use page.on("response", handler) instead, which stays attached for the whole session.
Block unneeded resources and reload login state
There are 2 patterns that cover most operational needs. Blocking images, fonts, and media can cut page time roughly in half on heavy listings (those are Playwright's default heavy-resource tags; layer URL-pattern matching on top to drop trackers or analytics). Reloading saved login state means the login flow runs once per session window, not once per scrape:
from playwright.sync_api import sync_playwright
from pathlib import Path
AUTH = Path("auth.json")
BLOCK_TYPES = {"image", "media", "font"}
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=True)
try:
# Reload saved auth if we have it; otherwise start fresh and save after login.
context = browser.new_context(storage_state=str(AUTH)) if AUTH.exists() else browser.new_context()
def filter_route(route):
if route.request.resource_type in BLOCK_TYPES:
return route.abort()
return route.continue_()
context.route("**/*", filter_route)
page = context.new_page()
page.goto("https://your-app.example.com/dashboard")
# If this is the first run, log in here, then persist the auth state.
if not AUTH.exists():
# ... fill the login form, submit, wait_for_url(...) ...
context.storage_state(path=str(AUTH))
finally:
browser.close()
The storage_state() method serializes cookies, localStorage, and IndexedDB. Most reload failures come from a hand-edited or tool-merged auth.json: Playwright requires sameSite to be exactly Strict, Lax, or None (case-sensitive), and any other value (including Unspecified or lowercase) raises expected one of (Strict|Lax|None) on new_context(). (Playwright tightened sameSite validation in a recent release; before that, malformed cookies got dropped silently instead of raising this error, so legacy fixtures saved earlier can also fail in newer versions.)
Refresh auth.json on a schedule that matches the session expiry, and the login flow runs once per week instead of once per scrape. With the page rendering, asset blocking, and auth reuse covered, the next class of failure is the target actively pushing back.
How to handle anti-bot blocks when scraping dynamic content
Anti-bot defenses fall into a few buckets: header and TLS fingerprinting, behavioral biometrics (mouse movement entropy, scroll velocity, dwell time, keystroke cadence), IP reputation, and challenges (CAPTCHA, JS proof-of-work). You don't beat all of them at once. Instead, you make the scraper boring enough that the cheaper defenses don't trigger, and you stay below the threshold of the expensive ones.
Which defense is hitting your scraper depends on the target. Many bot-management platforms, including Cloudflare and others, may evaluate signals such as TLS characteristics, browser behavior, and IP reputation. The exact detection methods are proprietary. The diagnostic ladder below walks you through isolating the layer.
Headers and TLS fingerprints
Most basic blocks trigger on missing or stale headers. The default python-requests User-Agent string is an obvious signal on protected targets. Match a real browser's header set: a current Chrome User-Agent, Accept-Language, Accept-Encoding of gzip, deflate, br, zstd, Connection: keep-alive, and Upgrade-Insecure-Requests: 1. Copy the full dictionary from a real DevTools session and refresh the User-Agent every few months.
Beyond headers, 2 fingerprint-layer vectors moved into default anti-bot stacks recently: TLS fingerprinting (JA3, and increasingly JA4/JA4+) on the HTTP side and the Chrome DevTools Protocol Runtime.Enable signal on the browser side. Either one can flag your scraper even when the headers look right.
TLS fingerprinting is based on the exact set of cipher suites, extensions, and ALPN values that your client offers in the TLS handshake. Python Requests and httpx present a fingerprint that real Chrome doesn't send, which makes them more likely to be challenged by Cloudflare and similar vendors. At any volume against a fingerprint-checking target, a practical answer is curl_cffi, which ships with the TLS and HTTP/2 stack from real Chrome and Safari builds, so the fingerprint that a server sees matches a genuine browser at both layers.
TLS handles the HTTP side; the browser itself can also expose your scraper.
The Runtime.Enable signal s another possible automation indicator. Playwright and Puppeteer both call the Chrome DevTools Protocol's Runtime.Enable command to hook JavaScript execution contexts. Some researchers and practitioners have reported browser automation detection techniques involving Chrome DevTools Protocol signals, but detection methods vary by vendor and are not publicly documented in full. Stealth plugins that only patch navigator.webdriver may not address protocol-layer signals. patchright ships its own patched Chromium that is designed not to emit the Runtime.Enable signal.
Beyond both of those, a third shift showed up in anti-bot stacks from recent years onward: per-site behavioral baselines. Cloudflare's per-customer bot defenses and DataDome's intent-based detection compare each request to the target site's own traffic mix rather than a global model. A "perfect" Chrome fingerprint that's statistically rare on that specific site gets flagged for its rarity alone, which means a stealth tool has to look indistinguishable per-site, not just per-browser. Even a clean fingerprint won't help if the IP underneath is the wrong one.
How proxies help
A single IP sending 10,000 requests per hour to one host will usually be rate-limited or blocked. Proxies distribute those requests across multiple IP addresses, helping each connection stay below the target’s per-IP limits.
The main options are datacenter, residential, and mobile proxies. Datacenter IPs are fast but easier to identify, while residential and mobile IPs resemble traffic from real home or cellular users and are better suited to protected targets.
Why use Live Proxies?
Live Proxies provides premium residential and mobile proxy infrastructure for production scraping, SEO monitoring, ad verification, market research, and eCommerce data collection.
Key capabilities include:
- Millions of IPs across 55+ countries, with strong coverage in the US, UK, and Canada
- Private IP allocation that reduces overlap between customers targeting the same websites
- Custom IP allocations for high-volume enterprise projects
- Unlimited concurrent threads with HTTP and SOCKS5 support
- Sticky sessions lasting up to 24 hours
- 24/7 technical support
For example, Live Proxies can privately allocate 50,000 IPs to an enterprise customer scraping Walmart and eBay. Those IPs will not be assigned to another customer working with the same or similar targets, helping preserve their reputation and reduce premature blocks.
Which Live Proxies product should you use?
Rotating residential proxies use real home IPs that change naturally as peers reconnect or receive new addresses from their ISPs. They are suitable for large scraping queues, SERP tracking, price monitoring, and market research.
Static residential proxies use genuine home IPs selected for their long-term stability. They are useful for authenticated sessions, account management, and other tasks requiring a consistent residential identity.
Rotating mobile proxies route requests through real mobile carrier networks. They are best suited to mobile app endpoints, mobile-only content, carrier-specific results, and targets with strict IP reputation checks.
Rotating versus sticky sessions
A rotating session selects another IP from your allocated pool as requests continue. This works well for bulk anonymous scraping.
A sticky session maintains the same IP for up to 24 hours. Use it when pagination, authentication, or a multi-step workflow must remain connected to the same network identity.
Live Proxies uses the standard user:pass@gateway:port format and integrates with Requests, httpx, curl_cffi, Playwright, and other scraping tools.
For browser automation, Chromium’s WebRTC requests can sometimes bypass the proxy and expose the host IP. Add --webrtc-ip-handling-policy=disable_non_proxied_udp to the launch arguments and verify the connection at BrowserLeaks WebRTC before running the scraper in production.
When the page throws a CAPTCHA
Some defenses hand you a challenge instead of blocking you. Once a CAPTCHA (reCAPTCHA, hCaptcha, Cloudflare Turnstile) is in front of you, you have 3 options, in order of cost-effectiveness:
- Switch IP and fingerprint and retry. A fresh residential IP often isn't challenged at all, which is the cheapest path when it works.
- Skip the URL and queue it for review later. No cost, but you don't get the data.
- Route the URL to a CAPTCHA-solving service. $0.50 to $3 per 1,000 solves in 2026, which adds up at volume.
Pick the option that matches your tolerance for cost and latency. Here's what one of these challenges looks like in practice:
Cloudflare Turnstile widget in its 'Verifying...' state on a demo login page. This is the exact widget shape a scraper encounters when a Cloudflare-protected target challenges instead of outright blocking.
CAPTCHAs are one visible form of block. When the target blocks without giving you a challenge, walk the diagnostic ladder below.
When you get blocked: the diagnostic ladder
Your scraper stops getting 200 responses. Walk this list before assuming the target is unscrapable. (The same 6 steps map onto most of the IP-ban recovery work we cover at scale.)
- Check the status code. A 429 is a temporary rate limit, so slow down and respect the Retry-After header. A 403 usually means the request looked wrong on this attempt; rotate IP and try again. A 503 with a Cloudflare HTML body is a JS challenge, so you need a browser path or patchright.
- Switch IPs. Rotating to a different residential IP may resolve some IP-based blocks, but success depends on the target site's detection mechanisms. If you're routing through datacenter proxies against a consumer-grade site, stop tuning headers and switch tiers first. Datacenter-on-consumer-sites is a common stuck-for-days misconfiguration on r/webscraping. Confirm the proxy is connecting and exiting from the expected IP with the Live Proxies proxy tester before assuming the target is the blocker (the tester confirms connectivity, not whether the IP has been burned on your specific target).
- Match the locale. US-English headers paired with a Berlin proxy is an automation signal. Pair the proxy's exit country with the Accept-Language and timezone_id in your Playwright context: browser.new_context(locale="de-DE", timezone_id="Europe/Berlin", extra_http_headers={"Accept-Language": "de-DE,de;q=0.9"}).
- Switch fingerprints. Try a different impersonate label in curl_cffi, or switch from Playwright to patchright (the patched Chromium variant).
- Try the mobile site. Many targets serve simpler markup with weaker defenses to mobile devices. In Playwright, emulate a real phone with one line: browser.new_context(**pw.devices["iPhone 15"]), which sets the mobile User-Agent, viewport, device-pixel-ratio, and touch flag together.
- Slow down further. Cut concurrency in half, double the delay between requests, run for an hour, and see if the target stops blocking. A sensible starting point on a fresh target is 1 concurrent worker with a 2-second delay; double the delay each retry pass.
If none of the 6 steps recovers the scrape, escalate to a higher-trust IP tier. Static residential proxies are generally designed to provide longer-lived IP assignments than rotating residential proxies, although actual session duration depends on the provider and network conditions. They may work where rotating residential proxies do not, but success depends on the target's detection mechanisms. If static residential also fails, the target may be actively defending against automated access. At that point, your options are to abandon the URL, source the data from the site's official API or RSS feed, or buy a licensed dataset.
Building a production scraper
A scraper that runs once on your laptop and a scraper that runs every hour on a schedule are different programs. There are 3 things that change when you scale: pacing matters, monitoring matters, and silent failures become expensive. Here's a single-file template with Retry-After-aware retries, Pydantic validation, structured JSON logging, and idempotent SQLite writes:
"""scraper.py: end-to-end pattern for an API-backed dynamic site.
"""
import json
import logging
import sqlite3
import time
from datetime import datetime, UTC
from curl_cffi.requests import Session
from pydantic import BaseModel, Field, ValidationError
from tenacity import retry, stop_after_attempt, wait_exponential
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("scraper")
class Product(BaseModel):
sku: str
name: str
price: float = Field(ge=0)
in_stock: bool = False # fail-safe: missing key means "not confirmed in stock"
@retry(stop=stop_after_attempt(5),
wait=wait_exponential(multiplier=1, min=2, max=60),
reraise=True)
def fetch_page(session, url, params, proxy=None):
proxies = {"http": proxy, "https": proxy} if proxy else None
resp = session.get(url, params=params, impersonate="chrome146",
proxies=proxies, timeout=20)
if resp.status_code == 429:
# Retry-After is integer seconds or an HTTP-date; handle both.
try:
wait_s = float(resp.headers.get("Retry-After", "5"))
except ValueError:
wait_s = 30 # date variant: conservative fallback
time.sleep(wait_s)
raise RuntimeError("rate_limited")
resp.raise_for_status()
return resp.json()
def upsert(conn, item):
conn.execute("""
INSERT INTO products(sku, name, price, in_stock, scraped_at)
VALUES (:sku, :name, :price, :in_stock, :scraped_at)
ON CONFLICT(sku) DO UPDATE SET
name=excluded.name, price=excluded.price,
in_stock=excluded.in_stock, scraped_at=excluded.scraped_at
""", item)
def main(url, proxy=None, page_delay=0.5):
conn = sqlite3.connect("products.db")
conn.execute("""CREATE TABLE IF NOT EXISTS products (
sku TEXT PRIMARY KEY, name TEXT, price NUMERIC,
in_stock INTEGER, scraped_at TEXT)""")
session = Session()
cursor, total = None, 0
try:
while True:
params = {"limit": 100, **({"after": cursor} if cursor else {})}
t0 = time.time()
data = fetch_page(session, url, params, proxy)
for raw in data.get("items", []):
# The API may include its own scraped_at field; we attach ours after validation.
raw.pop("scraped_at", None)
try:
item = Product(**raw).model_dump()
item["scraped_at"] = datetime.now(UTC).isoformat()
upsert(conn, item)
total += 1
except ValidationError as e:
logger.warning(json.dumps({
"event": "validation_failed",
"sku": raw.get("sku"),
"errors": e.errors(),
}))
conn.commit()
logger.info(json.dumps({
"event": "page_done",
"items": len(data.get("items", [])),
"total": total,
"elapsed_ms": round((time.time() - t0) * 1000),
}))
cursor = data.get("next_cursor")
if not cursor:
break
time.sleep(page_delay)
finally:
session.close()
conn.close()
if __name__ == "__main__":
main("https://api.example.com/v1/products",
proxy=None) # set to "http://USER:[email protected]:8000" once you have one
Point it at a working endpoint and the scraper emits one structured line per page. The output below is from running an adapted version of the template against quotes.toscrape.com/api/quotes. (That API returns quotes, page, and has_next keys instead of the template's generic items and next_cursor, so the changes in main amount to renaming the items references and swapping the cursor-based pagination check for a has_next / page one.)
{"event": "page_done", "items": 10, "total": 10, "elapsed_ms": 897}
{"event": "page_done", "items": 10, "total": 20, "elapsed_ms": 314}
{"event": "page_done", "items": 10, "total": 30, "elapsed_ms": 297}
And the same logs in a real terminal:
Terminal output from running the production scraper template against quotes.toscrape.com/api/quotes. 3 structured JSON lines, 1 per page, with total incrementing and elapsed_ms reflecting real network timing.
The structured logs come out one grep-able JSON line per page, so a 2 a.m. incident becomes a one-line query ("items": 0 on the most recent line points to an API break; rising elapsed_ms points to the target throttling).
Validation failures need a threshold rule in addition to the per-record logging: the template logs each validation_failed event, and the schema-drift halt below is the rule you bolt on. If more than 2% of records in a run fail Pydantic validation, halt the scrape and alert; a site that breaks one schema in five usually broke its whole API, and continuing to write corrupted rows is worse than stopping.
For a JS-rendered target with no API, replace fetch_page with a Playwright helper that uses expect_response(), and keep everything else identical. The validation, logging, retry, and storage patterns don't care whether the JSON came from curl_cffi or from Playwright's network listener.
Adapting the template
Before you run the template on your own target, make 2 essential edits and 1 conditional one. Change the url in if name == "main" to your endpoint, and rewrite the Product Pydantic model so its fields match the JSON shape that you saw in DevTools (otherwise every record will hit ValidationError and nothing will land). For a first smoke test you can leave proxy=None; replace it with your basic-auth URL when you're ready to route through a proxy. To confirm rows actually landed after the run, query the SQLite output directly: sqlite3 products.db "SELECT name, price FROM products LIMIT 5".
Patterns to add yourself
Once that runs cleanly, there are 2 production extensions left for you to add.
First, a schema-drift halt: increment a validation_failed counter inside the except ValidationError block and, after the loop exits, run if total and validation_failed / total > 0.02: logger.error(json.dumps({"event": "schema_drift", "failed": validation_failed, "total": total})); sys.exit(1).
Second, an outer crash guard: wrap the while True: body in a try / except Exception as e that emits a final {"event": "scrape_failed", "error": str(e)} line and exits non-zero. This makes a 5-retry exhaustion from fetch_page appear as a structured failure that your scheduler can detect rather than an uncaught traceback.
Operational pitfalls: memory drift and Docker
Beyond the code patterns, the Playwright variant of this template adds 2 operational concerns.
First, there's Chromium memory drift: most open Playwright-Python memory-leak issues come from long-lived BrowserContext objects and event listeners that get attached but never removed. Recycle the context every few hundred pages with a pattern like if pages_done % 200 == 0: context.close(); context = browser.new_context(**ctx_kwargs). Call page.remove_listener() for every page.on() that you attach and page.unroute() for every page.route(). Prefer killing and respawning the whole browser at a memory threshold over running it forever.
Second, if you Dockerize Playwright, launch the container with –init –ipc=host –shm-size=2g; the default 64MB /dev/shm is a common cause of silent Chromium crashes in containers.
Deployment options
Once the patterns and pitfalls are handled, the deployment question is where the scraper runs. This single-file template runs fine under cron on a VPS or as a GitHub Actions scheduled job. For sustained crawls beyond one machine, swap cron for a Kubernetes CronJob with scrapy-redis or Celery.
For JS-heavy or heavily-fingerprinted targets, recent pattern is to put Playwright on dedicated workers behind a residential pool rather than self-hosting on a single VPS. Managed browser runtimes exist for teams that prefer to rent that layer, but the proxy layer, not the browser layer, is what usually breaks first at scale. The validation, retry, and storage skeleton stays the same; only the runtime changes.
Bottom line
Scraping dynamic content reliably depends on fingerprinting, pacing, and retries. All of this work compounds on the IP pool underneath it. On a clean, privately-allocated pool, every hour you spend on that work goes further, because the IP is not already burned.
Our rotating residential proxies allocate IPs privately per customer to reduce overlap with other scrapers on the same targets, and integrate with Requests, httpx, curl_cffi, and Playwright through standard HTTP basic auth.
FAQs
How do I scrape dynamic content in Python without Selenium?
Use httpx or Requests against the page's backing API or its embedded JSON instead of rendering the page. If the data only appears after JavaScript execution and there's no API to call, Playwright is a modern alternative, with comparable browser automation capability, faster startup, and built-in auto-waiting.
How do I handle dynamic content when web scraping infinite scroll pages?
Build a stop rule based on item count stability, not on time. Scroll, wait for the next batch to load, count items, and repeat; if the count is unchanged for 3 consecutive scrolls, you've reached the end. Always set a hard cap on iterations so a UI bug can't trap your scraper in a loop.
How do I scrape dynamic content that only appears after I click or hover?
Drive the interaction with browser automation, then read the result. In Playwright, call page.click() or page.hover() on the element, wait for the request it triggers or the element it reveals, and then read the content. If you can see the network call that the click fires in the DevTools Network tab, you can skip the browser and call that endpoint directly with httpx.
Is scraping dynamic content legal?
It depends on what you scrape and where you operate. Courts in the United States have generally treated public data more permissively than data behind a login, which can raise unauthorized-access concerns. The main risk factors are the site's Terms of Service, the copyright on the content, and personal-data laws such as GDPR and CCPA. This is general information, not legal advice.
