Many ecommerce stores expose structured JSON through public endpoints, embedded page data, or internal API calls, but availability varies by platform and store configuration. Large marketplaces commonly use anti-bot systems that evaluate TLS fingerprints alongside IP reputation, cookies, browser signals, request patterns, and user behavior. So when ecommerce web scraping fails in 2026, the site blocks the request before the parser runs. Access blocks are a common cause of scraping failures, but parsers can also break when selectors, page structures, or API schemas change.
Automated traffic now makes up more than 53% of all web traffic, up from 51% the year before, according to the 2026 Imperva Bad Bot Report. Retailers meet that automation with anti-bot layers that stop most scrapers before they reach a single product. Through 2026, reliable ecommerce scraping depends less on clever parsing and more on getting past that block.
TL;DR
- Ecommerce web scraping in 2026 means handling site blocks rather than fixing parsers.
- Try plain HTTP and a site's JSON API first. Use a headless browser only when nothing else works.
- Many stores expose a public products endpoint that returns prices, variants, and stock as JSON.
- Store the raw text alongside the parsed values. Use minor units for price and a boolean for in-stock.
- Pace requests, reuse sessions, and add proxies when scale or geo pricing requires it.
- Validate every crawl. Check missing prices, duplicate IDs, and zero-price or out-of-stock items before you trust the data.
What is ecommerce web scraping and why use it
Ecommerce web scraping is the automated collection of product data such as prices, variants, stock, and reviews from online store pages and the APIs behind them. A retailer's catalog is one of the few large, public, frequently-changing datasets you can collect without an account.
Teams use ecommerce web scraping to answer questions the storefront does not surface:
- Price monitoring tracks how a competitor's prices change daily.
- Catalog tracking detects new products, removed products, and category changes.
- Trend research shows which products get the most reviews or sell out fast.
- Product matching maps the same item across several stores by identifier.
- Market intelligence feeds pricing and assortment data into planning.
Is it legal to scrape data from ecommerce websites?
Public category and product pages, including their JSON endpoints, carry lower legal risk than pages behind a login. Some U.S. court decisions have found that accessing publicly available information may not violate the CFAA in certain circumstances. However, terms of service, privacy laws, copyright, and state laws may still create legal risk. Anything behind a login comes under the site's account rules, which adds more legal risk.
Before any crawl, run through these 4 checks:
- Terms of service may restrict automated access even on public pages.
- The robots.txt protocol lists which paths a site disallows for crawlers.
- Privacy laws such as the GDPR (EU) and CCPA (California) apply when data identifies a person.
- Copyright applies to creative content like product descriptions, images, and reviews, particularly when you republish or redistribute.
These rules keep scraping ethical:
- Keep request rates low and steady.
- Drop personal-data fields you don't need. Hash or redact the ones you keep, so reports still work without the raw value.
- Store the source URL and a timestamp with every record.
- If you keep personal data, respond to removal requests quickly.
This is not legal advice. When a project is commercial or covers many sites, check the specific site's terms and your local rules with a qualified professional.
What ecommerce data should you scrape first
Capture stable identifiers first. A product ID, a variant SKU, and the canonical URL let you match the same item across pages and across crawls. The match still works even when the title or price changes.
Group your target fields by page type:
| Page type | Fields worth capturing |
|---|---|
| Category page | Product name, product URL, list price, thumbnail, rating, page number |
| Product page | Product ID, variant SKU, canonical URL, price, discount, stock, description, specs, images |
| Review page | Review ID, rating, review text, date, helpful votes, sort order |
Pull the identifier and canonical URL on every page first. If you skip them, the same product can appear as 2 separate rows, and you'll track its price history in 2 separate places.
For cross-store matching (the "Product matching" use case from above), also capture universal codes like UPC, EAN, or ASIN when the page exposes them
How to scrape data from an ecommerce website in 10 minutes
Many direct-to-consumer brands run on Shopify. Many Shopify stores expose product data through endpoints such as /products.json, but stores may restrict access or return different fields.The endpoint follows Shopify's documented product JSON format. So when that endpoint is available, calling it directly is usually the fastest way to scrape the store. The examples below run against tentree, a real Shopify store.
Quick setup
Create a virtual environment and install the core stack:
python3 -m venv .venv
source .venv/bin/activate
pip install httpx==0.28.1 beautifulsoup4==4.15.0 lxml==6.1.1 pandas==3.0.3 python-dotenv==1.2.2
*On Windows, use .venv\Scripts\activate instead of the source line.
Minimal working scraper
This script calls the public product feed, reads the first 5 products, and writes products.csv:
import csv
import httpx
# Many Shopify stores expose a public products.json endpoint.
STORE = "https://www.tentree.com"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0 Safari/537.36"
)
}
def fetch_products(limit=5):
url = f"{STORE}/products.json"
resp = httpx.get(
url,
params={"limit": limit},
headers=HEADERS,
timeout=25,
follow_redirects=True,
)
resp.raise_for_status()
return resp.json()["products"]
def main():
rows = []
for p in fetch_products(limit=5):
variants = p["variants"]
in_stock = any(v["available"] for v in variants)
rows.append(
{
"product_id": p["id"],
"title": p["title"],
"from_price": variants[0]["price"],
"in_stock": in_stock,
"url": f"{STORE}/products/{p['handle']}",
}
)
with open("products.csv", "w", newline="", encoding="utf-8") as f:
fields = ["product_id", "title", "from_price", "in_stock", "url"]
writer = csv.DictWriter(f, fieldnames=fields)
writer.writeheader()
writer.writerows(rows)
for row in rows:
print(row["title"], "|", row["from_price"], "|", row["in_stock"])
if __name__ == "__main__":
main()
A product is in stock when at least one variant is available. Running the script produces this output:
InMotion Apex Hat | 45.00 | True
Haiti Striped Palm Zip Hoodie | 98.00 | True
Haiti Palm Logo Relaxed Hoodie | 88.00 | True
Haiti Palm Logo Relaxed Hoodie | 88.00 | True
Haiti Palm Logo Relaxed Hoodie | 88.00 | True
The CSV holds the same rows plus product_id and the product URL. The 3 "Haiti Palm Logo" rows are 3 separate listings sharing a title, with distinct product_id values in the CSV.
How to scrape ecommerce prices and discounts accurately
A product has a list price and a sale price. Some products also have a unit price (per 100g, per liter), or a coupon price that only applies at checkout. A JSON feed returns decimals you can parse directly, while rendered HTML returns strings like $1,299.00 that need cleaning first. The parser below handles both formats.
The same product can show a different price based on the visitor's country, currency, or logged-in status. Tax display also varies. Many EU stores show prices including VAT, while US stores typically exclude sales tax until checkout. Store the raw value exactly as returned, and a parsed numeric value next to it for the audit trail.
Price parsing basics
Parse money into an integer of minor units (cents). This module parses a price from the JSON feed, a price from rendered HTML, and a discount, all against live data.
import re
import httpx
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
CURRENCY_SYMBOLS = {"£": "GBP", "$": "USD", "€": "EUR", "¥": "JPY"}
STORE = "https://www.tentree.com"
def to_cents(price_str):
"""Convert a clean decimal string like '98.00' into integer cents."""
if price_str is None or price_str == "":
return None
return round(float(price_str) * 100)
def parse_price(raw):
"""Parse a messy price string from rendered HTML into (cents, currency)."""
if not raw:
return None, None
currency = next((c for s, c in CURRENCY_SYMBOLS.items() if s in raw), None)
match = re.search(r"\d[\d,]*\.?\d*", raw) # first number, drop separators
if not match:
return None, currency
return round(float(match.group(0).replace(",", "")) * 100), currency
def parse_discount(price, compare_at_price):
"""Percent off from a sale price and its original (compare-at) price.
compare_at_price is empty when the product is not on sale.
"""
sale, original = to_cents(price), to_cents(compare_at_price)
if not original or not sale or sale >= original:
return None
return round((1 - sale / original) * 100)
def find_sale_variant(products):
for product in products:
for v in product["variants"]:
cap = v.get("compare_at_price")
if cap and cap not in ("", v["price"]):
return product["title"], v["price"], cap
return None
if __name__ == "__main__":
products = httpx.get(
f"{STORE}/products.json",
params={"limit": 250},
headers=HEADERS,
timeout=30,
follow_redirects=True,
).json()["products"]
# JSON price from the real feed.
json_price = products[0]["variants"][0]["price"]
print("json price ", repr(json_price), "->", to_cents(json_price))
# Price scraped from a real rendered product page.
page = httpx.get(
"https://www.deathwishcoffee.com/products/death-wish-coffee",
headers=HEADERS,
timeout=25,
follow_redirects=True,
).text
raw_price = re.search(r"\$\d[\d,]*\.\d{2}", page).group(0)
print("html price ", repr(raw_price), "->", parse_price(raw_price))
# Real discount, if any product is on sale right now.
sale = find_sale_variant(products)
if sale:
title, price, original = sale
print(f"discount {title[:24]!r}: {parse_discount(price, original)}% off")
else:
print("discount no product on sale right now")
It parses a feed price, a price scraped from rendered HTML, and a live discount.
json price '45.00' -> 4500
html price '$19.99' -> (1999, 'USD')
discount 'Alpine Oversized Crew': 20% off
The parser treats a comma as a thousands separator. A locale that writes 19,90 instead of 19.90 parses to 100× the real price. So detect the site's locale once and add a locale-aware branch, rather than guessing per value.
Discount and promotion fields
Keep discount signals in their own columns. On a Shopify feed, price is the current price and compare_at_price is the original. The field is empty when nothing is on sale. The parse_discount function above turns those 2 into a percent, returning None when there is no markdown.
Store percent_off, the original price, and any promo label as distinct columns. That lets you later answer questions like "which categories have the biggest discounts on weekends" without re-parsing anything.
How to scrape ecommerce product pages for full details
A product feed entry has more fields than the list view. It includes the full description, the option list, every variant with its own price and stock, and the images. Build one consistent schema and map every store into it. 2 stores that name a field differently still produce the same shape.
Product identifiers
A product ID, a per-variant SKU, and the canonical URL are your keys for matching and deduping. The canonical URL drops tracking parameters, so 2 links to the same product collapse into one record. This scraper pulls the identifiers and the rest of the detail fields in one pass:
import json
import re
import httpx
STORE = "https://www.tentree.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
def fetch_first_product():
# The products.json list returns complete product objects, including the
# per-variant "available" flag that the single-product endpoint omits.
resp = httpx.get(
f"{STORE}/products.json",
params={"limit": 1},
headers=HEADERS,
timeout=25,
follow_redirects=True,
)
resp.raise_for_status()
return resp.json()["products"][0]
def strip_html(html):
return re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", html or "")).strip()
def shorten(text, limit):
"""Trim to a word boundary so the sample never cuts mid-word."""
if len(text) <= limit:
return text
return text[:limit].rsplit(" ", 1)[0] + "…"
def parse_product(p):
variants = p["variants"]
return {
"product_id": p["id"], # stable parent key
"title": p["title"],
"canonical_url": f"{STORE}/products/{p['handle']}",
"vendor": p["vendor"],
"product_type": p["product_type"],
"option_names": [o["name"] for o in p["options"]],
"from_price": variants[0]["price"],
"in_stock": any(v["available"] for v in variants),
"variant_count": len(variants),
"first_sku": variants[0]["sku"], # per-variant identifier
"image_url": p["images"][0]["src"] if p["images"] else None,
"description": shorten(strip_html(p.get("body_html")), 120),
}
if __name__ == "__main__":
product = fetch_first_product()
print(json.dumps(parse_product(product), indent=2, ensure_ascii=False))
The output is one record, ready to store:
{ "product_id": 8360518516922,
"title": "Haiti Striped Palm Zip Hoodie",
"canonical_url": "https://www.tentree.com/products/haiti-striped-palm-zip-hoodie-meteorite-black-creek-stone",
"vendor": "tentree",
"product_type": "Mens",
"option_names": [
"Color",
"Size"
],
"from_price": "98.00",
"in_stock": true,
"variant_count": 5,
"first_sku": "TCM6545-6164-S",
"image_url": "https://cdn.shopify.com/s/files/1/2341/3995/files/Black-Regular-Fit-Graphic-Hoodie-TCM6545-6164_2.jpg?v=1774030376",
"description": "Each item sold from the Haiti capsule, including this zip hoodie, will plant trees in Haiti to help restore ecosystems…"
}
Use product_id as the primary key for the product across every crawl. The variant SKU is the key for each size or color variant.
Specifications and attributes
The attribute fields vary by store. Here product_type and vendor are single values, and many stores add a tags list with material, gender, and category hints. Capture them however they appear, and keep both raw and normalized versions when you map them into your own column names.
Normalization can collapse 2 source labels into one of your keys. When that happens, the raw copy tells you which value was which, so you don't lose data to a cleanup step.
Images and media
Store image URLs and alt text. Don't store the image bytes. Downloading every full-size image makes a crawl slow and storage-heavy. Most analysis only needs the URL. The scraper above keeps the first image_url and leaves the file on the store's CDN.
If a project needs the files (such as visual matching), download them in a separate, rate-limited pass that reads the stored URLs. That keeps the main crawl fast and lets you throttle the image fetch on its own.
For color variants, capture the per-variant image too. The parent image shows the default color only.
How to scrape ecommerce variants like size and color
Size and color drive both price and stock independently of the parent product. A hoodie can be in stock in medium black and sold out in small white, at 2 different prices. One price per product loses that detail.
Capture a variant matrix. Each row holds one combination, with the option values, the variant SKU, its price, and its availability. Store it as a separate variants table keyed by the parent product ID, or as nested rows on the product record.
A Shopify variant carries option1, option2, and option3, which map to the product's option names (here Color and Size). This builder flattens them into one row each. A stop rule skips products with too many options.
import httpx
STORE = "https://www.tentree.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
MAX_VARIANTS = 200 # stop rule: skip products with huge option trees
def fetch_first_product():
resp = httpx.get(
f"{STORE}/products.json",
params={"limit": 1},
headers=HEADERS,
timeout=25,
follow_redirects=True,
)
resp.raise_for_status()
return resp.json()["products"][0]
def variant_matrix(product):
"""Flatten a product's size and color options into one row each."""
option_names = [o["name"] for o in product["options"]]
variants = product["variants"]
if len(variants) > MAX_VARIANTS:
return [] # too many; pull these per-variant instead
rows = []
for v in variants:
row = {
"sku": v["sku"],
"price": v["price"],
"available": v["available"],
}
for i, name in enumerate(option_names, start=1):
row[name.lower()] = v.get(f"option{i}")
rows.append(row)
return rows
if __name__ == "__main__":
product = fetch_first_product()
rows = variant_matrix(product)
print(f"title: {product['title']}")
print(f"options: {[o['name'] for o in product['options']]}")
print(f"variants: {len(rows)}")
for row in rows[:3]:
print(row)
Each variant becomes a row with its own SKU, price, and stock:
title: Haiti Striped Palm Zip Hoodie
options: ['Color', 'Size']
variants: 5
{'sku': 'TCM6545-6164-S', 'price': '98.00', 'available': True, 'color': 'METEORITE BLACK CREEK STONE', 'size': 'S'}
{'sku': 'TCM6545-6164-M', 'price': '98.00', 'available': True, 'color': 'METEORITE BLACK CREEK STONE', 'size': 'M'}
{'sku': 'TCM6545-6164-L', 'price': '98.00', 'available': True, 'color': 'METEORITE BLACK CREEK STONE', 'size': 'L'}
This hoodie has 5 sizes in one color. Multi-color products would show both color and size varying across rows.
If a product passes the cap, pull its variants from the store's per-variant data instead of expanding every combination. The stop rule prevents one outlier product from blocking the whole run.
How to scrape ecommerce stock availability and delivery
Stores expose stock as in stock, out of stock, limited stock, backorder, or pre-order. A JSON feed usually gives a boolean per variant. A rendered page gives a text label and a button state.
Stock and delivery signals
On a JSON feed, read the per-variant available flag and roll it up to the product.
import httpx
STORE = "https://www.tentree.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
def fetch_products(limit=10):
resp = httpx.get(
f"{STORE}/products.json",
params={"limit": limit},
headers=HEADERS,
timeout=25,
follow_redirects=True,
)
resp.raise_for_status()
return resp.json()["products"]
def product_stock(variants):
"""Roll per-variant availability up to a product-level summary."""
in_stock = [v for v in variants if v["available"]]
return {
"in_stock": len(in_stock) > 0,
"variants_total": len(variants),
"variants_in_stock": len(in_stock),
}
if __name__ == "__main__":
for product in fetch_products(limit=3):
summary = product_stock(product["variants"])
print(product["title"][:38], "->", summary)
The rollup runs on real products straight from the feed.
InMotion Apex Hat -> {'in_stock': True, 'variants_total': 1, 'variants_in_stock': 1}
Haiti Striped Palm Zip Hoodie -> {'in_stock': True, 'variants_total': 5, 'variants_in_stock': 5}
Haiti Palm Logo Relaxed Hoodie -> {'in_stock': True, 'variants_total': 5, 'variants_in_stock': 5}
When a store has no feed and you scrape the rendered page instead, read both the text label and the button state. A disabled add-to-cart button, or a label like "Only 2 left", is sometimes a clearer signal than a plain "In stock".
A JSON feed gives a boolean, which tells you a variant is available but not how many are left. When you need the quantity, the rendered page or a separate inventory call is the source of truth.
Delivery and pickup details are often location-based. Capture delivery dates and shipping cost labels only when they are visible, and keep them optional. A scraper that requires a delivery estimate breaks when a product omits it.
Store delivery fields next to the location context you sent (postcode, country). Without that context, a saved delivery date means little, because the same product shows different dates to different regions.
How to scrape ecommerce reviews and ratings
Reviews rarely live in the product feed. Most stores fetch them from a separate reviews app through its own API, which the product page calls after it loads. Find the request that returns the review JSON in the Network tab and reproduce that call directly.
For products with thousands of reviews, full collection is slow and rarely needed. Take a sample instead. Use the most recent N reviews, the most helpful N, or a fixed time window. Record which sort order you used, because "top" reviews and "newest" reviews show different patterns.
Capture the verified-purchase flag when the API exposes one. Verified reviews carry different weight than anonymous ones for analysis.
Review pagination
Reviews use page-based pagination (?page=2) or cursor-based pagination (a token that points to the next batch). Walk the pages until there is no next link or you hit your sample limit. Store a stable key so re-runs do not double-count. Use the review ID when one exists, or a hash of the text plus date when it doesn't.
import hashlib
import re
import unicodedata
import httpx
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
def fetch_reviews(product_id=1):
# Reviews live in a separate API. Public stores like this one embed them in
# the product; real shops expose a reviews app's API you find in Network.
url = f"https://dummyjson.com/products/{product_id}"
resp = httpx.get(url, headers=HEADERS, timeout=20)
resp.raise_for_status()
return resp.json().get("reviews", [])
def clean_review(text):
"""Normalize whitespace and Unicode in a raw review string."""
if not text:
return ""
text = unicodedata.normalize("NFKC", text)
return re.sub(r"\s+", " ", text).strip()
def review_key(review):
"""Use the review ID, or a stable hash of text + date when absent."""
if review.get("id"):
return str(review["id"])
raw = f"{review.get('comment', '')}|{review.get('date', '')}"
return hashlib.md5(raw.encode("utf-8")).hexdigest()
if __name__ == "__main__":
for review in fetch_reviews(product_id=1):
print(review_key(review)[:12], "|", clean_review(review["comment"]))
The script fetches reviews from the API, then cleans and keys each one.
85979f842ae8 | Would not recommend!
a038a91e4ac1 | Very satisfied!
871fc0e894ee | Highly impressed!
Review pages reorder constantly. The same review can appear on page 1 today and page 2 tomorrow. A stable key keeps re-runs from double-counting it.
Review cleaning
Raw review text carries extra whitespace, line breaks, and repeated boilerplate. The clean_review function above normalizes Unicode and collapses whitespace so search and sentiment analysis work cleanly.
Keep a language code with each review when the API exposes one. Mixed-language reviews skew sentiment scores. A language tag lets you filter or route them later.
Reviews also contain personal data such as a reviewer's name and email. The minimize-and-redact rule from the legality section applies here.
import hashlib
import json
import httpx
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
DROP = {"reviewerEmail"} # don't store it at all
HASH = {"reviewerName"} # keep a token so reports work, not the raw name
def redact(record):
"""Drop or hash personal fields, keeping non-personal ones intact."""
clean = {}
for key, value in record.items():
if key in DROP:
continue
if key in HASH and value:
digest = hashlib.sha256(str(value).encode()).hexdigest()[:12]
clean[f"{key}_token"] = digest
else:
clean[key] = value
return clean
def fetch_reviews(product_id=1):
url = f"https://dummyjson.com/products/{product_id}"
return httpx.get(url, headers=HEADERS, timeout=20).json().get("reviews", [])
if __name__ == "__main__":
review = fetch_reviews()[0]
print("raw: ", json.dumps(review))
print("redacted:", json.dumps(redact(review)))
The email is gone and the name is a token, but the rating, comment, and date survive.
raw: {"rating": 3, "comment": "Would not recommend!", "date": "2025-04-30T09:41:02.053Z", "reviewerName": "Eleanor Collins", "reviewerEmail": "[email protected]"}
redacted: {"rating": 3, "comment": "Would not recommend!", "date": "2025-04-30T09:41:02.053Z", "reviewerName_token": "b0f1828ce4d5"}
The same redact step applies to marketplace seller names and any other personal field you pass through.
The function above only checks top-level keys. Extend it to walk recursively when your API wraps reviewer data inside a user or author object.
The comment body itself can carry personal data even after name and email are removed. A review can name a workplace, a location, a health condition, or another reviewer. For public sharing or long-term storage, extend the same minimize-and-redact rule to text fields.
AI-generated reviews are another concern. LLMs produce reviews at scale, both from retailers seeding to boost ranking and from bad actors planting negative reviews to harm competitors.
Detection here is probability-based, not certain. A downstream classifier flags suspiciously generic phrasing or unnaturally fast bursts of similar reviews. That catches the obvious cases, which is enough to inform validation. A hard filter would drop legitimate reviews.
How to scrape dynamic ecommerce websites when content loads with JavaScript
When product data is not in the initial HTML, capture the JSON API the page calls or render the page with a headless browser. The API route is faster and lighter, so try it first.
Before any browser code, request the raw page with httpx and search it for a price. If the price is already there, you don't need a browser at all.
Find and use the site API
In the Network tab, filter to Fetch/XHR while the page loads. Product data usually arrives as JSON. The public products.json feed from earlier is one example.
Tentree's public /products.json endpoint returns the full catalog in one JSON response. The Preview tree shows each product's title, vendor, variants, and tags.
Reproduce it with the right parameters and parse the JSON directly:
import httpx
# The Network tab on this product page shows the storefront calling its own
# products JSON. You can reproduce that call directly and skip the HTML.
STORE = "https://www.tentree.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
def fetch_products(limit=10, page=1):
resp = httpx.get(
f"{STORE}/products.json",
params={"limit": limit, "page": page},
headers=HEADERS,
timeout=25,
follow_redirects=True,
)
resp.raise_for_status()
return [
{
"id": p["id"],
"title": p["title"],
"from_price": p["variants"][0]["price"],
"in_stock": any(v["available"] for v in p["variants"]),
}
for p in resp.json()["products"]
]
if __name__ == "__main__":
for row in fetch_products(limit=3):
print(row)
The JSON parses straight into records:
{'id': 8360519663802, 'title': 'InMotion Apex Hat', 'from_price': '45.00', 'in_stock': True}
{'id': 8360518516922, 'title': 'Haiti Striped Palm Zip Hoodie', 'from_price': '98.00', 'in_stock': True}
{'id': 8360518353082, 'title': 'Haiti Palm Logo Relaxed Hoodie', 'from_price': '88.00', 'in_stock': True}
GraphQL endpoints (commonly at /graphql) follow the same pattern with POST instead of GET. Copy the query body from the Network tab and reproduce the call.
Marketplaces share a similar pattern internally. Walmart currently embeds its product data in a __ NEXT_DATA __script tag, the standard payload Next.js server-rendered pages emit. The container path is shifting as Walmart migrates to App Router across the site. Most large stores have an internal API you can find in the Network tab. For retailer-specific walkthroughs, see how to scrape Walmart and how to scrape eBay.
Amazon, Walmart, and other defended marketplaces may require browser automation, realistic sessions, and residential proxies at scale. The required setup depends on the target, location, and request rate.
Parse embedded data from the HTML
Most product pages carry structured data as JSON-LD, a script tag that search engines read, even when the store has no JSON feed. Death Wish Coffee carries JSON-LD on every product page.
The same JSON-LD block search engines read for rich results carries @type Product, name, brand, and offers – the fields the parser extracts.
Fetch the page with httpx, find the script type="application/ld+json" tag with BeautifulSoup, and parse the JSON inside.
import json
import httpx
from bs4 import BeautifulSoup
# Product pages embed structured data as JSON-LD. When a store has no
# JSON feed, parse it from the HTML with BeautifulSoup + lxml.
URL = "https://www.deathwishcoffee.com/products/death-wish-coffee"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0 Safari/537.36"
)
}
def extract_product(html):
soup = BeautifulSoup(html, "lxml")
for tag in soup.find_all("script", type="application/ld+json"):
try:
data = json.loads(tag.string or "{}")
except json.JSONDecodeError:
continue
for item in data if isinstance(data, list) else [data]:
if isinstance(item, dict) and item.get("@type") == "Product":
offer = item.get("offers", {})
if isinstance(offer, list):
offer = offer[0]
return {
"title": item.get("name"),
"price": offer.get("price"),
"currency": offer.get("priceCurrency"),
"in_stock": "InStock" in (offer.get("availability") or ""),
}
return None
if __name__ == "__main__":
resp = httpx.get(URL, headers=HEADERS, timeout=25, follow_redirects=True)
resp.raise_for_status()
print(extract_product(resp.text))
The parser pulls a record out of the page's markup:
{'title': 'Dark Roast Coffee', 'price': 19.99, 'currency': 'USD', 'in_stock': True}
JSON-LD follows a published schema, so the same data parsing logic works across many stores with no per-site selectors. When a store omits JSON-LD, fall back to CSS selectors on the product elements, or to a headless browser.
Use an LLM to extract fields when markup is messy
Some stores have inconsistent markup, hand-rolled HTML with no JSON-LD, or fields scattered across DOM positions that change weekly. An LLM with a fixed JSON schema can extract fields directly, without per-site selectors. Budget around $0.005 to $0.02 per page at current Claude API pricing. That cost keeps long-tail coverage affordable. When you scale up on one site, a per-site parser becomes cheaper than calling the LLM every time.
The LLM only returns fields and types defined in the schema, so the output is validated at the API layer rather than after parsing free text. Scope the input to the main element to keep token cost down. Then send the text to Claude with a save_product tool whose schema is the only allowed output shape. Install anthropic with pip install anthropic==0.50.0.
import json
import anthropic
import httpx
from bs4 import BeautifulSoup
# When markup is messy or per-store, an LLM can extract fields directly.
# Requires ANTHROPIC_API_KEY in the environment.
URL = "https://www.deathwishcoffee.com/products/death-wish-coffee"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
PRODUCT_SCHEMA = {
"type": "object",
"properties": {
"title": {"type": "string"},
"price": {"type": "number"},
"currency": {"type": "string"},
"in_stock": {"type": "boolean"},
},
"required": ["title", "price", "currency", "in_stock"],
}
def extract_with_llm(html):
"""Send the page text to Claude with a tool whose schema is the output shape."""
soup = BeautifulSoup(html, "lxml")
main = soup.find("main") or soup.body
snippet = main.get_text(separator=" ", strip=True)[:4000]
client = anthropic.Anthropic()
resp = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=512,
tools=[{
"name": "save_product",
"description": "Save the extracted product fields.",
"input_schema": PRODUCT_SCHEMA,
}],
tool_choice={"type": "tool", "name": "save_product"},
messages=[{
"role": "user",
"content": f"Extract product fields from this page:\n\n{snippet}",
}],
)
for block in resp.content:
if block.type == "tool_use":
return block.input
return None
if __name__ == "__main__":
resp = httpx.get(URL, headers=HEADERS, timeout=25, follow_redirects=True)
resp.raise_for_status()
print(json.dumps(extract_with_llm(resp.text), indent=2))
A real product page returns a clean record matching the schema:
{ "title": "Dark Roast Coffee",
"price": 19.99,
"currency": "USD",
"in_stock": true
}
The per-page cost dominates above a few thousand pages a day. For high-volume pipelines, use the LLM once to generate per-site extraction code instead of calling it on every page. In practice this cuts token cost substantially versus calling the LLM on every page. An LLM can return a confident wrong answer with no error, so keep a validation step that flags rows whose fields fall outside expected ranges.
Escalate automatically when a source fails
Relying on one method breaks the crawl when that method fails. Wire them in order.
Each arrow fires only when the tier above fails. Cost rises with each escalation, so the crawl runs on the cheapest path that works.
Try the fast JSON API first. Fall back to parsing the page's JSON-LD when the API is blocked or does not have the product. The crawl keeps running on the slower path instead of failing.
import json
import httpx
from bs4 import BeautifulSoup
STORE = "https://www.deathwishcoffee.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
def _get(url, **params):
resp = httpx.get(
url, params=params, headers=HEADERS, timeout=30, follow_redirects=True
)
resp.raise_for_status()
return resp
def to_cents(price):
"""Both tiers normalize to integer cents, so the output type is stable."""
return round(float(price) * 100)
def from_api(handle):
resp = _get(f"{STORE}/products.json", limit=250)
for product in resp.json()["products"]:
if product["handle"] == handle:
return {
"title": product["title"],
"price_cents": to_cents(product["variants"][0]["price"]),
"source": "products.json",
}
raise LookupError("handle not in feed")
def from_jsonld(handle):
resp = _get(f"{STORE}/products/{handle}")
soup = BeautifulSoup(resp.text, "lxml")
for tag in soup.find_all("script", type="application/ld+json"):
try:
data = json.loads(tag.string or "{}")
except json.JSONDecodeError:
continue
for item in data if isinstance(data, list) else [data]:
if isinstance(item, dict) and item.get("@type") == "Product":
offer = item.get("offers", {})
if isinstance(offer, list):
offer = offer[0]
return {
"title": item.get("name"),
"price_cents": to_cents(offer.get("price")),
"source": "json-ld",
}
raise LookupError("no product JSON-LD on the page")
def get_product(handle, prefer_api=True):
"""Try the fast API; fall back to the page when it's blocked or absent."""
if prefer_api:
try:
return from_api(handle)
except (httpx.HTTPError, LookupError):
pass # API blocked or missing the product; escalate
return from_jsonld(handle)
if __name__ == "__main__":
print("api works: ", get_product("death-wish-coffee"))
print("api skipped: ", get_product("death-wish-coffee", prefer_api=False))
Both tiers return the same product in the same shape. The try/except triggers the fallback when the API is blocked or missing the product.
api works: {'title': 'Dark Roast Coffee', 'price_cents': 1999, 'source': 'products.json'}
api skipped: {'title': 'Dark Roast Coffee', 'price_cents': 1999, 'source': 'json-ld'}
The same pattern extends to deeper tiers. If the page itself is blocked, the next fallback is a headless browser. The tier below that is a browser paired with residential proxies. Try the cheapest method first, and escalate only when it fails.
Use Playwright or Selenium for dynamic pages
Use a headless browser when no usable API exists, or when the page needs interaction such as infinite scroll or filter clicks. A browser is slower and heavier, so limit its use. Install Playwright and a Chromium build with pip install playwright==1.60.0 and playwright install chromium. This script renders a real product page, waits for the price to appear, then reads it:
import re
from playwright.sync_api import sync_playwright
# Use a browser only when plain HTTP can't get the data. Use an explicit
# wait and a stop rule, and scope extraction to the product's own price, not
# the first price on the page.
URL = (
"https://www.tentree.com/products/"
"haiti-striped-palm-zip-hoodie-meteorite-black-creek-stone"
)
def scrape_rendered(url):
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url, wait_until="domcontentloaded", timeout=45000)
# Explicit wait: continue once a price renders, fail fast if it never does.
page.wait_for_function(
"() => /\\$\\d+\\.\\d{2}/.test(document.body.innerText)",
timeout=20000,
)
title = page.title()
html = page.content()
browser.close()
# The product's canonical price is in the embedded JSON, in minor units.
match = re.search(r'"price"\s*:\s*(\d{3,7})', html)
price = f"${int(match.group(1)) / 100:.2f}" if match else None
return {"title": title, "price": price}
if __name__ == "__main__":
print(scrape_rendered(URL))
It returns this output:
{'title': 'Mens Haiti Striped Palm Zip Hoodie | Recycled Materials', 'price': '$98.00'}
Use wait_for_function on a specific signal, not a fixed sleep. A sleep breaks when the page takes longer to render. networkidle rarely fires on a real store full of analytics calls.
Read the price from the embedded JSON. The first $ in the visible text often belongs to a recommended item. On a page with many products, scope the extraction to the product's own JSON block.
The wait pattern checks for a $ price. Swap the character class to [€£¥] (or use a CSS selector) for non-USD sites.
Plain Playwright renders correctly, but on sites running DataDome or HUMAN (formerly PerimeterX), a default headless browser is flagged quickly. For those targets, the browser tier has its own anti-detect tools:
- Undetected CDP drivers such as nodriver and zendriver run Chromium over the DevTools protocol without the usual webdriver flags.
- Stealth-mode frameworks like SeleniumBase offer a UC mode and a CDP mode built to reduce detection.
- Stealth browser engines include Camoufox, a Firefox build focused on a less detectable fingerprint.
- Adaptive libraries like Scrapling wrap stealth fetchers behind a higher-level scraping API.
Outcomes shift per target as anti-bot defenses update, so live-test before standardizing on any one. Pair your choice with clean residential IPs because none of these tools fixes the IP-reputation layer. Use one only after the cheaper methods have failed.
Cookie walls, age gates, and country redirects
Real ecommerce pages often intercept the first load. Many EU and UK retailers show a cookie consent modal that blocks the DOM until accepted. Most alcohol, vape, and tobacco stores show an age gate that hides the product behind a click-to-confirm. Many retailers redirect a non-domestic IP to a localized subdomain like uk.brand.com, which can break URL-based scraping if the script does not follow redirects.
Cookie and age gates use the same handling pattern. Right after page.goto, the script tries each known button label and clicks the first one found. Use page.get_by_role so the lookup is language-aware and resilient to layout changes.
# After page.goto, try a short click pass for common consent/age buttons.
# Extend per locale for non-English markets (e.g. "Tout accepter", "Alle akzeptieren").
CONSENT_LABELS = ["Accept all", "Accept cookies", "I agree", "I am over 18", "Enter site"]
def dismiss_intercept(page, timeout=2000):
"""Click a known consent or age-gate button if one is present."""
for label in CONSENT_LABELS:
try:
page.get_by_role("button", name=label, exact=False).click(timeout=timeout)
return label
except Exception:
continue
return None
Country redirects need a different fix. One option is to set the Accept-Language header to a regional value and route through a proxy in that country. The other is to let the redirect happen and store the final URL so later runs can target the regional site directly.
How to scrape ecommerce category pages and pagination safely
Category and collection pages let you cover the catalog. They list every product in a section. Walk them with a web crawler to discover the full catalog before scraping detail. Without limits, the crawl never ends or collects the same product twice.
When available, [store]/sitemap.xml lists every product URL directly and bypasses category crawling.
Set a maximum number of pages per run, and dedupe by a stable ID as you go. With those 2 rules, the crawl finishes and you can re-run it.
URL patterns and page offsets
3 common pagination patterns are a page number, a limit and offset pair, or a cursor token. The product feed paginates with page and limit (up to 250 per page). Store the page number with each row so you can resume and trace where a record came from:
import time
import httpx
STORE = "https://www.tentree.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
PAGE_SIZE = 250 # Shopify allows up to 250 products per page
MAX_PAGES = 2 # stop rule: never walk the whole catalog by accident
def crawl():
seen_ids = set()
rows = []
page = 1
while page <= MAX_PAGES:
resp = httpx.get(
f"{STORE}/products.json",
params={"limit": PAGE_SIZE, "page": page},
headers=HEADERS,
timeout=30,
follow_redirects=True,
)
resp.raise_for_status()
products = resp.json()["products"]
if not products:
break # empty page means the catalog ended
for p in products:
if p["id"] in seen_ids:
continue # dedupe by stable product id
seen_ids.add(p["id"])
rows.append(
{
"product_id": p["id"],
"title": p["title"],
"from_price": p["variants"][0]["price"],
"in_stock": any(v["available"] for v in p["variants"]),
"page": page,
}
)
page += 1
time.sleep(1) # be polite between pages
return rows
if __name__ == "__main__":
rows = crawl()
print(f"collected {len(rows)} unique products")
for r in rows[:3]:
print(r)
Crawling 2 pages of 250 yields the first 500 products of the catalog:
collected 500 unique products
{'product_id': 8360519663802, 'title': 'InMotion Apex Hat', 'from_price': '45.00', 'in_stock': True, 'page': 1}
{'product_id': 8360518516922, 'title': 'Haiti Striped Palm Zip Hoodie', 'from_price': '98.00', 'in_stock': True, 'page': 1}
{'product_id': 8360518353082, 'title': 'Haiti Palm Logo Relaxed Hoodie', 'from_price': '88.00', 'in_stock': True, 'page': 1}
Deduping by product ID
The crawler dedupes by the numeric product ID, which is stable even when a title or handle changes.
To dedupe across runs too, persist the seen IDs between crawls in a file or table. A re-run then skips products it already has, which cuts both load and duplicate rows.
How to detect and avoid anti-bot blocks in ecommerce web scraping
Marketplaces like Amazon and Walmart run their own anti-bot stacks. Many other large retailers use services such as Cloudflare, DataDome, Akamai, and HUMAN (formerly PerimeterX). All of these check your connection before serving a page.
The common signs of a block are an HTTP 403 or 429, a CAPTCHA page, an empty HTML shell, or a redirect to a challenge page. Some blocks return a 200 status with no usable content.
Plain Python HTTP libraries get challenged by Cloudflare and similar services because their TLS and HTTP/2 handshakes look different from a real browser's. The check happens at the packet layer before any HTTP headers are read, which is why header tweaks or IP rotation alone do not defeat TLS fingerprinting. At low request rates against undefended targets, they still work. At scale or against defended targets, they fail. Steady pacing avoids blocks, while bursts trigger them.
Detect a block before you trust the response
Soft blocks do not show in the status code. A challenge or CAPTCHA page returns 200 and looks valid until you try to parse it. If you miss it, you collect empty or invalid data silently.
Reverb returned HTTP 200 with a Cloudflare challenge in the body. The status code says success; the content says verify.
Classify the response by its content. Confirm the expected field is present.
import httpx
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
BLOCK_MARKERS = (
"just a moment",
"verify you are human",
"captcha",
"access denied",
"enable javascript and cookies",
)
def looks_blocked(status_code, text, expected_marker=None):
"""Return True if a response is a block, even when the status is 200.
Status codes lie: a challenge page often returns 200. So check the
body, and confirm the expected content is present.
"""
if status_code in (403, 429) or status_code >= 500:
return True
lowered = text.lower()
if any(marker in lowered for marker in BLOCK_MARKERS):
return True # soft block: a challenge page served with status 200
if expected_marker and expected_marker.lower() not in lowered:
return True # 200, but the expected field is missing
return False
def probe(url, expected_marker=None):
try:
resp = httpx.get(url, headers=HEADERS, timeout=20, follow_redirects=True)
except httpx.RequestError as exc:
return f"network error: {exc!r}"
blocked = looks_blocked(resp.status_code, resp.text, expected_marker)
return f"status {resp.status_code}, blocked={blocked}"
if __name__ == "__main__":
# An open store returns product JSON; a defended one returns a block.
print("tentree (open feed): ", probe(
"https://www.tentree.com/products.json?limit=1", expected_marker="product"
))
print("gymshark (defended): ", probe("https://www.gymshark.com/products.json"))
Against a real open feed and a real defended store, the probe distinguishes a usable response from a block:
tentree (open feed): status 200, blocked=False
gymshark (defended): status 403, blocked=True
Beyond success and block, outcomes include moved pages, removed pages, server errors, and layout changes. Each one needs a different fix. The options are retry, escalate, or stop. Many failures are malformed or moved URLs, so confirm the URL is valid before you escalate to costlier residential proxies.
Call looks_blocked at fetch time, after a 200 but before parsing, so a blocked response never gets to your dataset. Treat a True result as a block, not as a transient error. Change the session or IP before retrying, or escalate.
This check is separate from the data-quality checks. It catches a response that looks valid but is a block.
Match a real browser's TLS fingerprint
For a TLS-fingerprint challenge, use a client that presents a real browser's fingerprint. curl_cffi is a requests-style library that impersonates a Chrome TLS and HTTP/2 handshake, so the connection presents like a browser rather than default Python. Install it with pip install curl_cffi==0.15.0.
from curl_cffi import requests
resp = requests.get(
"https://www.tentree.com/products.json?limit=1",
impersonate="chrome",
timeout=25,
)
print("status:", resp.status_code)
print("products:", len(resp.json()["products"]))
The call returns JSON, the same as a plain client on an open store.
status: 200
products: 1
The difference appears in a fingerprint check. Ask any TLS-echo service what fingerprint each client sends. The example below uses tls.peet.ws, but any JA4-aware endpoint works. The plain client and the impersonating client present different JA4 fingerprints.
import httpx
from curl_cffi import requests as cffi
URL = "https://tls.peet.ws/api/all" # echoes back the TLS fingerprint it sees
plain = httpx.get(URL, timeout=25).json()["tls"]["ja4"]
chrome = cffi.get(URL, impersonate="chrome", timeout=25).json()["tls"]["ja4"]
print("plain client JA4:", plain)
print("curl_cffi JA4: ", chrome)
The 2 fingerprints differ in a way an anti-bot system can read.
plain client JA4: t13d1712h1_ab0a1bf427ad_8e6e362c5eac
curl_cffi JA4: t13d1516h2_8daaf6152771_d8a2da3f94cd
The h1 versus h2 flag exposes the difference. The plain client uses HTTP/1.1, while the impersonating client uses HTTP/2 like Chrome. The exact JA4 strings shift as TLS libraries update.
A matching fingerprint can help pass TLS checks, but it is not enough by itself. Modern anti-bot systems score the whole request together. The score combines IP reputation, the TCP/IP stack, TLS, header order, and browser signals. If those layers disagree, such as a Chrome TLS fingerprint coming from a flagged datacenter IP, the score still fails.
A heavily-defended store returns 403 to both clients from a datacenter IP. The impersonating client also needs clean residential IPs, because IP reputation is the layer a fingerprint alone cannot fix.
Pace requests and back off on errors
A steady, slightly random delay looks more human than a fixed one and lowers the chance of getting rate-limited. Use a base delay plus jitter, and back off when errors appear. This fetcher retries only the failures that backoff fixes, the 429 and 5xx codes, waiting longer each time. The 403 is excluded because backoff does not fix it.
import os
import random
import time
import httpx
from dotenv import load_dotenv
load_dotenv() # read PROXY_* from a .env file
RETRYABLE = {429, 500, 502, 503, 504} # backoff fixes these; a 403 it won't
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def build_proxy():
"""Read rotating residential proxy creds from the environment."""
host = os.environ.get("PROXY_HOST")
if not host:
return None # run direct when no proxy is configured
user = os.environ["PROXY_USER"]
pwd = os.environ["PROXY_PASS"]
return f"http://{user}:{pwd}@{host}"
def fetch(url, max_retries=4):
proxy = build_proxy()
client = httpx.Client(
headers=HEADERS, proxy=proxy, timeout=25, follow_redirects=True
)
try:
for attempt in range(max_retries):
try:
resp = client.get(url)
except httpx.RequestError as exc:
wait = 2**attempt + random.uniform(0, 1)
print(f"network error ({exc!r}); retry in {wait:.1f}s")
time.sleep(wait)
continue
if resp.status_code in RETRYABLE:
wait = 2**attempt + random.uniform(0, 1) # backoff + jitter
print(f"got {resp.status_code}; backing off {wait:.1f}s")
time.sleep(wait)
continue
resp.raise_for_status()
# Apply the soft-block check from the detect-a-block section above
if looks_blocked(resp.status_code, resp.text):
raise RuntimeError(f"blocked at {url}; change IP/session")
return resp
raise RuntimeError(f"gave up on {url} after {max_retries} tries")
finally:
client.close()
if __name__ == "__main__":
resp = fetch("https://www.tentree.com/products.json?limit=1")
print("status:", resp.status_code)
print("products:", len(resp.json()["products"]))
A clean run returns quickly:
status: 200
products: 1
The exponential backoff (2 ** attempt) plus a random fraction spreads retries out instead of overwhelming a struggling server. Pick a base delay you can sustain for the whole crawl rather than the fastest one that works for the first page. When the response includes a Retry-After header, use that value instead of your own backoff.
A 403 is not fixed by waiting. The helper excludes it from the retryable set. When one happens, change the IP or session before retrying, or you'll collect another 403. The client uses httpx, but a Chrome User-Agent over Python's own TLS is the mismatch the previous section warned about. On a site that fingerprints TLS, swap httpx for a curl_cffi session so the User-Agent and the handshake agree.
Send a realistic browser identity and reuse the session
A realistic request sends a real browser User-Agent and an Accept-Language header, and reuses connections and HTTP cookies across requests. The httpx.Client above sets the headers once and reuses them on every call, along with any cookies the site sets. Keep User-Agent rotation minimal. A client that suddenly changes its browser identity triggers anti-bot flags, while a steady one passes through.
For higher volume, many categories, or geo-specific pricing, the IP itself becomes the limit. Proxies handle this case.
How to choose proxies for ecommerce web scraping
Proxies matter when you scrape at volume, cover many categories at once, or need prices as a shopper in a specific country sees them. A single IP making thousands of requests is the easiest pattern for a site to flag, and a price page often shows different prices by region. Rotating the IP spreads requests across many IPs, and routing through one of the available proxy locations returns that country's pricing.
Live Proxies currently offers rotating residential and rotating mobile IPs with sticky sessions for longer runs. Residential and mobile IPs come from real home and carrier connections, which tend to have higher acceptance rates than datacenter ranges. A rotating plan of N IPs returns N unique IPs at any moment, and natural peer rotation typically expands that to more unique IPs across a 30-day window. Read the trade-offs between residential, datacenter, and mobile proxies before you choose.
Datacenter proxies are cheaper and fine for undefended targets like a public Shopify feed. Marketplaces and CDN-defended sites usually flag them, which is when you need residential or mobile.
2 features matter for ecommerce crawls specifically.
- Sticky sessions aim to retain the same IP for the configured period, subject to the underlying peer remaining online. The actual duration may be shorter than the maximum supported time. That stability keeps a multi-step flow such as variant selection and a delivery estimate on the same IP. Without a session ID, requests rotate through the pool, which suits high-volume discovery.
- Private IP allocation reserves IPs for one customer per target. The same IPs may serve another customer scraping unrelated targets, but never the same one, which reduces the overlap that gets shared pools flagged.
Before you choose a provider, check how the residential IPs are sourced. The cheapest pools are sometimes built from malware-infected consumer devices. This is both an ethics problem and a reliability problem, since a compromised device can drop mid-request. Prefer a provider that says where its IPs come from. Live Proxies sources its residential IPs from a network of partnered peers who connect to it openly.
Static residential is a third option between rotating residential and a fully rotating pool. Live Proxies sources these by filtering home IPs that stay stable on their ISPs, so the IP usually holds for around 30 days.
The IPs are still real residential connections. Most "static residential" products on the market are static ISP or datacenter IPs, which are fast but more detectable. That fits flows that need both consistency and a home-IP acceptance profile. Examples are a stored shopping session, a logged-in pricing view, or a multi-step checkout estimate.
IP reputation may still be affected by prior activity or a target site's own detection, and performance depends on IP quality and request behavior. Proxies reduce block risk but do not remove it. Large pools cost more, so add them when scale or geo pricing requires it. You can check pricing on the Live Proxies pricing page and verify an IP with the proxy tester.
The fetch function already loads proxy credentials from a .env file. The exact format depends on plan type. A B2C plan uses an IP-per-line format with a session ID for sticky sessions, where dropping the session digit switches to rotation. A B2B plan uses one gateway with separate rotating and sticky endpoints, both available on the dashboard.
# B2C sticky (60-min session)
PROXY_HOST=203.0.113.36:7383
PROXY_USER=LV71125532-mDmfksl3onyoy-1
PROXY_PASS=bW2VN4Zc5YSyK5nF82tK
# B2C rotating (drop the -1 session digit)
PROXY_USER=LV71125532-mDmfksl3onyoy
# B2B (gateway routes to rotating or sticky based on dashboard config)
PROXY_HOST=b2b.liveproxies.io:7383
PROXY_USER=username-access_code-sid # add -sid for sticky, omit for rotating
PROXY_PASS=password
For more on avoiding blocks at scale, see how Live Proxies help prevent IP bans in large-scale web scraping.
How to scrape a defended ecommerce store end-to-end
Amazon or Walmart coverage needs a working pipeline, not just the tool list above. Install Scrapling and its stealth dependencies first.
pip install "scrapling[fetchers]==0.4.9"
patchright install chromium
python3 -m camoufox fetch
Scrapling wraps Patchright, a stealth Playwright fork, with Camoufox's fingerprint-randomized Firefox build. A single fetch call hides webdriver flags and randomizes canvas and WebGL fingerprints. It also presents a real browser TLS handshake.
Bypass Cloudflare on a defended store
Plain httpx returns 403 from Gymshark's /products.json, as the anti-bot section showed. The same site returns 200 to Scrapling's StealthyFetcher, from the same IP.
from scrapling import StealthyFetcher
def scrape_gymshark_home():
page = StealthyFetcher.fetch(
"https://www.gymshark.com", headless=True, wait=4000
)
return {
"status": page.status,
"html_bytes": len(page.html_content),
"title": page.css("title::text").get("").strip(),
"has_product_markup": "/products/" in page.html_content,
}
if __name__ == "__main__":
import json
print(json.dumps(scrape_gymshark_home(), indent=2))
The same IP that got 403 from httpx gets a full page through the stealth browser.
{ "status": 200,
"html_bytes": 1430243,
"title": "Gymshark Official Store - Gym Clothes & Workout Clothes",
"has_product_markup": true
}
Extract a real product from Walmart
Walmart embeds its product data in a _ _ NEXT_DATA _ _ script tag, so the work after the stealth fetch is one regex and one JSON walk. Walmart is moving to Next.js App Router across the site, which puts the data in different containers. Wrap the deep dict path in try/except so the parser fails gracefully when those container paths change.
import json
import re
from scrapling import StealthyFetcher
URL = (
"https://www.walmart.com/ip/"
"Klondike-Original-Crunchy-Ice-Cream-Bars-Kosher-Certified-6-Count/10801678"
)
def scrape_walmart(url):
page = StealthyFetcher.fetch(url, headless=True, wait=4000)
if page.status != 200:
return {"status": page.status, "blocked": True}
match = re.search(
r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', page.html_content
)
if not match:
return {"status": 200, "blocked": False, "data": None}
data = json.loads(match.group(1))
product = data["props"]["pageProps"]["initialData"]["data"]["product"]
price_info = product.get("priceInfo", {}).get("currentPrice", {}) or {}
return {
"status": 200,
"blocked": False,
"title": product.get("name"),
"price": price_info.get("price"),
"currency": price_info.get("currencyUnit"),
"availability": product.get("availabilityStatus"),
"rating": product.get("averageRating"),
"review_count": product.get("numberOfReviews"),
}
if __name__ == "__main__":
print(json.dumps(scrape_walmart(URL), indent=2))
A real Walmart product page, scraped end-to-end from a datacenter IP:
{ "status": 200,
"blocked": false,
"title": "Klondike Original Ice Cream Bars Kosher Certified Ice Cream Simple Dessert Made With No Artificial Growth Hormones 4.5 fl oz, 6 Count",
"price": 4.44,
"currency": "USD",
"availability": "IN_STOCK",
"rating": 4.6,
"review_count": 6357
}
Coverage and limits
The same approach gets a 200 from Amazon product pages. Selectors like #productTitle and #acrPopover return real product data. Verify against the live page.
Amazon changes its selectors every few years. Amazon also swaps between several price containers depending on the product type, the buyer's session, and whether a variant requires selection. Plan for the price field to need product-specific handling, even on a successful fetch.
A stealth-browser fetch is only one layer. 3 things still affect whether this holds in production.
- Watch IP reputation. Residential proxies through StealthyFetcher's proxy= parameter make the same code work for longer at scale.
- Watch the request rate. The 403-vs-429 distinction from the pacing section still applies. The browser fetches one page at a time, so handle concurrency above it.
- Watch for drift. Selectors and _ _ NEXT_DATA _ _ paths change as the marketplaces update their frontends. The block-detection function from earlier catches the failure mode, but the parser then needs an update.
Many marketplaces run a separate mobile flow with lighter anti-bot. Mobile API endpoints (found by inspecting mobile app traffic), AMP pages, and the desktop URL with a real mobile User-Agent often clear the same product with less stealth than a full desktop fetch. Most retailers have retired the old m.* subdomains in favor of responsive design, so check the current URL pattern for each target.
The product fields are typically identical between mobile and desktop. Only the URL, the User-Agent, and sometimes the rendering path differ. When the desktop fetch is failing or expensive, try the mobile path with a real mobile User-Agent before adding more stealth-browser cost.
The stealth browser solves the access problem but not the maintenance problem.
How to clean and validate ecommerce web scraping data
Scraped ecommerce data is messy by nature. Prices disappear when a layout shifts, variants are confused with their parent product, and the same item appears twice from 2 URLs. Validate before you trust any of it.
Run a short data verification pass after every crawl and review the counts. A spike in missing prices usually means a selector or an API field changed. It does not mean the store removed its prices.
Basic validation checks
Check the high-signal failures. The common ones are missing prices, duplicate IDs, zero or negative prices, and out-of-stock counts. Run them over a dataframe against the real catalog.
import httpx
import pandas as pd
STORE = "https://www.tentree.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
def fetch_rows(limit=250):
resp = httpx.get(
f"{STORE}/products.json",
params={"limit": limit},
headers=HEADERS,
timeout=30,
follow_redirects=True,
)
resp.raise_for_status()
rows = []
for p in resp.json()["products"]:
variants = p["variants"]
rows.append(
{
"product_id": p["id"],
"price_cents": round(float(variants[0]["price"]) * 100),
"in_stock": any(v["available"] for v in variants),
}
)
return rows
def validate(df):
return {
"rows": len(df),
"missing_price": int(df["price_cents"].isna().sum()),
"duplicate_ids": int(df["product_id"].duplicated().sum()),
"zero_or_negative_price": int((df["price_cents"].fillna(1) <= 0).sum()),
"out_of_stock": int((~df["in_stock"]).sum()),
}
if __name__ == "__main__":
df = pd.DataFrame(fetch_rows())
for check, count in validate(df).items():
print(f"{check}: {count}")
Running against the live catalog, it flags one zero-price item and 9 products out of stock.
rows: 250
missing_price: 0
duplicate_ids: 0
zero_or_negative_price: 1
out_of_stock: 9
A flag is not always a bug. That zero-price item could be a gift card, which is valid data. Treat the counts as rows to investigate before deleting anything. After the automated checks, spot-check a handful of rows manually against the live page. Numbers can pass every rule and still be wrong if a parser grabbed the wrong field.
Compare today's validation counts to yesterday's. A 50%+ jump in missing_price or out_of_stock usually means the parser broke, even when individual rows look OK.
URL and text normalization
Normalize URLs and text. Trim whitespace, lowercase the domain, and strip tracking parameters (utm*, ref, gclid, fbclid_). Keep both the raw and cleaned versions of each field. The raw copy is your fallback when a cleaning rule removes too much.
Consistent URLs feed the dedupe step directly. 2 links that differ only by a tracking parameter should collapse to one canonical URL, or your unique-product count drifts upward over time. For more on clean collection, see how to improve data collection.
How to store ecommerce scraped data for tracking over time
Tracking captures what a one-time export misses. Price moves, stock swings, and catalog changes only show up across snapshots. Many dated snapshots become a dataset.
Store a snapshot per crawl with a crawl date and a stable ID on every row. CSV is fine for a small, short project. Use a database once you track many products over many days.
Snapshot strategy
Save a dated snapshot on each run rather than overwriting the last one. A folder pattern like data/2026-05-29/products.csv keeps history readable and lets you drop a bad crawl without touching the rest. Daily snapshots suit fast-moving prices. Weekly snapshots suit slow catalogs.
The crawl date makes a time series possible. With the same product ID across dated snapshots, you can line up any product's price on any 2 days and compute the change.
Database basics
Move to a database when CSV files get too large to handle or you need fast lookups. SQLite needs no server and fits a single-machine project. PostgreSQL fits when several jobs or people read and write at once. Use a primary key on product ID, crawl date, currency, and region. Index the columns you filter on.
import sqlite3
from datetime import date
import httpx
STORE = "https://www.tentree.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
DB = "catalog.db"
SCHEMA = """
CREATE TABLE IF NOT EXISTS price_snapshot (
product_id INTEGER NOT NULL,
crawl_date TEXT NOT NULL,
currency TEXT NOT NULL,
region TEXT NOT NULL,
price_cents INTEGER,
in_stock INTEGER,
PRIMARY KEY (product_id, crawl_date, currency, region)
);
CREATE INDEX IF NOT EXISTS idx_product ON price_snapshot (product_id);
"""
def fetch_rows(limit=20, currency="USD", region="us"):
resp = httpx.get(
f"{STORE}/products.json",
params={"limit": limit},
headers=HEADERS,
timeout=30,
follow_redirects=True,
)
resp.raise_for_status()
for p in resp.json()["products"]:
variants = p["variants"]
yield {
"product_id": p["id"],
"currency": currency,
"region": region,
"price_cents": round(float(variants[0]["price"]) * 100),
"in_stock": int(any(v["available"] for v in variants)),
}
def save_snapshot(rows, crawl_date):
conn = sqlite3.connect(DB)
conn.executescript(SCHEMA)
conn.executemany(
"INSERT OR REPLACE INTO price_snapshot "
"(product_id, crawl_date, currency, region, price_cents, in_stock) "
"VALUES (?, ?, ?, ?, ?, ?)",
[(r["product_id"], crawl_date, r["currency"], r["region"],
r["price_cents"], r["in_stock"]) for r in rows],
)
conn.commit()
conn.close()
if __name__ == "__main__":
today = date.today().isoformat()
save_snapshot(fetch_rows(), today)
conn = sqlite3.connect(DB)
count = conn.execute(
"SELECT COUNT(*) FROM price_snapshot WHERE crawl_date = ?", (today,)
).fetchone()[0]
print(f"rows stored for {today}: {count}")
conn.close()
The composite primary key means re-running the same day updates the row instead of creating a duplicate.
rows stored for 2026-06-03: 20
The INSERT OR REPLACE keeps one row per product per day. If you re-run a crawl after a failure, you overwrite the partial data instead of doubling it.
As the catalog ages, drop daily snapshots after a fixed window (for example, 90 days) and keep only weekly or monthly aggregates.
For multi-market crawls, the schema needs more than a product ID and date. The same product can have different prices in different countries or currencies, so currency and region are part of the primary key. Without them, a $100 hoodie in the US would overwrite the £80 hoodie in the UK.
How to analyze ecommerce scraping results
Once you store a dated snapshot per crawl, a few analyses pay for the whole pipeline. These are price changes over time, stock trends, new and removed products, and shifts in review sentiment. Each one is a query over those snapshots.
Start with the 2 that drive most decisions. These are price monitoring and catalog change tracking. Both need only a product ID, a price, and a date, which you already capture.
Price monitoring
Compare each product's price between 2 snapshots and flag the ones that moved past a threshold. Build the logic against in-memory data first.
def find_price_changes(baseline, current, threshold=0.10):
"""Flag products whose price moved more than the threshold fraction."""
changes = []
for product_id, new_price in current.items():
old_price = baseline.get(product_id)
if not old_price or not new_price:
continue
delta = (new_price - old_price) / old_price
if abs(delta) >= threshold:
changes.append(
{
"product_id": product_id,
"old": old_price,
"new": new_price,
"change_pct": round(delta * 100, 1),
}
)
return changes
if __name__ == "__main__":
baseline = {1: 999, 2: 1999, 3: 549, 4: 1299, 5: 899}
current = {1: 894, 2: 1635, 3: 495, 4: 1141, 5: 796}
for change in find_price_changes(baseline, current):
print(change)
Only the products past the 10 percent threshold show up (product 3, marked down 9.84 percent, stays out).
{'product_id': 1, 'old': 999, 'new': 894, 'change_pct': -10.5}
{'product_id': 2, 'old': 1999, 'new': 1635, 'change_pct': -18.2}
{'product_id': 4, 'old': 1299, 'new': 1141, 'change_pct': -12.2}
{'product_id': 5, 'old': 899, 'new': 796, 'change_pct': -11.5}
In production, the 2 price maps come from the SQLite database the snapshot step wrote. Read yesterday's and today's snapshots, join them on the composite key, and report moves past the threshold.
import sqlite3
from datetime import date, timedelta
DB = "catalog.db"
THRESHOLD = 0.10 # flag changes of >=10%
DIFF_QUERY = """
SELECT
today.product_id,
yesterday.price_cents AS old_cents,
today.price_cents AS new_cents
FROM price_snapshot AS today
JOIN price_snapshot AS yesterday
ON yesterday.product_id = today.product_id
AND yesterday.currency = today.currency
AND yesterday.region = today.region
WHERE today.crawl_date = :today
AND yesterday.crawl_date = :yesterday
AND yesterday.price_cents > 0
AND today.price_cents > 0
"""
def find_changes(db_path=DB, today=None, yesterday=None, threshold=THRESHOLD):
today = today or date.today().isoformat()
yesterday = yesterday or (date.today() - timedelta(days=1)).isoformat()
conn = sqlite3.connect(db_path)
rows = conn.execute(DIFF_QUERY, {"today": today, "yesterday": yesterday}).fetchall()
conn.close()
changes = []
for pid, old, new in rows:
delta = (new - old) / old
if abs(delta) >= threshold:
changes.append(
{
"product_id": pid,
"old_cents": old,
"new_cents": new,
"change_pct": round(delta * 100, 1),
}
)
return sorted(changes, key=lambda r: abs(r["change_pct"]), reverse=True)
if __name__ == "__main__":
moves = find_changes()
print(f"price moves >= {THRESHOLD * 100:.0f}%: {len(moves)}")
for m in moves:
direction = "down" if m["change_pct"] < 0 else "up"
print(f" {m['product_id']}: {m['old_cents']:>5} -> {m['new_cents']:>5} " f"({m['change_pct']:+.1f}% {direction})")
After 2 daily crawls, it reports real moves on real product IDs, sorted by magnitude.
price moves >= 10%: 3
8360516878522: 6075 -> 4500 (-25.9% down)
8360519663802: 3825 -> 4500 (+17.6% up)
8360518516922: 11760 -> 9800 (-16.7% down)
The same SQL extends to weekly or monthly comparisons by swapping the 2 date parameters. Feed the list into an alert (email, a chat message, a dashboard row) so a price drop gets to you the day it happens.
Catalog change tracking
Detect new and removed products by comparing the set of product IDs between snapshots. An ID present today but not yesterday is new. An ID present yesterday but missing today is removed or sold out. The same price_snapshot table holds the IDs, so the diff is 1 SELECT plus 2 Python set operations.
import sqlite3
from datetime import date, timedelta
DB = "catalog.db"
def ids_on(crawl_date, db_path=DB):
conn = sqlite3.connect(db_path)
rows = conn.execute(
"SELECT product_id FROM price_snapshot WHERE crawl_date = ?",
(crawl_date,),
).fetchall()
conn.close()
return {pid for (pid,) in rows}
def catalog_diff(today=None, yesterday=None):
"""Set-diff product IDs between 2 crawl dates."""
today = today or date.today().isoformat()
yesterday = yesterday or (date.today() - timedelta(days=1)).isoformat()
today_ids, yesterday_ids = ids_on(today), ids_on(yesterday)
return {"added": today_ids - yesterday_ids, "removed": yesterday_ids - today_ids}
if __name__ == "__main__":
diff = catalog_diff()
print(f"added: {len(diff['added'])}")
print(f"removed: {len(diff['removed'])}")
A typical day in a tracked catalog shows a small number of changes.
added: 2
removed: 1
For a richer view, store a first_seen and last_seen date per product. Those 2 dates also surface category reshuffles and slow-moving stock. When last_seen keeps advancing but the category changes, that signals a merchandising shift.
Stock trends extend the same SQL pattern with in_stock instead of price_cents. Review sentiment runs comments through a classifier (an LLM or a hosted sentiment API) and tracks the average per product over time.
How to scale web scraping ecommerce websites in 2026
Scaling means turning one reliable script into a pipeline that runs unattended. The pieces are scheduling, retries, monitoring, and cost control. Add them gradually, and keep the pipeline simple enough to debug.
Aim for a crawl that finishes, tells you when it didn't, and costs what you expect. Going faster than that adds blocks, not throughput.
Scheduling and monitoring
Run the crawl on a schedule with cron or a task scheduler, and log every run's success and failure counts. Write failed URLs to a list so a later pass can retry only those, instead of re-crawling everything. Alert when the success rate drops below a threshold you set, because a silent failure costs more. A cron entry runs a daily crawl and appends output to a log:
# Run the crawler every day at 03:00 and keep a log
0 3 * * * cd /srv/scraper && .venv/bin/python crawl.py >> logs/crawl.log 2>&1
The redirect captures both normal output and errors, so the log shows what happened during unattended runs. Rotate the log file so it does not grow without limit.
Concurrency with care
More parallel requests mean more blocks, so raise concurrency slowly. Start with a low number of workers. Watch the error rate, and increase only while it stays flat. A crawl at 5 steady workers all night is more reliable than one at 50 that gets blocked in the first hour.
A long catalog crawl is too slow to run one page at a time, but unbounded concurrency gets you blocked. An asyncio.Semaphore is the cap.
import asyncio
import httpx
STORE = "https://www.tentree.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; price-research/1.0)"}
PAGE_SIZE = 250
MAX_PAGES = 4
CONCURRENCY = 3 # cap: never more than 3 requests in flight at once
async def fetch_page(client, sem, page):
async with sem: # the semaphore bounds how many run at once
resp = await client.get(
f"{STORE}/products.json",
params={"limit": PAGE_SIZE, "page": page},
)
resp.raise_for_status()
products = resp.json()["products"]
print(f"page {page}: {len(products)} products")
return products
async def crawl():
sem = asyncio.Semaphore(CONCURRENCY)
async with httpx.AsyncClient(
headers=HEADERS, timeout=30, follow_redirects=True
) as client:
# return_exceptions keeps one failed page from sinking the whole batch.
results = await asyncio.gather(
*(fetch_page(client, sem, p) for p in range(1, MAX_PAGES + 1)),
return_exceptions=True,
)
pages = [r for r in results if isinstance(r, list)]
failed = len(results) - len(pages)
if failed:
print(f"{failed} page(s) failed; keeping the rest")
seen = {p["id"] for page in pages for p in page}
return len(seen)
if __name__ == "__main__":
total = asyncio.run(crawl())
print(f"collected {total} unique products across {MAX_PAGES} pages")
The print order varies between runs because the pages finish whenever they finish.
page 1: 250 products
page 2: 250 products
page 3: 250 products
page 4: 250 products
collected 1000 unique products across 4 pages
The return_exceptions=True matters at scale. Without it, one page hitting a 429 throws away every other page in the batch. Raise CONCURRENCY slowly and watch the error rate.
When one machine is not enough, distribute work across workers with a shared queue (Redis or similar) that dedupes by stable ID across them.
Prefer API calls over headless browsers when you scale. An API request is cheap and fast, while a browser is slow and heavy. Reserve browsers for the pages that need rendering, and let plain HTTP handle the rest. For language-specific patterns, Python web scraping covers structuring a larger crawler.
Small-batch debugging
When a scraper breaks, the instinct is to switch the proxy and re-launch the whole crawl. That wastes a run and hides the cause. Run a small test batch first. Confirm it works from the logs, then scale back up.
Diagnose the failure before you change anything. A proxy block, a navigation change, and a product-page parse error each need a different fix. Watch the scraper across several runs before you trust it, since a layout or anti-bot change can break it days later.
Regression tests for scrapers
Scrapers fail silently when a site updates its markup. Regression tests catch the failures before they get to production.
Save one real HTTP response per target site to a fixtures folder, mock the network with respx (the httpx-native mock library), and assert that the parser returns the expected canonical record. Install with pip install respx==0.21.1.
import json
from pathlib import Path
import httpx
import respx
# Real Shopify response captured once and stored in tests/fixtures/.
# Refresh on the same 90-day cadence as the pinned versions.
FIXTURE = Path("tests/fixtures/tentree_products.json")
EXPECTED = {
"product_id": 8360519663802,
"title": "InMotion Apex Hat",
"from_price": "45.00",
"in_stock": True,
}
def parse_product(payload):
"""Extract one canonical product record from the products.json payload."""
product = payload["products"][0]
return {
"product_id": product["id"],
"title": product["title"],
"from_price": product["variants"][0]["price"],
"in_stock": any(v["available"] for v in product["variants"]),
}
@respx.mock
def test_parse_product_against_fixture():
fixture_payload = json.loads(FIXTURE.read_text())
respx.get("https://www.tentree.com/products.json").mock(
return_value=httpx.Response(200, json=fixture_payload)
)
resp = httpx.get("https://www.tentree.com/products.json")
assert parse_product(resp.json()) == EXPECTED
if __name__ == "__main__":
test_parse_product_against_fixture()
print("ok: parser matches the captured fixture")
Refresh manually on a schedule. Automatic refreshes silently break the test, because the parser then matches whatever the site returned today. The schedule keeps the fixture as ground truth between manual reviews.
Wire the test into CI. When a parser change reorders a field or drops one, the test fails before the change gets to production. When a scheduled refresh updates the expected record, the team reviews the new ground truth before merging it.
What ecommerce web scraping costs, and when to buy instead of build
The cost of ecommerce web scraping is rarely the proxy bill. Most of it is engineering time. That time goes into building scrapers, fixing selectors when pages change, handling blocks, and checking data quality. Infrastructure (proxies, compute, storage) is usually the smaller share, so judging a project on price per GB alone misleads you.
Build versus buy is not one decision for the whole project. Treat it per feed, like a portfolio, and score each feed on 2 axes.
Criticality measures what breaks if this feed stops for 24 hours. A pricing feed that drives decisions ranks high. A one-off research pull ranks low.
Maintenance effort measures how often the site changes or blocks you. A stable government page is low. A defended marketplace is high.
The 2 axes map to 4 decisions:
| Low maintenance | High maintenance | |
|---|---|---|
| High criticality | Build and maintain in-house | Buy a managed feed because the effort cost dominates |
| Low criticality | Build if convenient | Buy a no-code or hosted option because the time is not worth it |
On the diagonals, the choice is obvious. The off-diagonals are opportunity-cost calls. Weigh whether your team's time is worth more on the core product than on a scraper. Revisit the call over time, because a feed that is critical today may not be in a year.
Wrong data is a cost teams often miss. A blocked page is obvious. But a trap page (a honeypot) can return a 200 response with fake prices, such as everything priced at $9.99. If that feeds a pricing model, one bad decision can cost more than the entire scraping setup. Detection and validation also deserve budget.
To estimate your own total cost, add engineer time to infrastructure. Engineer time is hourly cost times maintenance hours per month. Infrastructure is proxies plus compute plus storage. Compare that total to a managed quote for the same coverage.
Watch the unexpected-cost trap. When a block forces a switch to residential proxies, the bill can jump several times overnight, so size that headroom in advance.
If you decide to buy, 3 categories cover most of the path. These are visual scrapers, hosted scraping APIs, and agentic browsers.
A 4th option for one-time data needs is buying a pre-scraped dataset from a data marketplace, which suits historical or research-grade questions over fresh-data tracking.
No-code visual scrapers
A visual scraper has 4 steps. You select elements on the page, set the pagination rule, run a test batch, and export a CSV. It is the fastest path to a one-off list with no code.
The main limit is fragility. Because these tools target elements by their on-page position or class, a layout change can break the recipe with no warning. They suit small, occasional pulls more than a daily pipeline you depend on. They are often the right call for low-criticality, high-maintenance feeds.
Hosted ecommerce scrapers and APIs
Hosted scrapers and scraping APIs are a form of managed web scraping. They handle rotation, retries, and parsing on their side, and usually return structured JSON. You send a URL or a product ID and receive clean fields, without running a browser yourself.
They save time when a site is heavily defended or when you'd rather not maintain infrastructure. The cost climbs with volume and can exceed what a custom script with proxies would cost at scale. AI web scraping and data collection is making these managed options more common.
Agentic browsers and computer-use APIs
A newer option uses an LLM to drive a real browser. The model reads the page through screenshots or DOM text, then decides what to click and what to type. It reads back the result for the next step. The tools here are Browser Use, Stagehand, Anthropic Computer use, and OpenAI Operator. Each wraps a Playwright or Chrome DevTools session with an LLM that takes instructions like "navigate to the product page and extract the price".
The cost shape is different from a normal scraper. At current pricing, a direct HTTP fetch costs about $0.0001 per page, a stealth browser costs about $0.001, and an agentic-browser call costs $0.01 to $0.10. The agentic call is 2 to 3 orders of magnitude more expensive, because every step is an LLM call and a typical product extraction takes 3 to 8 steps.
These tools fit 2 specific cases. One is sites where a stealth fetch fails and writing a per-site stealth pipeline is not worth the effort, such as a one-off price check on a regional retailer with strong anti-bot defenses. The other is low-criticality catalogs where the engineering cost of building a custom scraper exceeds the per-page LLM cost over the project's lifetime.
These tools do not fit high-criticality production pipelines. The per-page cost is too high at volume, and the LLM step adds unpredictable latency (5 to 30 seconds per page on current models). The failure mode is also hard to debug, because the agent's reasoning is not deterministic. A direct fetch or a stealth browser with per-site selectors remains the right choice when the feed is critical.
Conclusion
When ecommerce web scraping breaks in production, the site blocks the request before the parser runs. Access blocks are a common cause of scraping failures, but parsers can also break when selectors, page structures, or API schemas change. So most of the engineering work is about not getting blocked, and that determines the implementation order from cheapest tier to most expensive.
Three workloads sit outside this guide's scope. They are account-walled prices behind a login, CAPTCHA solving at scale, and the legal-review process for commercial collection. For one-time historical data, a pre-scraped dataset from a data marketplace is often cheaper than running any of these tiers.
Take the 10-minute script, point it at a store you care about, and run it tonight. After it runs on a schedule for a few nights without breaking, you have a working pipeline. When a single machine is not enough anymore for your request volume, the Live Proxies enterprise plan scales to higher throughput with dedicated IP allocation. The next time a scraper breaks, check the block before you blame the parser.
FAQs
How do I scrape JavaScript ecommerce sites?
First request the raw page and search it for the data. Many stores embed it or expose a JSON API you can call directly. Use a headless browser like Playwright only when the data is not in the HTML. Wait on a specific element rather than a fixed sleep, and set a timeout as a stop rule.
How do I scrape ecommerce reviews at scale?
Reviews usually load from a separate API the page calls, so find that request in the Network tab and page through it. Sample by newest or most helpful rather than collecting everything, and record the sort you used. Dedupe by review ID. Use a hash of text plus date when no ID exists.
How do I scrape prices without getting blocked?
Pace requests with a base delay plus jitter, reuse one client so cookies persist, and send realistic headers. Scraping fewer pages consistently is often safer than crawling fast. Add rotating proxies when volume or geo-specific pricing makes a single IP the limit.
How do I bypass Cloudflare for ecommerce scraping?
Match a real browser's TLS and HTTP/2 handshake with curl_cffi, or run a stealth-browser library like Scrapling that bundles Patchright and Camoufox. Pair either with residential IPs, because a Chrome fingerprint over a flagged datacenter IP still fails. Cloudflare also reads behavior, so pace requests.
How often should I re-scrape an ecommerce catalog?
Match frequency to volatility. Re-scrape fast-moving prices daily, and check stock more often for high-demand items that sell out. Reviews change slowly. Scraping them weekly or monthly is usually enough. Track how often each field changes, then tune the schedule to that rate.
Does this approach work on non-Shopify stores like WooCommerce or Magento?
Yes, the method is the same and only the endpoints differ. WooCommerce and Magento expose their own JSON routes, and most product pages still carry JSON-LD. Check the Network tab for the store's data call before you render, then reuse the same parsing and validation code once you point it at the right endpoint.
Should I use a retailer's official API instead of scraping?
Use the official API when one exists and returns the fields you need, because it is stable and sanctioned. Amazon, eBay, and Walmart run partner or affiliate APIs, but access is gated and the data is often narrower than the storefront. Scrape the public pages when there is no API, when approval is slow, or when you need fields the API omits, such as a competitor's live price.
