Selenium Web Scraping With Python: Full 2026 Guide

Your requests + BeautifulSoup scraper returns an empty page on a site that clearly has content in the browser. So you switch to Selenium. It works. Then the scraper starts getting CAPTCHAs and 429s after a few dozen requests. And when you run it in Docker, Chrome won't start at all. None of this is unusual because bad bots are over 33% of all internet traffic, and the defenses tuned to block them catch scrapers too. This guide covers the fixes for each issue with verified code you can run locally.

TL;DR

Selenium web scraping uses a real browser to extract data from websites that load content with JavaScript, meaning the data isn't in the page source, only in the rendered DOM.

• Setup is pip3 install selenium. Selenium 4.6+ downloads a matching ChromeDriver automatically (and Chrome itself if it's missing), so you don't need to manage driver binaries manually
• Use --headless=new for modern headless Chrome, and WebDriverWait instead of time.sleep() for DOM elements – waits handle timing; sleeps guess
• Handle pagination with EC.staleness_of and export scraped data to CSV or JSON
• Remove the automation signals that anti-bot systems detect, using Chrome flags and CDP (Chrome DevTools Protocol) script overrides
• Scale past IP blocks with rotating residential proxies so requests come from many home IPs instead of one that gets flagged

What is Selenium web scraping and when should you use it?

Selenium is an open-source browser automation framework that controls Chrome, Firefox, or Edge through Python code. Chrome is often the practical default because it is widely used and has strong Selenium support. The anti-detection library ecosystem, including tools like undetected-chromedriver and SeleniumBase UC Mode, is also strongly focused on Chrome. Unlike HTTP-only web scraping libraries, Selenium loads the full page in a real browser engine. The browser runs JavaScript and builds the full DOM, including content that loads after the initial page.

How to tell if you need Selenium: open your target page in Chrome, right-click, and select View Page Source.

Search for the data you want. If it's in the HTML, use requests + BeautifulSoup (faster). If you see empty <div> containers or JavaScript variables instead of actual data, the content is JS-rendered.

You'll want Selenium or another browser automation tool like Playwright.

Use Selenium to scrape data when any of these conditions apply:

• The page renders content with JavaScript (React, Vue, Angular, or vanilla AJAX calls).
• You need to interact with the page: clicking buttons, filling complex forms, scrolling to trigger lazy loads, or navigating multi-step login flows that are hard to replicate with HTTP libraries.
• The site checks browser-level fingerprints (TLS handshake, HTTP/2 ordering, JavaScript runtime) that HTTP libraries struggle to mimic. Small-scale HTTP scraping may still work on sites without strict bot protection, but Cloudflare-class protection tends to challenge it quickly.
• You need to capture data that loads behind user interactions like dropdowns, tabs, or "load more" buttons.

For static pages, requests + BeautifulSoup is faster. See the broader Python web scraping guide for when each approach fits. Many scraping projects combine both: Selenium loads the page in a real browser, BeautifulSoup parses the rendered HTML. The combined pattern is short:

from bs4 import BeautifulSoup

driver.get(url)
soup = BeautifulSoup(driver.page_source, "html.parser")
titles = [a.get("title") for a in soup.select("article.product_pod h3 a")]

Selenium renders and waits; BeautifulSoup handles parsing with a faster select() than Selenium's find_elements and a richer tree-traversal API.

Quick decision guide

Different scraping problems need different tools. Use this as a quick check before committing to Selenium:

How to scrape data using Selenium in Python in 10 minutes

You need Python 3.10 or later. Download it from python.org if you don't have it. On macOS and some Linux systems, use python3 and pip3 instead of python and pip.

Use a virtual environment to avoid conflicts with other Python packages on your system:

python3 -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate

Install and verify

Install Selenium inside the virtual environment:

pip3 install selenium

Selenium Manager (bundled since Selenium 4.6) automatically downloads a compatible Chrome browser and matching ChromeDriver (the small program Selenium uses to control the browser). You don't need a separate driver installation step.

Verify the installation:

python3 -c "import selenium; print(selenium.__version__)"

Expected output:

4.43.0

Your version number may differ. Any Selenium 4.6 or later should work with the examples in this guide; 4.6 is when Selenium Manager (used above) shipped.

Minimal working scraper

Save this script as scraper.p​y and run it with python3 scraper.p​y:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

try:
    driver.get("https://books.toscrape.com/")
    print(f"Page title: {driver.title}")

    books = driver.find_elements(By.CSS_SELECTOR, "article.product_pod")
    print(f"Found {len(books)} books on the page\n")

    for book in books[:5]:
        title = book.find_element(By.CSS_SELECTOR, "h3 a").get_attribute("title")
        price = book.find_element(By.CSS_SELECTOR, ".price_color").text
        print(f"  {title} - {price}")
finally:
    driver.quit()

Expected output:

Page title: All products | Books to Scrape - Sandbox
Found 20 books on the page

  A Light in the Attic - £51.77
  Tipping the Velvet - £53.74
  Soumission - £50.10
  Sharp Objects - £47.82
  Sapiens: A Brief History of Humankind - £54.23

The try/finally block ensures the browser closes even if an error occurs.

How to find CSS selectors for any site: right-click an element in Chrome and select Inspect. The Copy selector option in Chrome (right-click the highlighted HTML → Copy → Copy selector) works as a starting point but is often too specific. It produces long nth-child chains like #root > div > div.App > main > article:nth-child(3) > h3 > a that can break whenever the site's HTML shifts. For selectors you'll reuse, rewrite them by hand using classes and IDs. In the example above, article.product_pod means "an <article> tag with the class product_pod", h3 a means "an <a> tag inside an <h3>", and .price_color means "any element with the class price_color".

find_element vs find_elements****: find_element (singular) returns the first matching element or raises NoSuchElementException if none exist. find_elements (plural) returns a Python list of all matches, or an empty list if none exist.

Beyond CSS selectors: Selenium supports several locator strategies via By.*. By.CSS_SELECTOR handles most cases cleanly and is the default in the examples here. Use By.XPATH when you need something CSS can't express, like selecting by visible text (//button[contains(text(), 'Submit')]) or walking to a parent or preceding sibling. By.ID, By.CLASS_NAME, and By.NAME are shortcuts for common single-attribute matches. Default to CSS; use XPath when you genuinely need it.

What happens when you scale this

The scraper works on books.toscrape.com because it's a sandbox with no rate limiting, no anti-bot protection, and no IP blocking. Most production sites add rate limits, bot protection, and IP-reputation checks, the standard defenses against abuse and overload that any scraper at real volume has to handle.

Run this same pattern against a production eCommerce site or search engine. After enough requests from the same IP, you'll typically start seeing CAPTCHAs, HTTP 429 (Too Many Requests), HTTP 403 Forbidden, or empty responses where data used to be. The threshold varies by site: sometimes dozens of requests, sometimes hundreds. Once the site's anti-bot system flags your IP, the signal usually appears as a status code, a CAPTCHA page, or a page that returns with zero products in the DOM.

The next few sections cover browser configuration and anti-detection techniques first; the proxy setup comes after, and it spreads your requests across many home IPs instead of one.

Production-ready environment flags for Selenium Python

The quickstart above works on a laptop. Moving the same scraper into Docker, CI, or a server introduces resource limits and Chrome-security-sandbox constraints that the minimal script doesn't handle. Two extra Chrome flags cover most of those failures.

How Selenium Manager resolves the driver

Selenium 4.6 introduced Selenium Manager, the built-in tool that resolved the driver in the quickstart. When you call webdriver.Chrome(), it checks for a compatible Chrome binary and ChromeDriver; if neither exists on your system, it downloads them to ~/.cache/selenium/. Knowing that location matters when you're debugging CI caches or pinning versions.

Headless flags for server and container environments

Use --headless=new. In Chrome 128+, --headless and --headless=new do the same thing, but --headless=new is clearer in code and should continue working if the Chrome defaults change. Chrome 132 (2025) removed the old stripped-down mode (--headless=old) from the standard Chrome binary entirely. It's now only available as a separate chrome-headless-shell binary.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=options)

try:
    driver.get("https://books.toscrape.com/")
    print(f"Page title: {driver.title}")
finally:
    driver.quit()

Expected output:

Page title: All products | Books to Scrape - Sandbox

If you're running Chrome in Docker as root, add --no-sandbox. Chrome won't start otherwise. On a normal laptop you don't need it, but keeping it is harmless for scraping workloads. The --disable-dev-shm-usage flag prevents Chrome from running out of shared memory on systems with limited /dev/shm space. Docker containers default to a 64MB /dev/shm, which Chrome can exceed once multiple tabs or heavy pages load; this flag forces Chrome to use /tmp instead.

Further reading: How to Scrape eBay with Python in 2026: Complete Guide and How to Scrape Glassdoor Jobs and Reviews with Python in 2026.

How to use Selenium for web scraping with stable waits

Pages that load content with JavaScript need time to finish rendering. Without proper waits, your scraper often reads the DOM before data appears and may return empty results, sometimes intermittently, which is the hardest version of this bug to debug.

Implicit vs explicit waits

You may see driver.implicitly_wait(10) in other tutorials. Implicit waits apply globally to every find_element call, waiting up to 10 seconds for any element before timing out. Explicit waits (WebDriverWait) wait for a specific condition on a specific element. Use explicit. Mixing the two causes unpredictable timing; the Selenium docs recommend using only explicit waits.

Explicit waits with expected conditions

Per the Selenium documentation:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

try:
    driver.get("https://books.toscrape.com/")

    wait = WebDriverWait(driver, 10)
    first_book = wait.until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "article.product_pod"))
    )
    title = first_book.find_element(By.CSS_SELECTOR, "h3 a").get_attribute("title")
    print(f"First book found via explicit wait: {title}")

    all_books = wait.until(
        EC.presence_of_all_elements_located((By.CSS_SELECTOR, "article.product_pod"))
    )
    print(f"Total books found: {len(all_books)}")
finally:
    driver.quit()

Expected output:

First book found via explicit wait: A Light in the Attic
Total books found: 20

The double parentheses in EC.presence_of_element_located((By.CSS_SELECTOR, "...")) are intentional. The function takes a single tuple argument (locator_strategy, value), so the outer pair is the function-call syntax and the inner pair builds the tuple that gets passed in.

WebDriverWait polls the DOM every 500ms (the default interval) for up to the timeout you specify (10 seconds in this example). Other useful conditions:

• EC.element_to_be_clickable – use before clicking a button that starts disabled (form submits that enable after validation)
• EC.visibility_of_element_located – use when an element is in the DOM but hidden by CSS until a tab or section opens
• EC.text_to_be_present_in_element – use for status indicators that change ("Processing…" → "Complete")
• EC.staleness_of – use after clicking a link or form submit to confirm the old page is gone before reading the new one

Waits handle timing issues inside the browser, but they can't help when the server rejects your connection because it flagged your IP.

How to scrape infinite-scroll pages with Selenium

Pages with infinite scroll (social feeds, product catalogs, search results) load new content as you scroll instead of paginating. There's no "next page" button – hitting the bottom triggers a JavaScript request that appends more items to the DOM. A naive scraper that loops on while True: scroll() never knows when to stop, and a fixed-iteration loop misses items if the page is slow.

The reliable pattern is: scroll, wait, count items, repeat. Stop when the count stops growing across three consecutive attempts. The example below uses quotes.toscrape.co​m/scroll, a public test site that loads 100 quotes in batches as you scroll:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

try:
    driver.get("https://quotes.toscrape.com/scroll")

    # Wait for the first batch to load.
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".quote"))
    )

    last_count = 0
    attempts_with_no_new_items = 0

    # Stop after 3 consecutive scrolls with no new items.
    while attempts_with_no_new_items < 3:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(2)  # Give the page time to load new items.

        current_count = len(driver.find_elements(By.CSS_SELECTOR, ".quote"))

        if current_count > last_count:
            print(f"Loaded {current_count} quotes")
            last_count = current_count
            attempts_with_no_new_items = 0
        else:
            attempts_with_no_new_items += 1

    print(f"Done. Total quotes loaded: {last_count}")

finally:
    driver.quit()

Expected output:

Loaded 40 quotes
Loaded 50 quotes
Loaded 60 quotes
Loaded 70 quotes
Loaded 80 quotes
Loaded 100 quotes
Done. Total quotes loaded: 100

The exact intermediate counts vary by run – the page often loads multiple batches in the time between scrolls, so you'll see counts jump in irregular increments (40, 50, 60, then maybe 80, 100). The final total is always 100, and the stop rule reliably catches the end.

The stop rule matters. Without it, a brief network slowness gets misread as "page is done" and the scraper exits with partial data. The counter resets every time new items appear, so a slow response just delays the stop, not triggers it falsely. Three consecutive failed attempts (about 6 seconds of nothing new) is a reasonable signal that the page is exhausted.

For slow sites, raise time.sleep(2) to 3 or 4 seconds. For sites with a "Load more" button instead of true infinite scroll, the same pattern works – replace scrollTo(...) with a click on the button: driver.find_element(By.CSS_SELECTOR, ".load-more").click().

How to scrape multiple pages without getting blocked in Selenium

Scraping a multi-page catalog requires detecting the next-page link, waiting for new DOM content, and avoiding IP blocks that can escalate as request volume grows.

Pagination with next-page navigation

Use EC.staleness_of to confirm the old page has been replaced before reading new data:

import time
import random
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

try:
    driver.get("https://books.toscrape.com/")
    wait = WebDriverWait(driver, 10)
    all_titles = []

    for page_num in range(1, 4):
        books = wait.until(
            EC.presence_of_all_elements_located(
                (By.CSS_SELECTOR, "article.product_pod")
            )
        )

        for book in books:
            title = book.find_element(
                By.CSS_SELECTOR, "h3 a"
            ).get_attribute("title")
            all_titles.append(title)

        print(f"Page {page_num}: scraped {len(books)} books")

        next_buttons = driver.find_elements(By.CSS_SELECTOR, "li.next a")
        if next_buttons:
            next_buttons[0].click()
            wait.until(EC.staleness_of(books[0]))
            time.sleep(random.uniform(1, 3))  # random delay to avoid rate limits
        else:
            print("No more pages")
            break

    print(f"\nTotal books scraped: {len(all_titles)}")
finally:
    driver.quit()

Expected output:

Page 1: scraped 20 books
Page 2: scraped 20 books
Page 3: scraped 20 books

Total books scraped: 60

The staleness_of(books[0]) wait pauses execution until the first book element from the previous page leaves the DOM, confirming the new page has loaded. The random.uniform(1, 3) delay works for light scraping; for stricter sites, use longer pauses (5–10 seconds) or mix short and long delays: time.sleep(random.choice([random.uniform(1, 3), random.uniform(8, 15)])) so the request interval isn't uniform. Adjust the range based on your target site's behavior.

Infinite-scroll pages (common on social feeds and product listings) don't have a next-page link. Scroll to trigger the next batch and wait for new items before reading: driver.execute_script("window.scrollTo(0, document.body.scrollHeight)"), then wait on element count growing past the previous value. Same principle as staleness_of: confirm new content has arrived before reading.

Why pagination triggers IP blocks

The script above loops over 3 pages. A real catalog has 50 or more, all requested from your single IP. Even with random delays, rate limiters still flag patterns like one IP pulling many pages in quick succession with identical browser fingerprints. Jittered delays don't hide the IP itself, so the site eventually returns a CAPTCHA, a 429, or empty data.

A common fix is to distribute these requests across multiple IP addresses using rotating residential proxies. Each request comes from a different residential IP, so the site receives traffic that looks closer to many individual users than one bot. The next few sections cover data export and browser-level anti-detection first; the proxy setup comes afterward in the proxy section.

How to export scraped data to CSV and JSON

CSV works for tabular data you plan to open in spreadsheets or pandas. JSON works when you need to preserve data types or add nested fields later:

import csv
import json
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

try:
    driver.get("https://books.toscrape.com/")
    wait = WebDriverWait(driver, 10)

    books = wait.until(
        EC.presence_of_all_elements_located(
            (By.CSS_SELECTOR, "article.product_pod")
        )
    )

    scraped_data = []
    for book in books:
        title = book.find_element(
            By.CSS_SELECTOR, "h3 a"
        ).get_attribute("title")
        price = book.find_element(By.CSS_SELECTOR, ".price_color").text
        rating_class = book.find_element(
            By.CSS_SELECTOR, "p.star-rating"
        ).get_attribute("class")
        rating = rating_class.replace("star-rating ", "")

        scraped_data.append({
            "title": title,
            "price": price,
            "rating": rating
        })

    # Save to CSV (newline="" prevents double-spaced rows on Windows)
    with open("books_output.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(
            f, fieldnames=["title", "price", "rating"]
        )
        writer.writeheader()
        writer.writerows(scraped_data)

    with open("books_output.json", "w", encoding="utf-8") as f:
        json.dump(scraped_data, f, indent=2, ensure_ascii=False)  # keeps £ signs readable

    print(f"Saved {len(scraped_data)} books to CSV and JSON")
finally:
    driver.quit()

Expected output:

Saved 20 books to CSV and JSON

Scraped prices are strings ("£51.77"). To use them in calculations, clean them during extraction: float(price.replace("£", "")). Clean data at scrape time. Cleaning it later across thousands of rows tends to be slower and more error-prone.

Validate the data before you ship it

Bad records pollute downstream pipelines and analytics. The cheapest fix is to validate at scrape time, fail loudly on bad data, and dedupe before writing to disk. The function below covers the seven checks that catch most real-world issues: non-empty titles, valid source URLs, numeric prices, parsed timestamps, duplicates removed, source URL stored, timestamp stored.

import re
from datetime import datetime, timezone
from urllib.parse import urlparse

def validate_and_clean(raw_records, source_url):
    """Clean prices, add metadata, dedupe, and validate every record."""
    timestamp = datetime.now(timezone.utc).isoformat(timespec="seconds")
    seen = set()
    cleaned = []

    for r in raw_records:
        # Dedupe by title
        if r["title"] in seen:
            continue
        seen.add(r["title"])

        # Parse price ("£51.77" -> 51.77)
        price = float(re.sub(r"[^\d.]", "", r["price"]))

        record = {
            "title": r["title"],
            "price": price,
            "rating": r["rating"],
            "source_url": source_url,
            "scraped_at": timestamp,
        }

        # Validate every field
        assert record["title"], "Empty title"
        parsed = urlparse(record["source_url"])
        assert parsed.scheme in ("http", "https") and parsed.netloc, "Invalid URL"
        assert record["price"] >= 0, "Negative price"
        datetime.fromisoformat(record["scraped_at"].replace("Z", "+00:00"))

        cleaned.append(record)

    return cleaned

# Replace this with scraped_data from the export script above.
raw = [
    {"title": "A Light in the Attic", "price": "£51.77", "rating": "Three"},
    {"title": "Tipping the Velvet", "price": "£53.74", "rating": "One"},
    {"title": "Tipping the Velvet", "price": "£53.74", "rating": "One"},  # duplicate
]

records = validate_and_clean(raw, "https://books.toscrape.com/")
print(f"Validated and deduped: {len(records)} records")
print(records[0])

Expected output:

Validated and deduped: 2 records

{'title': 'A Light in the Attic', 'price': 51.77, 'rating': 'Three', 'source_url': 'https://books.toscrape.com/', 'scraped_at': '2026-05-18T12:01:02+00:00'}

Pipe your scraped_data through validate_and_clean before exporting it to CSV or JSON. Run this as a separate step after scraping completes; assertions raise loudly the moment something is wrong, so bad records never reach disk. For larger pipelines, replace assertions with structured logging plus a quarantine list for bad records to keep the scraper running through partial failures.

How to speed up a Selenium scraper

Default Selenium settings wait for every resource (images, fonts, CSS, third-party scripts) before returning control. For scraping, you usually only need the DOM. Three flags cut page load time substantially without changing the data you extract:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--headless=new")

# Return as soon as DOMContentLoaded fires (don't wait for images, fonts, ads).
options.page_load_strategy = "eager"

# Don't fetch images at all. Saves bandwidth and rendering time.
options.add_experimental_option(
    "prefs",
    {"profile.managed_default_content_settings.images": 2}
)

driver = webdriver.Chrome(options=options)
try:
    start = time.time()
    driver.get("https://books.toscrape.com/")
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "article.product_pod"))
    )
    elapsed = time.time() - start
    print(f"Loaded in {elapsed:.2f}s")

    books = driver.find_elements(By.CSS_SELECTOR, "article.product_pod")
    print(f"Found {len(books)} books")
finally:
    driver.quit()

Expected output:

Loaded in 2.50s
Found 20 books

Exact timing depends on your network and machine – typical range is 2-4 seconds with these flags, 4-8 seconds without. The relative speedup matters more than the absolute number.

Other optimizations worth knowing: --disable-gpu (no rendering hardware acceleration, irrelevant in headless), --blink-settings=imagesEnabled=false (an alternative way to block images), and --disable-extensions (skip the extension load on launch). For sites that depend on JavaScript that runs after DOMContentLoaded, switch page_load_strategy back to "normal" – eager mode can return before the data you need has rendered.

How to scale Selenium scraping jobs

A single Chrome instance scrapes one page at a time. To scrape hundreds or thousands of URLs in reasonable time, run multiple browsers in parallel using Python's concurrent.futures.ProcessPoolExecutor. Each worker process owns its own Chrome instance, so they don't share state and one crash doesn't kill the rest.

import time
from concurrent.futures import ProcessPoolExecutor, as_completed
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

URLS = [
    "https://books.toscrape.com/catalogue/page-1.html",
    "https://books.toscrape.com/catalogue/page-2.html",
    "https://books.toscrape.com/catalogue/page-3.html",
    "https://books.toscrape.com/catalogue/page-4.html",
]

def scrape_one(url):
    """Spin up a fresh Chrome, scrape one page, tear it down."""
    options = Options()
    options.add_argument("--headless=new")
    options.page_load_strategy = "eager"
    driver = webdriver.Chrome(options=options)
    try:
        driver.get(url)
        WebDriverWait(driver, 10).until(
            EC.presence_of_all_elements_located(
                (By.CSS_SELECTOR, "article.product_pod")
            )
        )
        books = driver.find_elements(By.CSS_SELECTOR, "article.product_pod")
        return [
            {
                "title": b.find_element(By.CSS_SELECTOR, "h3 a").get_attribute("title"),
                "price": b.find_element(By.CSS_SELECTOR, ".price_color").text,
            }
            for b in books
        ]
    finally:
        driver.quit()

if __name__ == "__main__":
    start = time.time()
    all_books = []
    with ProcessPoolExecutor(max_workers=4) as executor:
        futures = {executor.submit(scrape_one, url): url for url in URLS}
        for future in as_completed(futures):
            try:
                all_books.extend(future.result())
            except Exception as e:
                print(f"Failed {futures[future]}: {e}")
    elapsed = time.time() - start
    print(f"Scraped {len(all_books)} books across {len(URLS)} pages in {elapsed:.2f}s")

Expected output:

Scraped 80 books across 4 pages in 19.40s

Timing varies, typical range is 15-25 seconds for 4 pages with 4 workers, compared to roughly 30-40 seconds sequentially. The speedup grows with more URLs.

Three patterns to know when scaling beyond this example. First, restart each worker's Chrome periodically (every 50-100 pages) to release accumulated memory. Second, wrap scrape_one in retry logic with exponential backoff – at scale, transient failures (network blips, brief 429s) are routine, and one bad page shouldn't sink the batch. Third, when you need more than a single machine, move from ProcessPoolExecutor to a task queue (Celery, RQ, or a managed service like AWS Batch) so workers can run on different machines pulling from the same job queue.

At higher scale, the bottleneck shifts from CPU to network and IP reputation. That's where rotating residential proxies become necessary – covered in the proxies section below.

How to reduce bot detection in Selenium web scraping

Most anti-bot systems fingerprint Selenium through a small set of well-known signals: navigator.webdriver returning true in automated Chrome (and false in normal Chrome), injected cdc_ globals that ChromeDriver adds and anti-bot scripts look for, and user agent strings containing "HeadlessChrome". The first code block below handles navigator.webdriver and the user agent. SeleniumBase, covered afterward, handles cdc_ variables.

Disable automation flags

Remove the signals that Chrome sets when controlled by Selenium:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_experimental_option("excludeSwitches", ["enable-automation"])

user_agent = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) "
    "Chrome/147.0.0.0 Safari/537.36"  # match your Chrome version (chrome://settings/help)
)
options.add_argument(f"user-agent={user_agent}")

driver = webdriver.Chrome(options=options)

# Remove the webdriver property via Chrome DevTools Protocol (CDP)
driver.execute_cdp_cmd(
    "Page.addScriptToEvaluateOnNewDocument",
    {"source": "Object.defineProperty(navigator, 'webdriver', "
     "{get: () => undefined})"}
)

try:
    driver.get("https://books.toscrape.com/")
    webdriver_flag = driver.execute_script("return navigator.webdriver")
    ua = driver.execute_script("return navigator.userAgent")
    print(f"navigator.webdriver = {webdriver_flag}")
    print(f"User agent contains 'HeadlessChrome': {'HeadlessChrome' in ua}")
finally:
    driver.quit()

Expected output:

navigator.webdriver = None
User agent contains 'HeadlessChrome': False

The --disable-blink-features=AutomationControlled flag prevents Chrome from setting navigator.webdriver = true. The excludeSwitches option disables internal automation flags anti-bot scripts detect. The CDP command overrides navigator.webdriver to return undefined (Python sees this as None).

Real anti-bot systems check dozens of additional signals (screen size, WebGL renderer, timezone, navigator.plugins, fonts, canvas fingerprints), the flags above handle the well-known ones. Treat this as a baseline; for well-protected targets, jump to SeleniumBase below.

Match the user agent version to your installed Chrome (check at chrome://settings/help). Anti-bot systems detect mismatches between the reported version and the browser's real TLS fingerprint, so update this whenever Chrome updates.

SeleniumBase for stronger anti-detection

For sites with stronger anti-bot protection, SeleniumBase builds on Selenium with stealth helpers like UC Mode and CDP Mode. Install it:

pip3 install seleniumbase

UC (Undetected Chromedriver) Mode patches the ChromeDriver binary at runtime to remove cdc_ variables, disables several automation switches Chrome sets on launch, and adjusts the initial CDP handshake. CDP Mode bypasses WebDriver entirely by communicating with Chrome through DevTools Protocol:

from seleniumbase import SB

with SB(uc=True, test=True, locale="en") as sb:
    sb.activate_cdp_mode("https://target-site.com")
    sb.sleep(2)
    # Built-in method to handle Cloudflare challenges
    sb.uc_gui_click_captcha()
    sb.sleep(2)
    content = sb.get_page_source()

Replace target-site.co​m with the URL you want to scrape. This block isn't runnable as-is because it needs a real target site with Cloudflare protection.

uc_gui_click_captcha() requires a visible browser window because it moves the OS mouse cursor, UC Mode doesn't work in headless mode. On headless Linux servers, wrap the script with Xvfb (xvfb-run python3 scraper.p​y) or pass xvfb=True to SB(...).

These techniques handle fingerprint and behavioral signals, not IP reputation. Flagged datacenter ranges, known proxy IPs, and recently rate-limited addresses still dominate the decision. Other libraries solving similar problems: undetected-chromedriver (drop-in ChromeDriver patch) and nodriver (skips WebDriver entirely). The IP problem is next.

How to use residential proxies with Selenium

Browser-level stealth and IP reputation are two different problems.

Why does my scraper still get IP-blocked even with stealth flags?

Pagination and fingerprint evasion don't help when the IP itself is flagged. Rotating proxies handle the IP side by routing your requests through different IP addresses, and residential proxies route through real home IPs assigned by consumer ISPs (Internet Service Providers). Most anti-bot systems check each IP's ASN (Autonomous System Number). Datacenter IPs are cheap and fast, but their public IP ranges make them easy for anti-bot vendors to flag. Consumer ISP ranges typically resemble normal user traffic (a side-by-side breakdown covers the trade-offs in detail). Residential IPs generally have higher acceptance rates, but actual success still depends on IP quality and how you pace your requests.

The request flow: your scraper drives Selenium, which controls Chrome (with anti-detection flags applied). Outbound traffic routes through a rotating residential proxy pool, so the target site sees traffic that looks like it's coming from many different home IPs instead of one address it can flag.

Pair SeleniumBase with residential proxies

SeleniumBase accepts proxy credentials directly in its constructor using user:pass@host:port format:

from seleniumbase import SB

# Replace with your proxy provider credentials.
# Format: USERNAME-ACCESS_CODE:PASSWORD@HOST:PORT
#   LV71125532    = USERNAME
#   mDmfksl3onyoy = ACCESS_CODE
proxy = "LV71125532-mDmfksl3onyoy:[email protected]:7383"

with SB(uc=True, test=True, proxy=proxy) as sb:
    sb.activate_cdp_mode("https://target-site.com")  # replace with your target URL
    sb.sleep(2)
    content = sb.get_page_source()

This block requires real proxy credentials, a real target URL, and a visible browser window (UC Mode doesn't work in headless mode, as noted in the anti-detection section).

Proxy format note: SeleniumBase requires user:pass@host:port. Different proxy providers present credentials in different formats: some use host:port:user:pass, others use user:pass@host:port or a URL like htt​p://user:pass@host:port. Rearrange the values from your provider's dashboard to match SeleniumBase's format. The standard Selenium extension approach below takes host, port, user, and password as separate parameters, so it works with any provider regardless of how they display their credentials.

Why do shared proxy pools still trigger blocks on target sites?

Shared residential proxy pools can still create problems when many users send traffic to the same target from overlapping IPs. If another scraper has already used the same IP heavily on that site, the IP may carry risk before your scraper even starts. This is why proxy allocation matters, not only proxy type.

Live Proxies allocates IPs per target. IPs you use for 1 target aren't assigned to other customers scraping that same target, which reduces overlap on your specific site. This helps avoid shared-pool noise from other users, though IP reputation still depends on an IP's prior activity and each target's own detection logic, not allocation alone.

Proxy authentication in standard Selenium

The Chrome --proxy-server flag accepts a host and port but doesn't support inline username and password authentication. The following helper function creates a lightweight Chrome extension at runtime that handles the authentication handshake:

import atexit
import zipfile
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

def create_proxy_driver(host, port, user, password):
    """Create a Chrome driver with authenticated proxy support."""
    manifest = (
        '{"version":"1.0","manifest_version":3,"name":"Proxy",'
        '"permissions":["proxy","webRequest","webRequestAuthProvider"],'
        '"host_permissions":["<all_urls>"],'
        '"background":{"service_worker":"bg.js"}}'
    )

    background = """
    chrome.proxy.settings.set({
        value: {
            mode: "fixed_servers",
            rules: {
                singleProxy: {scheme:"http", host:"%s", port:parseInt(%s, 10)},
                bypassList: ["localhost"]
            }
        },
        scope: "regular"
    });
    chrome.webRequest.onAuthRequired.addListener(
        (details, callback) => {
            callback({authCredentials: {username:"%s", password:"%s"}});
        },
        {urls:["<all_urls>"]},
        ["asyncBlocking"]
    );
    """ % (host, port, user, password)

    ext_path = os.path.join(os.getcwd(), "proxy_auth.zip")
    with zipfile.ZipFile(ext_path, "w") as zp:
        zp.writestr("manifest.json", manifest)
        zp.writestr("bg.js", background)

    options = Options()
    options.add_argument("--headless=new")
    options.add_extension(ext_path)
    driver = webdriver.Chrome(options=options)

    # Defer cleanup: Chrome on Windows holds the file handle, so removing
    # immediately after Chrome() returns raises PermissionError [WinError 32].
    atexit.register(lambda: os.path.exists(ext_path) and os.remove(ext_path))
    return driver

# Replace with your proxy provider credentials
driver = create_proxy_driver(
    host="131.XXX.XXX.36",
    port="7383",
    user="LV71125532-mDmfksl3onyoy",
    password="YOUR_PASSWORD"
)

try:
    driver.get("https://httpbin.org/ip")
    print(driver.find_element(By.TAG_NAME, "body").text)
finally:
    driver.quit()

Expected output (the IP shown will be your proxy's address, not your real IP):

{
  "origin": "203.0.113.42"
}

The extension intercepts the Chrome onAuthRequired event and supplies your credentials automatically. The function works with any proxy provider that uses HTTP basic authentication.

Chrome version note: Branded Chrome 137+ removed support for loading extensions via the command line (--load-extension). This affects only Google-branded Chrome. Chromium and Chrome for Testing (CfT) still support extension loading. If add_extension() fails silently, Selenium Manager is likely using your system's branded Chrome. Fix it by deleting or renaming your branded Chrome binary so Selenium Manager downloads CfT to ~/.cache/selenium/ instead.

Credential escaping: The %s placeholders in the JavaScript mean your proxy credentials shouldn't contain literal % characters (escape them as %%). Passwords containing double quotes (") will break the generated JavaScript string. Most provider dashboards don't include these characters, but if you adapt this for another provider, sanitize or escape credentials first.

The webRequestAuthProvider permission works in Selenium-launched Chrome, but enterprise-managed Chrome installs may enforce different extension policies that block it.

Rotate proxies across requests

Using the create_proxy_driver function from the previous code block, create a new driver instance with different credentials for each batch. Each new driver connects through a new proxy IP:

proxy_list = [
    ("131.XXX.XXX.36", "7383", "LV71125532-mDmfksl3onyoy-1", "YOUR_PASSWORD"),
    ("131.XXX.XXX.36", "7383", "LV71125532-mDmfksl3onyoy-2", "YOUR_PASSWORD"),
    ("131.XXX.XXX.36", "7383", "LV71125532-mDmfksl3onyoy-3", "YOUR_PASSWORD"),
]

for host, port, user, pwd in proxy_list:
    driver = create_proxy_driver(host, port, user, pwd)
    try:
        driver.get("https://target-site.com/page")  # replace with your target URL
        # Extract page data and append to results list
    finally:
        driver.quit()

Performance note: Creating a new Chrome instance per proxy is slow: startup typically takes a couple of seconds per instance, and each one consumes several hundred MB of RAM (measure on your own system; loaded pages use more). If you don't need different credentials per batch, use the rotating proxy format (no session ID, or SID) shown in the sticky sessions section below. It rotates IPs server-side, so a single Chrome instance handles all requests.

When to use sticky sessions

Some scraping flows require the same IP address across multiple requests. Login flows, multi-step checkouts, and paginated APIs that track your position with a server-side token need consistent IP addresses. Sticky sessions prevent IP rotation in a single session by assigning a session ID to each proxy connection. In the examples here, a session ID keeps you on the same IP for up to 24 hours (a peer disconnecting can end the session earlier). After that window, the session expires and the next request rotates to a new IP. Other providers use different session lengths, so check your provider's docs if you're adapting this elsewhere.

Credentials in this guide follow the format IP:PORT:USERNAME-ACCESS_CODE-SID:PASSWORD. The session ID (SID) at the end controls sticky versus rotating behavior:

# Sticky session: same IP for up to 24 hours
# The SID (-1, -2, etc.) after the access code locks each proxy to one IP
sticky_proxy = "131.XXX.XXX.36:7383:LV71125532-mDmfksl3onyoy-1:YOUR_PASSWORD"

# Rotating session - different IP on every request
# Remove the SID and each request rotates through your allocated IP pool
rotating_proxy = "131.XXX.XXX.36:7383:LV71125532-mDmfksl3onyoy:YOUR_PASSWORD"

The B2C dashboard displays proxies in the sticky format by default. The rotating form isn't listed. You build it yourself by dropping the -SID segment while keeping the same access code and password.

When scraping all 50 pages of a catalog, a sticky session usually keeps you on the same IP for the whole crawl if it finishes in the 24-hour window, which helps avoid mid-session blocks caused by IP changes. For scraping many independent pages (product listings, search results), rotating sessions spread your requests across every IP in your allocated pool.

Enterprise (B2B) clients use a gateway hostname instead of a direct IP, with both formats listed separately in the B2B dashboard:

Rotating:  b2b.liveproxies.io:7383:username-access_code:password
Sticky:    b2b.liveproxies.io:7383:username-access_code-sid:password

Common Selenium web scraping errors and how to fix them

Use this table for quick error triage. The "Fix" column points to the pattern; full implementations use the wait and retry patterns shown earlier in the guide.

Build your production Selenium scraper

4 principles help keep production scrapers running instead of failing unexpectedly:

• Use explicit waits for DOM elements. WebDriverWait with expected_conditions adapts to actual load times and replaces hard-coded sleeps for element detection. Between page requests, add short random delays with time.sleep(random.uniform(1, 3)) to avoid triggering rate limiters – these inter-request pauses serve a different purpose than DOM waits.
• Rotate IPs at scale. A single IP gets flagged after enough requests on most production sites – sometimes at dozens, sometimes at hundreds. Rotating residential proxies spread requests across different home IPs, which usually delays or avoids that flag under steady load.
• Close the browser. Always wrap your scraper in try/finally with driver.quit(). Leaked Chrome processes accumulate quickly, and a long-running script with missed quit() calls can exhaust memory on a small VM.
• Log with timestamps. Replace print() with the Python logging module (logging.basicConfig(level=logging.IN​FO, format='%(asctime)s %(message)s')). When a scraper fails at page 347 of 500, a timestamped log tells you when it failed. Pair the timestamp with context (current URL, selector, page number) to know where.

This script combines pagination and CSV export. Run it with python3 scraper.py:

import csv
import time
import random
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

# Write CSV header before scraping so each page can append incrementally
with open("books_full.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price"])
    writer.writeheader()

total = 0

try:
    driver.get("https://books.toscrape.com/")
    wait = WebDriverWait(driver, 10)

    for page_num in range(1, 4):
        books = wait.until(
            EC.presence_of_all_elements_located(
                (By.CSS_SELECTOR, "article.product_pod")
            )
        )

        page_data = []
        for book in books:
            title = book.find_element(By.CSS_SELECTOR, "h3 a").get_attribute("title")
            price = book.find_element(By.CSS_SELECTOR, ".price_color").text
            page_data.append({"title": title, "price": price})

        # Append after each page - a crash won't lose previously saved data
        with open("books_full.csv", "a", newline="", encoding="utf-8") as f:
            writer = csv.DictWriter(f, fieldnames=["title", "price"])
            writer.writerows(page_data)

        total += len(page_data)
        print(f"Page {page_num}: scraped {len(books)} books")

        next_buttons = driver.find_elements(By.CSS_SELECTOR, "li.next a")
        if next_buttons:
            next_buttons[0].click()
            wait.until(EC.staleness_of(books[0]))
            time.sleep(random.uniform(1, 3))
        else:
            break
finally:
    driver.quit()

print(f"\nDone - saved {total} books to books_full.csv")

Expected output:

Page 1: scraped 20 books
Page 2: scraped 20 books
Page 3: scraped 20 books

Done - saved 60 books to books_full.csv

To scale this to production, replace the direct driver with create_proxy_driver from the proxy section and use a rotating (no-SID) proxy format. Requests then spread across different residential IPs, so the target site receives them from many addresses instead of one.

Further reading: How to Scrape Facebook: Comments, Posts, Groups, Marketplace, and Scraping Tools in 2026 (Guide) and How to Scrape YouTube: A Complete Guide to Videos, Comments, and Transcripts (2026).

Wrapping up

You walked through Selenium scraping from the ground up: a minimal scraper, explicit waits for JavaScript-rendered content, infinite scroll and pagination patterns, exporting to CSV/JSON with validation, and scaling techniques (faster page loads, parallel browsers) for production volume.

For real-world targets, the same scraper that works on books.toscrape.co​m gets blocked on protected sites within hours. That's why anti-detection flags, SeleniumBase UC Mode, and rotating residential proxies have their own sections. Reach for them when the target requires them; skip them when it doesn't.

The Selenium fundamentals haven't changed much in 2026, but the anti-bot landscape has. Bots are now more than half of internet traffic, and most production sites have some kind of protection in place. Build your scraper to be polite (rate-limited, properly validated, ToS-respecting) and you'll have far fewer fires to put out.

FAQs

How do you scrape dynamic content with Selenium without getting blocked?

Combine three layers: Chrome flags that disable automation signals (--disable-blink-features=AutomationControlled, excludeSwitches), randomized 1–3 second delays, and rotating residential proxies. IP reputation matters most; stealth flags alone rarely pass a datacenter IP.

How do you wait for JavaScript-rendered content in Selenium?

Use WebDriverWait with EC.presence_of_element_located((By.CSS_SELECTOR, "your-selector")). It polls the DOM until the element appears, avoiding the guesswork of time.sleep(). For SPAs, wait on a specific data container, not the page load event, which fires before async data arrives.

How fast is Selenium for web scraping?

Speed depends on page weight, network, and the delays you add. A 500-page catalog with 2-second random delays spends about 17 minutes in delays alone, so expect 20+ minutes end-to-end in single-threaded runs. For more speed, parallelize with multiple Chrome instances or switch to Playwright.

What is the best alternative to Selenium for web scraping?

Playwright is the closest direct alternative: often faster and lighter on memory, with auto-waiting and network interception built in. For static pages, requests with BeautifulSoup is faster. For large-scale crawling, Scrapy adds built-in concurrency. High-volume scrapers typically need rotating IPs.

When should I use SeleniumBase instead of plain Selenium?

Start with plain Selenium. It's the W3C standard, well-documented, and enough for most scraping against sites without strong anti-bot protection. Switch to SeleniumBase when you hit Cloudflare-class protection or need its stealth features (UC Mode, CDP Mode) and built-in retry patterns that plain Selenium doesn't ship with.

Is Selenium web scraping legal?

It depends on what you scrape and where. Public data is generally defensible under the US CFAA. Scraping behind logins, personal data (GDPR, CCPA), or ToS violations carry separate risks. Check robots.txt and ToS first.