BeautifulSoup Web Scraping: How to Build a Web Scraper With Python (Step-By-Step Tutorial)

BeautifulSoup web scraping combines 2 Python libraries: requests downloads a page, and BeautifulSoup presents that HTML as a tree you can query by tag, class or CSS selector. BeautifulSoup doesn't do the parsing itself. That job goes to a parser you choose.

Give <ul><li>Alpha<li>Beta<li>Gamma</ul> to html.parser and the first item reads AlphaBetaGamma. Give it to lxml and it reads Alpha. Both report 3 items, and neither raises. That markup is legal, because the HTML Standard lets you drop an li end tag when another li follows.

Tools built to feed pages to language models ship BeautifulSoup as a hard dependency: crawl4ai, scrapegraphai and llama-index-readers-web all pin beautifulsoup4. This guide builds one scraper file from a single page to a validated CSV, which is where an AI web scraping pipeline starts.

TL;DR

BeautifulSoup web scraping fails quietly, so the skill is building the checks that catch it.

• Check view source first. The browser shows things that requests never receives.
• Name the parser and pass response.content. Defaults differ by machine, and .text corrupts non-ASCII.
• Test for [] after every search, or a selector matching nothing gives an empty dataset and no error.
• Parse prices with something that keeps the currency: the usual regex reads € 357,76 as 35776.
• Cap page counts, confirm a 200 holds your element, and add rotating proxies once one IP limits you.

What Is Web Scraping Using BeautifulSoup in Python?

Web scraping using BeautifulSoup is the process of downloading a page's HTML with an HTTP client, then using BeautifulSoup to turn that HTML into a searchable tree you can pull values from.

A fetch problem gives you the wrong bytes, and a parse problem gives you the wrong values out of the right bytes.

If you're new to the wider ecosystem, the Python web scraping tutorial covers how these pieces fit alongside Scrapy and Playwright.

The pages that work on day one usually share a shape: many similar records, each in its own repeating block of HTML.

• Directory listings, where each row is a company or a person, common in lead research. Rows describing identifiable people are personal data under the GDPR and the CCPA, so that use case needs a lawful basis before you collect, not after.
• Blog headlines, where you want the title, link and date
• Product listing pages, where you want name, price and availability, which is the basis of much price monitoring work

For records like these you're aiming for a table like this one:

The URLs are shortened here for width, though the real CSV carries them in full. When a price looks wrong later, source_url and scraped_at are what let you re-fetch that exact page and tell a site redesign apart from a parser bug.

How BeautifulSoup fits in a Python web scraper

BeautifulSoup sits in the middle of a pipeline:

Request → Parse → Extract → Validate → Save

BeautifulSoup handles only the parse and extract steps, and it runs the parse through the parser you chose. It never touches the network. If you hand it a URL string instead of HTML, it warns you rather than fetching anything:

BeautifulSoup("https://books.toscrape.com/", "lxml")
# MarkupResemblesLocatorWarning: The input passed in on this line looks
# more like a URL than HTML or XML.

That warning exists because passing a URL is such a common first mistake.

What BeautifulSoup cannot do

BeautifulSoup doesn't execute JavaScript. It reads the HTML you hand it and nothing more, which with requests is whatever the server sent. If a site fetches its data after the page loads, that data isn't in what the server sent you, so BeautifulSoup never sees it.

The practical test takes 2 requests. Both of these URLs show a list of quotes in a browser:

import requests
from bs4 import BeautifulSoup
for url in ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/js/"]:
    response = requests.get(url, timeout=10)
    soup = BeautifulSoup(response.content, "lxml")
    print(f"{url:<32}-> {len(soup.select('div.quote'))} matches")

The 2 URLs answer differently:

https://quotes.toscrape.com/    -> 10 matches
https://quotes.toscrape.com/js/ -> 0 matches

Same data, same site, same selector. The second page builds its list in the browser, so BeautifulSoup sees an empty container.

That doesn't mean you need a headless browser. On pages like this one the data is usually still in the response, sitting in a script tag as JSON.

When Should You Use BeautifulSoup for Web Scraping?

Use BeautifulSoup when the data is already in the HTML that the server sends you.

BeautifulSoup is the right tool when:

• The values you want appear in view page source
• The markup is reasonably stable between visits
• You want something cheap and fast, since parsing a 51 KB page takes about 5 ms, without the memory and startup cost of a browser

It isn't enough when:

• Content appears only after a scroll, click or login
• The page assembles itself from API calls that the browser makes later

The decision rule: open the page and view its source, with Ctrl+U on Windows and Linux or Cmd+Option+U on a Mac, then search for a value you want. Use view source, not Inspect Element. Inspect shows you the live DOM after JavaScript has run, which isn't what requests receives. If your value is in view source, you don't need a browser.

The question that comes up first: why not just hand the HTML to a language model? Because the two aren't alternatives. Several of the tools built for exactly that job ship BeautifulSoup as a hard dependency:

The reason is arithmetic. Models are billed per token, and raw HTML is mostly markup. On the 9 pages this guide scrapes, reducing a page with BeautifulSoup first removed 82 to 94% of the tokens that a model would otherwise read, measured with tiktoken on the cl100k_base encoding.

Parse deterministically, and spend model tokens only where determinism genuinely fails: reconciling many sites that describe the same thing differently, classifying text you've already extracted, or writing the selectors in the first place. A model proposes the selector once, you check it, and then you run it a million times without paying the model again.

One failure mode is worth knowing because it looks like a model problem and isn't. Hand a model the HTML of a JavaScript-rendered page and ask for selectors, and it will often invent plausible ones for elements that were never in the document you sent. It's answering from what a page like that usually looks like, because the values genuinely aren't there. That's the same condition the view source test detects, and it's cheaper to catch in view source than with an API bill. A model can't see markup that the server didn't send you, and neither can BeautifulSoup.

BeautifulSoup vs. Selenium

Selenium drives a real browser, so JavaScript runs and the DOM fills in. BeautifulSoup only parses HTML you already have.

The split depends on whether the content exists before the browser touches it:

• A job board that loads more roles when you click "show more" usually needs Selenium or a similar tool, because those roles don't exist until the click happens. If that click only fires an API call though, you can make the call yourself.
• A static blog archive needs BeautifulSoup, because every post title is already in the HTML. Running a browser here just makes the job slower and heavier for no benefit.

If you need the browser route, the Selenium web scraping guide covers driver setup, pagination and proxy authentication.

BeautifulSoup vs Playwright

Playwright is a newer browser automation tool than Selenium. Against BeautifulSoup though, the comparison is the same one as Selenium, because the question isn't which browser tool is nicer. It's whether you need a browser at all.

Answer that first. If the data is in view source, neither browser tool is worth its cost. If it isn't, you need a headless browser, and Selenium or Playwright runs one.

BeautifulSoup vs. Scrapy

Scrapy is a crawling framework. It handles scheduling, concurrency, retries, pipelines and export. BeautifulSoup is a parser and nothing else, so the two aren't competitors.

You can use both together. Scrapy comes with its own selector API, but nothing stops you from passing a response body to BeautifulSoup inside a spider callback when you want its more forgiving search API. A reasonable rule: reach for Scrapy once you're crawling thousands of pages across many sites and want that machinery handled for you. For the middle ground, the difference between a web crawler and a scraper is worth understanding before you pick.

How to Do Web Scraping Using BeautifulSoup in 10 Minutes

In 10 minutes you go from an empty folder to a CSV of 20 products, in 3 steps: install, fetch and parse, then write the file. The target is books.toscrape.com, a sandbox built for this purpose whose own banner reads "We love being scraped!". It has pagination, detail pages and spec tables, which is everything the step-by-step build needs.

Quick setup

Create a virtual environment and install the packages:

python3 -m venv .venv            # Windows: py -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install beautifulsoup4 requests lxml html5lib pandas

requests fetches the page and beautifulsoup4 gives you the tree to search. lxml is the one worth explaining: it's a separate parser that BeautifulSoup can use, faster than the built-in one and, more importantly, it handles broken markup close to the way a browser does. For a wider look at the options, see how data parsing works. html5lib is a third parser, installed here only because later sections compare it against the other 2. pandas is used once, much later, for the validation report.

Confirm what you actually installed, because version drift is a big reason why old tutorials stop working:

import bs4, requests, lxml.etree
print(bs4.__version__, requests.__version__, lxml.etree.__version__)

Yours will differ, and that's the point of checking:

4.15.0 2.34.2 6.1.1

The code here runs on beautifulsoup4 4.15.0, soupsieve 2.9, requests 2.34.2, lxml 6.1.1, html5lib 1.1, pandas 3.0.3 and Python 3.14.5, except where a section deliberately runs an older version to show what changed. The measurements are ours and the proxy and geo-targeting tests ran through Live Proxies residential exits.

Keep everything from here in one file, adding to the same script rather than replacing it. The parse_card function you write in a moment is the one the crawler calls on every page. Not everything belongs in that file though. The short blocks that show a warning or an error in a comment are demonstrations to run on their own, and a few others parse a snippet of markup written inline to make one point.

Troubleshooting the install. If you see bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml, you asked BeautifulSoup for a parser you didn't install. Run pip install lxml. On macOS, an SSL: CERTIFICATE_VERIFY_FAILED error often means you never ran Install Certificates.command from your Python install folder.

Minimal working scraper

A working scraper is one fetch, one parse and one loop:

import requests
from bs4 import BeautifulSoup

URL = "https://books.toscrape.com/"

response = requests.get(URL, timeout=10)
response.raise_for_status()

# Pass .content (bytes), not .text. This site sends no charset in the
# Content-Type header, so requests would fall back to ISO-8859-1 and the
# pound sign would come out as "£". BeautifulSoup reads the meta tag instead.
soup = BeautifulSoup(response.content, "lxml")

for card in soup.select("article.product_pod")[:5]:
    link = card.select_one("h3 a")
    print({
        # the visible text is truncated ("A Light in the ..."),
        # the full title lives in the title attribute
        "title": link["title"],
        "price": card.select_one("p.price_color").get_text(strip=True),
        "url": link["href"],
    })

That prints 5 dictionaries, one per card:

{'title': 'A Light in the Attic', 'price': '£51.77', 'url': 'catalogue/a-light-in-the-attic_1000/index.html'}
{'title': 'Tipping the Velvet', 'price': '£53.74', 'url': 'catalogue/tipping-the-velvet_999/index.html'}
{'title': 'Soumission', 'price': '£50.10', 'url': 'catalogue/soumission_998/index.html'}
{'title': 'Sharp Objects', 'price': '£47.82', 'url': 'catalogue/sharp-objects_997/index.html'}
{'title': 'Sapiens: A Brief History of Humankind', 'price': '£54.23', 'url': 'catalogue/sapiens-a-brief-history-of-humankind_996/index.html'}

In that output, 2 things aren't ready to use. The URLs are relative, and the prices are strings.

Export to CSV

Exporting is csv.DictWriter over a list of dictionaries, and it's the point where the rows change shape in 3 ways. URLs resolve to absolute, the price becomes a number with its currency stored beside it, and every row records where it came from and when. The card already carries a rating and a stock flag, so you get those at no extra cost.

The code is reorganised too. Extraction moves into a parse_card() function, because the crawler reuses it unchanged, and the loop now covers all 20 cards rather than the first 5:

import csv
import re
from datetime import datetime, timezone
from decimal import Decimal
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

URL = "https://books.toscrape.com/"
WORDS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
CURRENCY = {"£": "GBP", "$": "USD", "€": "EUR"}


def parse_card(card, source_url):
    link = card.select_one("h3 a")
    price_text = card.select_one("p.price_color").get_text(strip=True)
    # Every price on this page reads "£51.77", so the symbol is one character
    # and the rest is the number. Store the currency beside the amount rather
    # than putting it in the column name. Real sites write prices in formats
    # that break this split, which parse_price handles.
    symbol, amount = price_text[0], price_text[1:]
    price, currency = Decimal(amount), CURRENCY.get(symbol, symbol)

    # "star-rating Three" -> 3. The rating is in the class, not the text.
    classes = card.select_one("p.star-rating")["class"]
    rating = next((WORDS[c] for c in classes if c in WORDS), None)

    return {
        "title": link["title"],
        "price": price,
        "currency": currency,
        "rating": rating,
        "in_stock": "In stock" in card.select_one("p.availability").get_text(),
        "url": urljoin(source_url, link["href"]),
        "source_url": source_url,
        "scraped_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    }


response = requests.get(URL, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.content, "lxml")

rows = [parse_card(c, URL) for c in soup.select("article.product_pod")]

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

print(f"wrote {len(rows)} rows to books.csv")

The file has a header row and 20 records. Here are the first 2:

wrote 20 rows to books.csv

title,price,currency,rating,in_stock,url,source_url,scraped_at
A Light in the Attic,51.77,GBP,3,True,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html,https://books.toscrape.com/,2026-07-19T02:13:31+00:00
Tipping the Velvet,53.74,GBP,1,True,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html,https://books.toscrape.com/,2026-07-19T02:13:31+00:00

There is one more encoding trap on the way out. Excel on Windows can read a plain utf-8 CSV as cp1252, depending on version and locale. When it does, a price like €40,96 or a name like Café comes back as €40,96 and Café, the same £ mojibake from the other direction.

If a person will open the file in Excel, write it with encoding="utf-8-sig", which adds a byte-order mark that Excel recognises. If a program will read it, keep plain utf-8, because that mark can confuse a naive parser.

Most of the sandbox titles are ASCII, but 9 of the 1,000 aren't: one quotes the poem Pangur Bán, another names a Café, and 7 carry a curly apostrophe. So the trap is live even here, and the moment your rows carry a £, a ç or a ü, that one argument is the difference between clean data and a column of garbage.

Common quick start errors

Just 7 errors account for the bulk of failed first runs.

That last row deserves a warning. BeautifulSoup 4.15.0, is the release its own changelog calls the last to tolerate the old camelCase API. The changelog is unusually direct about what happens next:

In a subsequent point release, the DeprecationWarning issued when you use these obsolete features will be replaced by NotImplementedError, giving you a final chance to change your code before the implementations are removed entirely. Once the features are removed, code that tries to use them will start behaving strangely, since Beautiful Soup will generally interpret the method and attribute names as tag names.

So soup.findAll will eventually be read as "find a tag called findAll", which gives you None, so calling it raises TypeError: 'NoneType' object is not callable and your error message never mentions findAll at all. Affected names include findAll, findNext, findParent, replaceWith, nextSibling and renderContents. Rename them now while the warnings still tell you where they are.

How to Inspect a Page Before Web Scraping Using BeautifulSoup

Open DevTools, with F12 on Windows and Linux or Cmd+Option+I on a Mac, click the element picker, and hover over one item in the list. View source told you whether the data is there at all, and DevTools is for the next question: what to select. You're looking for the smallest repeating container that holds one complete record.

On the target page that container is article.product_pod, and there are 20 of them. Find the repeating block first, then extract fields inside it. Scraping all prices and all titles as 2 separate lists works until one item is missing a price. After that, every row is off by one.

Sometimes no container holds the whole record. Hacker News puts a story's title in one table row and its score and comment count in the next, as siblings, so the repeating block you find contains only half a record. Table layouts do this often, and so do dt and dd pairs. When it happens, anchor on the half that always exists, then step sideways to the rest with find_next_sibling(). On markup shaped like that story list, the pattern looks like this:

html = """<table>"""
  <tr class="athing"><td class="title"><a href="/item1">First story</a></td></tr>
  <tr><td class="subtext"><span class="score">120 points</span></td></tr>
</table>"""

row = BeautifulSoup(html, "lxml").select_one("tr.athing")  # the half that always exists
meta = row.find_next_sibling("tr")  # the half that might not
if meta and not meta.select_one("td.subtext"):
    meta = None  # that row is the next story, not this one's metadata

row.select_one("td.title a").get_text(strip=True)  # 'First story'
meta.select_one("span.score").get_text(strip=True) if meta else None  # '120 points'

The anchor row carries the title, so it's the one to loop over. That td.subtext check is the safeguard: find_next_sibling("tr") returns the next row whatever it holds, so on a story with no score it gives you the next story's title row, and reading a score out of that raises AttributeError on None. Confirm the sibling is the row you meant before you read it.

Find stable selectors

A selector tends to survive a redesign when it describes what an element is rather than where it currently sits or how it currently looks. Prefer these in order, most durable first:

1. Semantic containers: article, main, nav
2. IDs and data attributes: #product_description, [data-testid="price"]
3. Single meaningful classes: .price_color
4. Structural position, only as a last resort: div > div:nth-child(3)

Avoid long chains of generated class names like .css-1x7f2p9. Those can change whenever the site rebuilds its stylesheet. When plain CSS can't express what you need, the answer isn't XPath, which BeautifulSoup doesn't support at all. A find() with a small function does what a selector can't, because it runs your own Python on each tag: soup.find(lambda t: t.name == 'tr' and t.th and t.th.get_text(strip=True) == 'Availability') picks the row whose first cell reads Availability. :has() and the soupsieve extension :-soup-contains() reach the same row without leaving select().

If you do reach for structural position, check which soupsieve you have. BeautifulSoup delegates every select() call to that library, and version 2.9 fixed :nth-child(An+B) patterns that had been matching incorrectly. Running the same selectors against a list of 8 items on each version:

selector               soupsieve 2.8.4      soupsieve 2.9
li:nth-child(3)        ['3']                ['3']
li:nth-child(2n+1)     ['1','3','5','7']    ['1','3','5','7']
li:nth-child(2n-2)     []                   ['2','4','6','8']
li:nth-child(n-1)      []                   all eight

The old version didn't raise on those selectors but returned an empty list, which is the same signal you get from a selector that's simply wrong. Plain positions like :nth-child(3) were never affected. Formulas with a positive step were, though, whenever the list was short enough that the sequence landed on its last item. That comparison uses 8 items, which is why 2n+1 looks safe in it. On the old version, the same 2n+1 came back empty against a list of 1, and :nth-child(n+5) returned nothing against a list of 5. Check with python -c "import soupsieve; print(soupsieve.version)" before you lose hours debugging a selector that was right all along.

Watch for 2 selector traps:

First, copying a selector from your browser's dev tools often gives you table > tbody > tr. Browsers insert a element that isn't in the source HTML. Most parsers don't:

table = "<table><tr><td>r1</td></tr><tr><td>r2</td></tr></table>"
for parser in ["html.parser", "lxml", "html5lib"]:
    soup = BeautifulSoup(table, parser)
    copied = soup.select("table > tbody > tr")  # what DevTools hands you
    works = soup.select("table tr")  # what you should write
    print(f"{parser:<12} copied={len(copied)}  works={len(works)}")

The markup has 2 rows, so works=2 is the right answer everywhere:

html.parser  copied=0  works=2
lxml         copied=0  works=2
html5lib     copied=2  works=2

html5lib is the exception because it synthesises the a browser would add. Write table tr instead.

Second, class=_ matches the full class string exactly, in order:

btn = BeautifulSoup('<a class="btn btn-primary lg">Go</a>', "lxml")
btn.find_all(class_="btn btn-primary lg")   # 1 match
btn.find_all(class_="btn-primary btn lg")   # 0 matches, same classes reordered
btn.find_all(class_="btn")                  # 1 match, single class works
btn.select(".btn.btn-primary")              # 1 match, order independent

When you need to match several classes at once, CSS selectors are the safer choice.

Confirm content is not dynamic

Fetch your own target and count what your selector matches. If it returns 0, don't reach for a browser yet. Look at the script tags first, because many client-rendered pages ship their data as JSON right there in the HTML.

The browser shows 10 quotes. The HTML that requests receives contains none of them: the surrounding markup ends at line 26, and the whole dataset sits in a script tag just below it.

That var data is the whole dataset, already in JSON, sitting in the HTML you were about to give up on. So the fix is to find the script tag and parse it, not to start a browser:

import json, re
import requests
from bs4 import BeautifulSoup

response = requests.get("https://quotes.toscrape.com/js/", timeout=10)
soup = BeautifulSoup(response.content, "lxml")

match = None
for script in soup.find_all("script"):
    body = script.string or ""
    if "var data" in body:
        match = re.search(r"var data\s*=\s*(\[.*?\]);", body, re.S)
        break

# Check before you chain. Without this, a page that changed shape raises
# AttributeError on None rather than telling you the block is gone.
if match is None:
    raise SystemExit("no 'var data' block on this page")

quotes = json.loads(match.group(1))
print(f"parsed {len(quotes)} quotes straight out of the script tag")
for quote in quotes[:3]:
    print(f"  {quote['author']['name']:<20} {quote['text'][:52]}...")

That's 10 quotes, none of which were ever in an HTML element:

parsed 10 quotes straight out of the script tag
  Albert Einstein      “The world as we have created it is a process of our...
  J.K. Rowling         “It is our choices, Harry, that show what we truly a...
  Albert Einstein      “There are only two ways to live your life. One is a...

The first place to look is the markup itself: , NEXT_DATA on older Next.js sites, or any var data = blob. The second is the Network tab: filter by Fetch/XHR, reload, and see whether the page calls a JSON endpoint you could call directly.

Reach for a headless browser only after both of those checks find nothing.

Identify list pages vs detail pages

Many sites split into 2 page types, and the difference determines how you structure the crawl:

• List pages carry many items with a few fields each, plus the links you need. Cheap to scrape.
• Detail pages carry one item with every field. One request each, so expensive.

Work out the ratio before you commit. The 3 fields a detail page adds here are category, an exact stock count and a description, so visit one only when you genuinely need one of those, not by default. Splitting the crawl into 2 stages also isolates failures: one bad detail page doesn't ruin the whole category.

Further reading: AI Web Scraping with Python: How to Scrape Data with AI from a Website in 2026 and How to Scrape Dynamic Content from a Website in 2026.

Web Scraping Using BeautifulSoup and Requests

The fetch is where much of the silent corruption starts: a missing timeout, an unchecked status, or the wrong decoding.

Timeouts and status checks

Every request you write needs a timeout and a status check, and neither is on by default:

response = requests.get(url, timeout=10)
response.raise_for_status()

requests has no default timeout. Without one, a stalled connection hangs your script indefinitely. Always pass it.

raise_for_status() turns 4xx and 5xx into exceptions. Use it, but know its limit: it only checks the status line. A 200 response can still be a login wall, a cookie banner or a CAPTCHA page.

Headers that prevent breakage

Many servers behave differently when the client doesn't look like a browser. Sending a realistic User-Agent string is the usual starting point, though a complete and internally consistent header set matters more than the User-Agent alone. Build a header set that promises only what you can deliver:

HEADERS = {
    # Copy this from your own browser (chrome://version) rather than reusing
    # the string printed here. A User-Agent that stopped moving years ago is
    # itself a signal, which is the opposite of what a header set is for.
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate",
}

Don't copy Chrome's full Accept-Encoding value. Chrome advertises br and zstd. If you claim those without installing the decoders, you can get a 200 with a body you can't read, and no exception:

for enc in ["gzip, deflate", "gzip, deflate, br, zstd"]:
    r = requests.get(URL, headers={"Accept-Encoding": enc}, timeout=15)
    print(f"{enc:<24} -> {r.status_code} {r.headers.get('content-encoding')} "
          f"parseable={'In stock' in r.text}")

Both requests succeed. Only one of them gives you something you can read:

gzip, deflate            -> 200 None parseable=True
gzip, deflate, br, zstd  -> 200 br parseable=False

Install brotli if you want br, or ask only for what you can decode. On Python 3.14 zstd already decodes through the standard library. On earlier versions install urllib3[zstd], not the similarly named zstandard package, which urllib3 doesn't use.

Header order is the one part you can't set from the headers= dict at all. A Session adds its own defaults first:

s = requests.Session()
custom = {"sec-ch-ua": "x", "User-Agent": "x", "Accept": "x"}
req = s.prepare_request(requests.Request("GET", URL, headers=custom))
print(list(req.headers.keys()))

The order you set isn't the order that goes out:

['User-Agent', 'Accept-Encoding', 'Accept', 'Connection', 'sec-ch-ua']

That order is one of the signals that anti-bot systems look at. You can control it by replacing session.headers with your own dictionary and inserting the keys in the order you want, but anything you add after that still lands at the end.

Sessions and cookies

A Session reuses the underlying TCP connection and keeps cookies across requests. On 8 sequential requests to one host:

requests.get() each time : 12.61s  (1576 ms/req)
one Session reused       : 3.71s   (464 ms/req)
saved                    : 71%

Most of that saving is the connection you're no longer rebuilding, the TCP and TLS handshakes together. Sessions also keep the cookies that the site sets, which matters for locale, currency and pagination state. Open one, set your headers on it once, and reuse it for everything that follows:

with requests.Session() as session:
    session.headers.update(HEADERS)
    response = session.get(url, timeout=10)

Setting the headers on the session rather than on each call means every request through that session carries them, including the requests you add later.

Retry with backoff

requests mounts zero retries by default:

print(requests.Session().get_adapter("https://example.com").max_retries)

That's what every requests session comes with:

Retry(total=0, connect=None, read=False, redirect=None, status=None)

urllib3 provides a Retry class you can mount on an adapter, and it honours Retry-After once you configure it. But its defaults leave 2 gaps: the backoff factor is 0, so the attempts run with no wait between them, and the status list is empty, so a 503 isn't retried at all. The loop below is written by hand instead, because a scraper needs to inspect the body for a block that arrives as a 200, which a transport-level retry never sees.

Retry transient failures only. A 500 or 503 is worth another try, but a 404 isn't. Wait longer after each attempt, and add jitter so parallel workers don't retry at the same time:

import random, time
import requests

RETRY_STATUSES = {429, 500, 502, 503, 504}

def fetch(session, url, *, attempts=4, timeout=10):
    for attempt in range(1, attempts + 1):
        try:
            response = session.get(url, headers=HEADERS, timeout=timeout)
        except requests.RequestException as exc:
            if attempt == attempts:
                raise
            delay = 2**attempt + random.uniform(0, 1)
            print(
                f"  [{attempt}/{attempts}] {type(exc).__name__}, sleeping {delay:.1f}s"
            )
            time.sleep(delay)
            continue

        if response.status_code in RETRY_STATUSES and attempt < attempts:
            # Honour a numeric Retry-After, back off otherwise. The HTTP-date form
            # is not parsed here, so it falls through to the exponential delay.
            retry_after = response.headers.get("Retry-After")
            # Cap it. Retry-After is the remote server's number, and an unbounded
            # one parks a scheduled run for as long as that server likes.
            delay = (
                min(float(retry_after), 300)
                if (retry_after or "").isdigit()
                else 2**attempt
            )
            delay += random.uniform(0, 1)
            print(
                f"  [{attempt}/{attempts}] HTTP {response.status_code}, sleeping {delay:.1f}s"
            )
            time.sleep(delay)
            continue

        response.raise_for_status()
        return response

    raise RuntimeError(f"gave up on {url} after {attempts} attempts")

The prints are there so you can watch the backoff grow. Running it against an endpoint that always returns 503, with attempts=3:

  [1/3] HTTP 503, sleeping 2.9s
  [2/3] HTTP 503, sleeping 4.4s
   raised after retries: 503 Server Error: Service Temporarily Unavailable for url: https://httpbin.org/status/503

From here on, the crawler and the detail scraper both fetch through this helper, rather than repeating the backoff in each loop.

How to Extract Data with BeautifulSoup

BeautifulSoup gives you 2 search APIs, and neither is deprecated in favour of the other. The split that matters is narrow: CSS selectors read better on nested structure, and find_all accepts class names that the selector parser rejects.

Basics of find and find_all

find() returns the first match or None. find_all() returns a list, empty if nothing matched:

soup.find("h3")                      # first h3, or None
soup.find_all("article")             # list of every article
soup.find_all("a", limit=5)          # first 5 links
soup.find_all("p", class_="price_color")
soup.find_all("a", href=True)        # only anchors that have an href

The failure modes are different, and that asymmetry causes real bugs:

soup.find_all("nope")            # []   loops zero times, no error, silent
soup.find("nope")                # None
soup.find("nope").get_text()     # AttributeError: 'NoneType' object has no attribute 'get_text'

The empty list is the worse failure, because you don't find out until someone asks why the file is empty.

That difference in timing is why the empty list needs an explicit check.

Avoid 2 arguments here. text= is deprecated in favour of string=, and limit=0 doesn't mean "no results", since on 4.15.0 it's ignored and returns everything.

CSS selectors with select and select_one

select() takes a CSS selector and returns a list. select_one() returns the first match or None:

soup.select("article.product_pod")        # all cards
soup.select_one("h3 a")                   # first link inside an h3
soup.select("li.next a")                  # next-page link
soup.select("article:has(img)")           # cards containing an image
soup.select("p:-soup-contains('In stock')")

Nested structure reads better as one selector than as 3 chained find() calls, and :has(), :is(), :where() and :not() with lists all work here.

Some CSS that works in a browser behaves differently here:

• Use :-soup-contains(), not :contains(). The old spelling still parses, but it raises a FutureWarning telling you that :contains is deprecated and to move to the new one.
• :hover, :target and :focus-visible parse fine and always return [], because there's no browser here and nothing is ever hovered. States that live in the markup, like :checked and :disabled, do work.
• Pseudo-elements such as ::before raise NotImplementedError. There's no styling to read.

Where select() fails. Many Tailwind-style class names aren't valid CSS identifiers, so the selector parser rejects them while find_all handles them without complaint:

tw = BeautifulSoup('<div class="w-1/2">a</div><div class="hover:bg-blue-600">b</div>', "lxml")
tw.select(".w-1/2")                    # SelectorSyntaxError
tw.select(".hover:bg-blue-600")        # SelectorSyntaxError
tw.find_all(class_="w-1/2")            # 1 match
tw.find_all(class_="hover:bg-blue-600")# 1 match

If you must use a selector, escape it first with soup.css.escape("2xl:flex"), which returns '\32 xl\:flex'. That helper comes with BeautifulSoup, so it needs no extra import. select() is often treated as the strictly more modern of the two, but on a site built with Tailwind-style utility classes it's the one that breaks, so find_all(class=)_ is the easier path there.

Extract text cleanly

.get_text() collects the strings inside an element, however deep they sit. .text is an exact alias, no relation to response.text and its encoding trap, so use whichever you prefer. More usefully, get_text() accepts arguments:

snippet = BeautifulSoup("<div><p>Price</p><p>51.77</p></div>", "lxml")
snippet.get_text()                  # 'Price51.77'
snippet.get_text(separator=" ")     # 'Price 51.77'

Without a separator, adjacent block elements run together into one unusable string.

strip=True removes surrounding whitespace from each string. For finer control, .stripped_strings yields each non-empty string separately:

list(snippet.stripped_strings)      # ['Price', '51.77']

A note on a pattern you'll see in many tutorials: soup(["script", "style"]) followed by decompose() before calling get_text(). On lxml and html.parser that hasn't been necessary since BeautifulSoup 4.9.0, because get_text() already skips script and style content:

noisy = BeautifulSoup("<div>Real<script>var x=1;</script></div>", "lxml")
noisy.get_text()        # 'Real'

# the same markup under html5lib, which does not get this protection:
BeautifulSoup("<div>Real<script>var x=1;</script></div>", "html5lib").get_text()
# 'Realvar x=1;'

That protection comes from storing the contents as Script and Stylesheet objects, which the html5lib tree builder doesn't do. It also still matters if you walk the tree yourself.

Extract attributes like links and images

Text is often the wrong field. Look at one card from the target site:

<p class="star-rating Three">...</p>
<h3><a href="catalogue/a-light-in-the-attic_1000/index.html"
       title="A Light in the Attic">A Light in the ...</a></h3>

That small block of markup already contradicts 3 assumptions that people bring to a first scrape:

1. The rating is a word in the class attribute, not text anywhere on the page.
2. The visible title can be truncated. get_text() gives you "A Light in the ..." here, and half the cards on page 1 are cut the same way. The title attribute has the real one.
3. The href is relative, so it's useless on its own.

Reading those 3 from the card looks like this:

link = card.select_one("h3 a")
link["title"]                    # 'A Light in the Attic'
link.get_text(strip=True)        # 'A Light in the ...'   <- truncated
link["href"]                     # 'catalogue/a-light-in-the-attic_1000/index.html'
card.select_one("p.star-rating")["class"]   # ['star-rating', 'Three']

Always resolve links with urljoin against the page you found them on:

from urllib.parse import urljoin
urljoin("https://books.toscrape.com/", link["href"])
# https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html

This isn't cosmetic. On this site, page 1 links look like catalogue/the-book_1/index.html while page 2 links look like the-book_2/index.html, because the pages sit at different depths. Hardcoding a base URL works on page 1 and produces 404s from page 2 onward. urljoin handles both.

Handle missing elements safely

One card missing one field shouldn't kill the run. Use .get() for attributes and check before chaining:

def safe_field(card, selector, attr=None, default=None):
    node = card.select_one(selector)
    if node is None:
        return default
    return node.get(attr, default) if attr else node.get_text(strip=True)

# The same pattern on 3 fields. In parse_card, wrap the reads this way
# rather than replacing the keys it already returns:
row = {
    "title": safe_field(card, "h3 a", attr="title"),
    "price": safe_field(card, "p.price_color"),
    "image": safe_field(card, "img", attr="src"),
}

Store None rather than skipping the row. A field you recorded as missing shows up in the missing rate, and a jump in that rate is how you detect a broken selector. A row you never wrote is invisible, so a missing-rate check has nothing to count.

How to Scrape Multiple Pages with BeautifulSoup Pagination

Just 3 patterns cover most pagination you'll meet: query parameters (?page=2), path segments (/page/2), and a "next" link you follow. Following the next link is the one that doesn't need you to guess how many pages exist.

Build page URLs safely

Before you write any pagination loop, check /sitemap.xml and the Sitemap: line in robots.txt. Plenty of sites publish every URL you want in one file, which can turn a fragile multi-page crawl into one fetch and a list of links.

When there's no sitemap and the URLs are predictable, generate them, but always cap the range:

urls = [f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 51)]

Record the list URL on every row, which parse_card already does as source_url. When something looks wrong later, that column is what tells you which page it came from.

Find and follow next links

When the URLs aren't predictable, let the site tell you where to go next by reading its own next link:

next_link = soup.select_one("li.next a")
if next_link:
    url = urljoin(url, next_link["href"])

When the last page is reached, the site drops the next element, so select_one returns None and the if never fires. Without that check, reading next_link["href"] would raise a TypeError instead.

Deduplicate items across pages

Dedupe on something stable, like the item URL or an ID, rather than on a title, because 2 different books can share a name:

seen, unique = set(), []
for row in rows:
    if row["url"] in seen:
        continue
    seen.add(row["url"])
    unique.append(row)

Dedupe as you go rather than at the end, so a page that repeats itself doesn't quietly double your row count first.

Stop rules that prevent runaway crawls

A crawler with no stop condition runs until something else stops it, and that something is often the site you're crawling. Set all of the following, because they fail independently and any one of them can be the only thing that catches a bug:

1. Max pages. A hard ceiling, so a bug can't become a thousand requests.
2. No new items. If a page only returns items you already have, you're probably looping.
3. No next link. The natural end.

Here's the crawler with all 3:

# Continues the scraper, reusing parse_card and the fetch() retry helper.
START = "https://books.toscrape.com/"

def crawl(start=START, max_pages=3):
    rows, seen = [], set()
    url, page = start, 0

    with requests.Session() as session:
        while url and page < max_pages:  # stop rule 1
            page += 1
            # fetch() carries the retries; a transient 503 no longer aborts the crawl.
            soup = BeautifulSoup(fetch(session, url).content, "lxml")

            cards = soup.select("article.product_pod")
            if not cards:
                # 0 cards is a block or a renamed class, not an empty page.
                # Without this, stop rule 2 reads it as a clean finish.
                raise RuntimeError(
                    f"no cards on {url}, title={soup.title and soup.title.get_text(strip=True)!r}"
                )

            new = 0
            for card in cards:
                row = parse_card(card, url)
                if row["url"] in seen:  # stable key, not the title
                    continue
                seen.add(row["url"])
                rows.append(row)
                new += 1

            print(f"  page {page}: {len(rows):>3} total, {new:>2} new")

            if new == 0:  # stop rule 2
                break

            next_link = soup.select_one("li.next a")
            if not next_link:  # stop rule 3
                break
            url = urljoin(url, next_link["href"])

            time.sleep(
                1 + random.uniform(0, 0.5)
            )  # a fixed delay is a machine-perfect rhythm
    return rows

rows = crawl()

It stops after the third page because max_pages says so, not because the site ran out of pages:

  page 1:  20 total, 20 new
  page 2:  40 total, 20 new
  page 3:  60 total, 20 new

One thing this version doesn't do is survive its own crash. It holds every row in a list and only writes at the end, so a network error on page 40,000 loses the 39,999 pages before it.

On a long crawl, write each page's rows straight to disk inside the loop instead of holding them in a list: open the CSV in append mode and writerows after each page. Call writeheader only when the file is new, or a rerun writes a second header row into the middle of your data. A .jsonl file avoids that entirely, because one json.dumps(row, default=str) line per record has no header to repeat, which is why it's the easier format to append to and resume. The default=str is not optional, since json refuses to serialise the Decimal that parse_price returns.

Persist which pages you've finished too, so a rerun skips them instead of starting over. The in-memory dedupe set only prevents duplicates within one run, so a set that survives a restart has to live on disk.

How to Build a Python Web Scraper Using BeautifulSoup for Item Detail Pages

A 2-stage crawl costs 21 requests where the list page alone costs 1, and buys 3 fields the list page never carries.

List page to detail page flow

Stage 1 collects the links off the list page. Stage 2 visits each one in turn, reusing the same session so the connection isn't rebuilt every time.

The loop stays readable with 2 helpers. get_soup wraps the fetch and parse pair you've written several times already, and routes it through the fetch() retry helper so every detail request inherits the backoff rather than aborting on the first transient error:

def get_soup(session, url):
    return BeautifulSoup(fetch(session, url).content, "lxml")

Parse key value specs

Spec tables are often th/td pairs. Normalise the keys so downstream code isn't dealing with "Price (excl. tax)":

# soup here is one product's detail page: soup = get_soup(session, product_url)
specs = {}
for row in soup.select("table.table-striped tr"):
    if row.th is None or row.td is None:     # header and spacer rows have neither
        continue
    # \w keeps non-ASCII letters, so a Cyrillic or Greek label still gets a key.
    # [^a-z0-9] would delete the whole label and collapse every row onto "".
    key = re.sub(r"[^\w]+", "_", row.th.get_text(strip=True).lower()).strip("_")
    specs[key] = row.td.get_text(strip=True)

specs then holds the whole table, keyed for downstream code:

{'upc': 'a897fe39b1053632',
 'product_type': 'Books',
 'price_excl_tax': '£51.77',
 'price_incl_tax': '£51.77',
 'tax': '£0.00',
 'availability': 'In stock (22 available)',
 'number_of_reviews': '0'}

pandas.read_html is the obvious alternative, and it's the right tool when a page carries a wide data table you want as a DataFrame anyway. It's less useful here. A 2-column table of keys and values comes back with unnamed columns, so you still combine the 2 halves and still normalise the keys. And reading it from response.text rather than io.BytesIO(response.content) reproduces the same mojibake shown earlier.

Whichever route you take, keep the raw string alongside anything you parse from it. When availability changes format, having the original means you can re-parse without re-scraping:

match = re.search(r"\((\d+) available\)", specs.get("availability", ""))
units = int(match.group(1)) if match else 0

The list page only says "In stock", while the detail page says how many.

Clean product descriptions

The description on this site sits after the #product_description heading, not inside it:

heading = soup.select_one("#product_description")
description = ""
if heading:
    para = heading.find_next_sibling("p")
    if para:
        description = re.sub(r"\s+", " ", para.get_text(strip=True))

find_next_sibling() is the tool for "the thing after this thing". Then collapse runs of whitespace with \s+, and keep the raw HTML alongside it when the structure has downstream value.

That's every piece of parse_detail except 2. Here it's assembled, with the title read from the page's h1 and the category from the last link in the breadcrumb, the only place this site names it:

def parse_detail(soup, url, source_list_url):
    specs = {}
    for row in soup.select("table.table-striped tr"):
        if row.th is None or row.td is None:
            continue
        key = re.sub(r"[^\w]+", "_", row.th.get_text(strip=True).lower()).strip("_")
        specs[key] = row.td.get_text(strip=True)

    available = re.search(r"\((\d+) available\)", specs.get("availability", ""))
    heading = soup.select_one("#product_description")
    para = heading.find_next_sibling("p") if heading else None

    return {
        "title": soup.select_one("h1").get_text(strip=True),
        "category": [a.get_text(strip=True) for a in soup.select("ul.breadcrumb a")][-1],
        "price": specs.get("price_incl_tax", ""),      # still a string until parse_price replaces this
        "units_available": int(available.group(1)) if available else None,
        "description": re.sub(r"\s+", " ", para.get_text(strip=True)) if para else "",
        "url": url,
        "source_list_url": source_list_url,
    }

Turning that raw "£51.77" string into something you can sort and sum is its own problem.

With parse_detail defined, the driver is short. The [:4] keeps the demo to 4 requests:

with requests.Session() as session:
    soup = get_soup(session, START)
    urls = [urljoin(START, a["href"])
            for a in soup.select("article.product_pod h3 a")][:4]
    print(f"stage 1: collected {len(urls)} product URLs from {START}\n")

    print("stage 2: visiting each product page")
    for url in urls:
        try:
            detail = parse_detail(get_soup(session, url), url, source_list_url=START)
        # parse_detail raises more than network errors: LookupError on a missing
        # breadcrumb, AttributeError on a missing h1.
        except (requests.RequestException, LookupError, AttributeError) as exc:
            print(f"  skipped {url}: {type(exc).__name__}: {exc}")   # one bad page must not kill the batch
            continue
        print(f"  {detail['title'][:32]:<33}{detail['category']:<22}"
              f"{detail['price']:<8}{detail['units_available']} left  "
              f"{detail['description']} chars")
        time.sleep(1 + random.uniform(0, 0.5))

Over the first 4 products that prints:

stage 1: collected 4 product URLs from https://books.toscrape.com/

stage 2: visiting each product page
  A Light in the Attic             Poetry                £51.77  22 left  1017 chars
  Tipping the Velvet               Historical Fiction    £53.74  20 left  1029 chars
  Soumission                       Fiction               £50.10  20 left  1093 chars
  Sharp Objects                    Mystery               £47.82  20 left  1635 chars

Category, unit count and description length are all fields that the list page never carried.

Handle variants and multiple offers

Variants are often separate URLs rather than options inside one page. The tutorial site can't demonstrate that, so the example is a real commercial page. We fetched IKEA's BILLY bookcase on their UK site, checking robots.txt first, which allows /p/ and disallows /cart/.

The page is worth more than any invented example for 2 reasons. The first is that it publishes its own structured data, so you don't have to guess at selectors:

import json

# soup here is a product page you've fetched, in this case IKEA's BILLY bookcase.
for tag in soup.find_all("script", type="application/ld+json"):
    try:
        data = json.loads(tag.string or "")
    except json.JSONDecodeError:
        continue          # one malformed block must not hide a valid one below it
    # A page can carry several of these blocks, and a block can be a list rather
    # than an object, so neither "the first one" nor "it is a dict" is safe.
    for node in (data if isinstance(data, list) else [data]):
        if isinstance(node, dict) and node.get("@type") == "Product":
            # offers is a list whenever a product has several sellers or sizes.
            # Taking the first is a choice, not a fix: it drops the rest
            # without saying so. Loop over the list when you need every offer.
            offer = node.get("offers")
            offer = offer[0] if isinstance(offer, list) else (offer or {})
            print(node.get("sku"), node.get("color"),
                  offer.get("price"), offer.get("priceCurrency"))

That page carries 2 blocks, a BreadcrumbList and a Product, and only the second one matches:

002.638.50 white 55 GBP

Look for application/ld+json before you write a single CSS selector. Plenty of commercial pages carry a Product block with SKU, price and currency already parsed, and it tends to survive redesigns that break selectors. The 2 guards matter: if you drop the list check, a site whose block is a JSON array gives you AttributeError: 'list' object has no attribute 'get'. At multi-site scale, extruct reads JSON-LD, microdata, RDFa and OpenGraph in one call, with those shapes already handled.

The second is how the variants are actually modelled. The colours aren't options inside one page. Each is a separate URL with its own SKU, linked from the others:

colour         sku           price   currency   size
white          002.638.50    55      GBP        80 x 28 x 202 cm
black          404.773.40    70      GBP        80 x 28 x 202 cm
brown          505.086.52    70      GBP        80 x 28 x 202 cm
beige, brown   105.089.32    70      GBP        80 x 28 x 202 cm

Identical dimensions, 4 SKUs, and on that day the white one was £15 cheaper than its siblings. If you scrape the URL you happened to open, you get "the price of a BILLY bookcase" when there isn't one, just the price of whichever colour you landed on. So the variant problem is often a crawling problem before it's a parsing one: find the sibling links, follow them, and key your rows on SKU rather than on product name.

Real attribute values aren't tidy enumerations either, as beige, brown in that colour column shows. Anything downstream that expects one word per colour has a problem on day one.

Keep variants as their own rows or a separate table keyed by SKU. Flattening to one row per product with a single price column can't answer "which colours were out of stock", and you can't reconstruct that information afterwards.

How to Scrape Only Visible Webpage Text with BeautifulSoup

For article scraping and summarisation you want readable text rather than navigation and footers, and you want it to keep some shape. The same navigation and footers on every page become tokens you pay for over and over, and a single flat block of text loses the structure that makes a long document usable later. The examples run against one product page, so fetch it into soup first:

DETAIL = "https://books.toscrape.com/catalogue/sharp-objects_997/index.html"
soup = BeautifulSoup(requests.get(DETAIL, headers=HEADERS, timeout=10).content, "lxml")

Asking for all of its text at once gives you everything on the page, navigation included:

1. soup.get_text(separator="\n", strip=True) on the whole document
   2,291 chars over 42 lines
   first 5 lines: ['Sharp Objects | Books to Scrape - Sandbox', 'Books to Scrape',
                   'We love being scraped!', 'Home', 'Books']

The first 5 lines are the page title, the site name, a banner and 2 breadcrumbs. None of it's the content.

Remove noise tags

On lxml and html.parser, get_text() already skips the text inside script and style, but it leaves elements like nav and footer in the tree, and you rarely want them in extracted text:

for tag in soup(["noscript", "template", "svg", "nav", "footer"]):
    tag.decompose()

decompose() destroys the element. extract() removes it and returns it if you want it later.

On this product page that loop changes nothing, because the only tag from the list it carries is an empty . The loop earns its place on real sites, where a nav and a footer repeat on every page you fetch.

Target the main content container

Scoping to the content container is the change that matters. It removes only 7 lines, but it changes which lines you keep:

main = soup.select_one("article.product_page") or soup.body
clean = re.sub(r"\n{2,}", "\n", main.get_text(separator="\n", strip=True))

Against that same product page, both changes give:

2. Noise tags removed, scoped to article.product_page
   2,177 chars over 35 lines
   first 5 lines: ['Sharp Objects', '£47.82', 'In stock (20 available)', 'Warning!',
                   'This is a demo website for web scraping purposes...']
   7 lines fewer than step 1

Look for , , [role="main"] or a known content id, and fall back to body. The or soup.body matters, because select_one returns None on sites that don't use the tag you expected.

Preserve structure for readability

Splitting on headings keeps the shape:

from bs4 import Comment, Script, Stylesheet

HEADINGS = {"h1", "h2", "h3"}
SKIP_STRINGS = (Comment, Script, Stylesheet)

def to_sections(container):
    sections, current = [], {"heading": "(intro)", "parts": []}
    for node in container.descendants:
        if getattr(node, "name", None) in HEADINGS:
            if current["parts"]:
                sections.append(current)
            current = {"heading": node.get_text(strip=True), "parts": []}
        # These 3 all subclass str, so exclude them explicitly.
        elif isinstance(node, str) and not isinstance(node, SKIP_STRINGS):
            if node.parent.name not in HEADINGS and node.strip():
                current["parts"].append(node.strip())
    if current["parts"]:
        sections.append(current)
    return sections

# main is the scoped container: select_one("article.product_page") or soup.body
for section in to_sections(main):
    print(f"   {section['heading']:<44}{len(' '.join(section['parts'])):>4} chars")

Each section's heading and the length of its text, on that same page:

3. Structured sections rather than one string
   Sharp Objects                                162 chars
   Product Description                         1635 chars
   Product Information                          156 chars
   Soumission                                    40 chars
   Tipping the Velvet                            48 chars
   A Light in the ...                            48 chars

The last 3 aren't part of this book. They're the "you might also like" carousel, and they came through because article.product_page contains it. Scoping to a container narrows the problem but doesn't solve it, so check what your container actually holds before you trust the output.

That filter isn't optional. Comment, Script and Stylesheet all inherit from str in BeautifulSoup, so a naive isinstance(node, str) test silently swallows every HTML comment it meets along with the entire contents of every script and style tag. That product page carries 17 comments, including old Internet Explorer conditionals, and 5 of those sit inside article.product_page. Without the check, those 5 land in your extracted text, and a whole-document walk picks up all 17 plus the contents of every script and style tag. On lxml and html.parser, get_text() screens out all 3 for you, which is exactly why traversal that you write by hand has to exclude them explicitly.

Knowing how to do this by hand doesn't mean doing it by hand every time. For article-shaped pages, trafilatura is a widely used open source extractor, and the tool behind LLM training datasets such as FineWeb. Reach for it when the goal is clean text at scale, and keep the hand-written traversal when you need the text split into sections rather than one block. Neither one saves you from checking the output, since on this product page trafilatura pulls in the carousel too.

How to Clean and Validate a BeautifulSoup Dataset

Validation catches the structural failures reliably and the semantic ones badly. A field that's missing is easy to spot. A number that's wrong but looks ordinary is not.

Generated selectors raise the risk rather than lowering it. A model writes a working parser against one sample page in seconds, and it has no way to tell you that the selector it picked breaks on the second page. Generated extraction code gets safer to run unattended with 3 things: fixtures to test against, missing rates that expose silent breakage, and a sample that a person actually reads.

Validation checklist

Run these 7 checks on every dataset before anything downstream consumes it:

• Required fields present: confirm title, price and URL are non-empty, because a row without them still counts in your row total while being useless to whatever reads the file.
• URLs absolute and well formed: relative links break the moment the data leaves your script, so resolve them at extraction time.
• Numbers parsed as numbers: a price stored as "£51.77" sorts and sums incorrectly, and the error surfaces weeks later in a report.
• Duplicates removed on a stable key: dedupe on the item URL or ID, since 2 different products legitimately share a title.
• Values inside plausible ranges: a price of 0 or 999999 signals a parsing failure rather than a bargain.
• One currency per price column: if currency holds more than one value, the column isn't comparable and min, max and any average over it are meaningless.
• A sample read by a human: print 3 random rows every run and actually read them, because none of the other 6 checks catches text that's subtly the wrong field.

That currency check is worth keeping because a range check can't do its job for you. Range checks only catch parsing failures that land somewhere absurd, and the failures that cost you money usually don't.

Normalize text and numbers

Text is the simpler half:

title = re.sub(r"\s+", " ", raw_title).strip()
# Use explicit month numbers like this, not "%d %B %Y". A named month reads
# through LC_TIME. Python starts in the C locale, so English names parse until
# something calls setlocale, and then the same code raises.
date = datetime.strptime(raw_date, "%d %m %Y").date()

Numbers are where scrapers quietly give you wrong values. Many tutorials clean a price like this:

price = float(re.sub(r"[^\d.]", "", raw))  # "£51.77" -> 51.77

That works on the sandbox because every price on it looks like £51.77. Here's the same line against prices from a live German price comparison page:

page string     that regex      correct
€ 40,96         4,096.00        40.96
€ 32,99         3,299.00        32.99
€ 357,76        35,776.00       357.76
€ 618,88        61,888.00       618.88

Every one is wrong by a factor of 100, and not one of them raises. The regex deletes the comma as though it were a thousands separator. Run it on 1.299,00 € and the result is 1.299, wrong by a factor of 1,000 in the other direction. A range like £51.77 - £64.99 or a sale price sitting next to its strikethrough gives you 2 numbers glued together or a ValueError, depending on the page.

Worse, that wrong number then walks through validation untouched. Take € 357,76, mangled into 35776.0, and run it past the 7 checks. The field is present. The URL is fine. It's genuinely a number. It isn't a duplicate. And 35,776 passes a range check, because a site listing everything from cables to appliances needs a range wide enough to let it through. Even the currency check passes, because every row came from the same German site so the column holds one currency throughout. Only the last check, a human actually reading that row, has any chance of catching it. So 6 of the 7 checks pass on a value that's 100 times wrong.

That failure is the whole argument for a parser that raises rather than guesses, and for carrying the currency next to the number. In practice that means working out what each separator is doing instead of deleting it. The function below replaces the one-character split that parse_card used, as soon as your prices stop looking like £51.77:

import re
from decimal import Decimal

SPACES = " \xa0\u202f\u2009"                  # space, nbsp, narrow nbsp, thin space
NUMBER = re.compile(rf"\d[\d.,{SPACES}]*\d|\d")
SYMBOLS = {"£": "GBP", "$": "USD", "€": "EUR", "¥": "JPY", "₹": "INR"}
ZERO_DECIMAL = {"JPY", "KRW", "ISK"}             # currencies with no minor unit

class PriceError(ValueError):
    pass

def parse_price(text, decimal_hint=None):
    """Return (Decimal, currency) or raise. Never guess."""
    text = text or ""
    numbers = NUMBER.findall(text)
    if len(numbers) != 1:                        # a range, or a sale price pair
        raise PriceError(f"{len(numbers)} numbers in {text!r}, expected 1")

    body = numbers[0].strip(SPACES)
    currency = next((c for s, c in SYMBOLS.items() if s in text), None)
    # A leading minus or accounting parentheses make it negative. Without this the
    # sign is silently dropped and a refund of -5.00 is stored as income of 5.00.
    start = text.find(numbers[0])
    negative = bool(re.search(r"[-−]\s*[£$€¥₹]?\s*$", text[:start])) or (
        "(" in text[:start] and ")" in text[start + len(numbers[0]):])
    dot, comma = body.rfind("."), body.rfind(",")

    if dot > -1 and comma > -1:
        sep = "." if dot > comma else ","         # the later one is the decimal
    elif dot > -1 or comma > -1:
        found = "." if dot > -1 else ","
        tail = len(body) - max(dot, comma) - 1
        if tail == 3 and body.count(found) == 1:
            # "1.299" is 1299 in Berlin and 1.299 in Boston. Genuinely ambiguous.
            if currency in ZERO_DECIMAL:
                sep = "," if found == "." else "."
            elif decimal_hint is None:
                raise PriceError(f"{body!r} is ambiguous, pass decimal_hint")
            else:
                sep = decimal_hint
        elif body.count(found) > 1:
            sep = "," if found == "." else "."     # repeated, so it groups
        else:
            sep = found
    else:
        sep = "."

    group = "," if sep == "." else "."
    cleaned = body.replace(group, "").translate({ord(c): None for c in SPACES})
    value = Decimal(cleaned.replace(sep, "."))
    return (-value if negative else value), currency

Running both the old line and the new function over the same set of strings:

page string           tutorial regex    parse_price      currency
'£51.77'              51.77             51.77            GBP
'$1,299.00'           1,299.00          1299.00          USD
'€ 357,76'            35,776.00         357.76           EUR
'1.299,00 €'          1.299             1299.00          EUR
'1 299,00 €'          129,900.00        1299.00          EUR
'¥12,800'             12,800.00         12800            JPY
'$1,299.00 $1,499.00' ValueError        PriceError       2 numbers, expected 1
'Price not available' ValueError        PriceError       0 numbers, expected 1

The last column is trimmed to fit, and PriceError also quotes the string that failed, so the real message reads 2 numbers in '$1,299.00 $1,499.00', expected 1, which is the part you want in a log.

The code makes 3 decisions that matter more than the regex. It returns the currency it found, so the unit isn't thrown away along with the symbol. It returns a Decimal rather than a float, because money in binary floating point is its own kind of bug. And when a string is genuinely ambiguous, it raises instead of picking. A bare 1.299 is one thousand two hundred and ninety nine in Germany and just over one in the United States.

Now wire it into the script you've been building, because a parser only helps once something calls it. In parse_card the 2 lines that split the symbol off the front become a single call, and parse_detail stops storing a string:

# Move parse_price and PriceError above parse_card in your file. The
# rows = crawl() line you already have runs the moment Python reaches it,
# and by then parse_card needs both of them.
# Add PriceError to the detail driver's except tuple too, now that it exists.

# In parse_card, this pair:
#     symbol, amount = price_text[0], price_text[1:]
#     price, currency = Decimal(amount), CURRENCY.get(symbol, symbol)
# becomes a call that can raise. Catch it where the field is read, so one
# bad price costs you a value rather than the whole page:
try:
    price, currency = parse_price(price_text)
except PriceError as exc:
    print(f"  unparsed price {price_text!r} on {source_url}: {exc}")
    price, currency = None, None

# In parse_detail, the raw "£51.77" string becomes a number, and the row
# it returns gains a currency key of its own:
price, currency = parse_price(specs.get("price_incl_tax", ""))

parse_price raises on a price it can't read rather than writing a wrong number, which is what the currency and range checks assume. parse_card catches that and stores None, so the row still reaches the missing-rate check instead of vanishing from the file. A field you recorded as missing is countable. A row you never wrote is not. A price with no minor units lands there too: 4,000 is 4000 in London and 4.0 in Berlin, so parse_price refuses it until you pass decimal_hint="." for a site that groups with commas. The CURRENCY map is no longer needed, because parse_price does the currency lookup itself, but keep the Decimal import, since it returns one.

Keep raw and cleaned values side by side while you're still developing. When a price fails to parse, you want to see the string that broke it.

Track missing rates

Missing rates are your early-warning system. If a field was 0% missing yesterday and is 100% missing today, the selector broke or you're being served a different page. Computing them is a few lines on top of the 7 checks:

import pandas as pd

def validate(rows):
    df = pd.DataFrame(rows)
    prices = df["price"].dropna()          # unparsed prices are None, not dropped rows
    report = {
        # Count "" as missing too, or a selector that returns an empty string
        # reports 0% missing while the column is quietly blank.
        "missing_rates": {c: f"{(df[c].isna() | (df[c].astype('string') == '')).mean() * 100:.1f}%"
                          for c in df.columns},
        "duplicate_urls": int(df["url"].duplicated().sum()),
        "price_min": float(prices.min()) if len(prices) else None,
        "price_max": float(prices.max()) if len(prices) else None,
        "price_nonpositive": int((prices <= 0).sum()),
        # The check that a range test cannot make for you.
        "currencies": sorted(df["currency"].dropna().unique()),
        "currency_mixed": df["currency"].nunique() > 1,
    }
    return df, report

df, report = validate(rows)     # the report below is this dict, laid out for reading

# The 7th check, which no rule can make for you. Read these 3 rows yourself.
print("\nspot check, 3 random rows:")
for row in random.sample(rows, 3):
    print(f"   {row['title'][:38]:<40}{row['currency']}  {row['price']}  {row['rating']} stars")

Against the 60 rows from the 3-page crawl, the report reads:

rows: 60   columns: ['title', 'price', 'currency', 'rating', 'in_stock', 'url', 'source_url', 'scraped_at']

missing_rates:
   title            0.0%
   price            0.0%
   currency         0.0%
   rating           0.0%
   in_stock         0.0%
   url              0.0%
   source_url       0.0%
   scraped_at       0.0%
duplicate_urls         0
price_min              12.84
price_max              57.31
price_nonpositive      0
currencies             ['GBP']
currency_mixed         False

spot check, 3 random rows:
   America's Cradle of Quarterbacks: West   GBP  22.50  3 stars
   Penny Maybe                              GBP  33.29  3 stars
   The Natural History of Us (The Fine Ar   GBP  45.22  3 stars

Log those numbers on every run and alert when a rate moves more than a few points.

Store evidence for debugging

When a parse fails, you need the HTML that caused it. Keep a sample:

import hashlib
from pathlib import Path

if not rows:
    failures = Path("failures")
    failures.mkdir(exist_ok=True)     # write_bytes will not create it for you
    name = hashlib.sha256(url.encode()).hexdigest()[:12]
    (failures / f"{name}.html").write_bytes(response.content)

Save every failure, plus roughly 1 in 50 successes so you have a baseline to diff against. At the 51 KB of a list page, that sampling rate costs about 1 MB per 1,000 pages crawled.

A saved good page is worth more than a sample you only look at. Point a test at it and a broken selector fails in CI, days before an empty column turns up in a report:

from pathlib import Path
# Save one known-good page as a fixture first, once:
#   Path("fixtures/list.html").write_bytes(response.content)

def test_list_page_coverage():
    cards = BeautifulSoup(Path("fixtures/list.html").read_bytes(), "lxml").select("article.product_pod")
    assert len(cards) == 20, f"expected 20 cards, got {len(cards)}"
    rows = [parse_card(c, "https://books.toscrape.com/") for c in cards]
    for field in ("title", "price", "currency", "url"):
        missing = sum(1 for r in rows if not r[field])
        assert missing == 0, f"{field}: {missing}/20 missing, its selector probably broke"

Run it in CI to catch your own changes breaking the parser, and on a schedule against the newest saved page to catch changes on the site itself. The day the site renames price_color, the scheduled run fails loudly and names the field that went missing, instead of the run quietly writing a column of blanks.

What Are the Common BeautifulSoup Web Scraping Mistakes

Most broken scrapers fail for one of these 10 reasons, roughly in the order you'll meet them.

• No timeout: The script hangs forever on one stalled connection.
• No status check: You parse an error page and get zero rows with no explanation.
• response.text on a page with no charset header: Silent mojibake.
• No parser specified: Different machines produce different data.
• Brittle selectors: Long chains of generated class names can break on the next deploy.
• No pagination stop rule: One bug becomes a thousand requests.
• Title-based dedupe: 2 different products that share a title collapse into one.
• No scraped_at timestamp: You can't tell fresh rows from stale ones.
• Text instead of attributes: Truncated titles, missing ratings.
• findAll and text=: Both are deprecated, and findAll stops working in a later release.

When several of those are true at once, the order matters, because a broken fetch makes every parsing problem below it look worse than it is.

Fix order checklist

Fix in this order. Each step depends on the one before it.

1. Request success. Are you getting 200s with real HTML?
2. Parsing reliability. Does your selector find the right elements on 10 different pages?
3. Pagination. Are you getting every page, exactly once?
4. Cleaning. Are types, encodings and formats consistent?
5. Performance. Now it's finally worth making it faster.

Threads on a scraper whose selector misses half the cards only collect the wrong half faster.

Over-scraping too early

Start with one page. Get one record correct. Then 10, then a category, then everything.

If you send many fast requests while your selectors are still wrong, you get rate limited for data you're going to throw away. Cache the HTML while you iterate, and you can debug your parser without sending a single new request.

How to Handle Blocks in Web Scraping Using BeautifulSoup

A block shows up as one of these: 403 Forbidden, 429 Too Many Requests, a CAPTCHA page, or HTML that carries almost no readable text.

The recovery flow is boring: send a complete header set, reuse a session, slow down, and retry with backoff. For the full treatment, see preventing IP bans in large-scale web scraping.

Work up those layers in order of cost, and expect to need more than one of them.

Read robots.txt before you tune the pace

robots.txt lists the paths that a crawler is asked to leave alone, and the standard library parses it:

import requests
from urllib.robotparser import RobotFileParser

# RobotFileParser.read() fetches with urllib and takes no timeout, so a slow
# robots host can hang a scheduled run forever. Fetch it yourself with one.
rp = RobotFileParser()
try:
    resp = requests.get("https://example.com/robots.txt", headers=HEADERS, timeout=10)
except requests.RequestException:
    rp.disallow_all = True                   # can't reach it, so stay off
else:
    if resp.status_code >= 500 or resp.status_code in (401, 403):
        rp.disallow_all = True               # broken or off limits, so stay off
    elif resp.status_code < 400:             # 2xx/3xx: real rules to read
        # decode the bytes as UTF-8 per RFC 9309, not resp.text, whose encoding
        # is guessed from headers and can corrupt non-ASCII.
        rp.parse(resp.content.decode("utf-8", "replace").splitlines())
    else:
        rp.allow_all = True                  # other 4xx: no robots.txt, nothing barred
rp.can_fetch("*", "https://example.com/catalogue/")

That last branch is the one people leave out, and leaving it out reverses the answer. can_fetch() returns False for everything until the parser has been told something, so a site with no robots.txt at all silently bars your whole crawl unless you set allow_all yourself. A 401 or 403 goes the other way here. RFC 9309 groups every 4xx as unavailable and lets you crawl, but urllib.robotparser treats a robots.txt you aren't allowed to read as a full disallow, and that is the cautious reading to keep.

In our test, urllib.robotparser behaved differently across Python versions when * and $ appeared in robots.txt rules. Running identical rules on both:

Python 3.13.13 /report.pdf -> True
Python 3.14.5 /report.pdf -> False

Because parser behaviour can vary by version, verify can_fetch() on the Python version you actually deploy if these rules affect your crawl.

Many publishers now keep separate rules for AI crawlers alongside the classic ones, and robots.txt files have grown accordingly. The Guardian's robots.txt named 39 user-agents, and 37 of them sat in a single group under a blanket Disallow: /, including ClaudeBot, CCBot, PerplexityBot, Bytespider and Amazonbot. So the rules that apply to you may be different from the rules under *. Pass the user-agent you actually send rather than accepting the default:

rp.can_fetch("MyScraper/1.0 (+https://example.com/bot)", url)

robots.txt is a request, not an access control, and it is one check rather than the check. A site's terms of use are separate, and personal data carries its own obligations whatever robots.txt says. Whether a given scrape is actually allowed is a legal question rather than a technical one, so treat clearing robots.txt as the start of that question rather than the end of it.

Rate limiting that works

Sleep between requests and randomise it:

time.sleep(1 + random.uniform(0, 0.5))

Fixed delays produce a machine-perfect request rhythm. Jitter breaks it up. Start at 1 request per second or slower, watch your error rate, and only speed up if it stays clean.

Work out what that pace means before you start, because it decides whether a job is an afternoon or a week. That sleep averages 1.25 seconds, and a session request measured 464 ms on our connection, so a page cost about 1.7 seconds end to end. Measure your own before you plan from it. At 1,000 pages that's half an hour. At 10,000, closer to 5 hours. At 100,000, about 2 days of wall-clock time. At that point the answer isn't a faster loop but fewer pages, a sitemap, or an API. Honour Retry-After when a server sends one, since that's the site telling you exactly what it wants.

The 2 codes aren't symmetric. A 429 is an instruction to wait, and a 403 is a refusal. Retrying a 403 faster is how a temporary block can become a permanent one.

Detect block pages

A 200 doesn't mean you got the page. Check that what you came for is actually present:

def looks_blocked(response, expect_selector):
    soup = BeautifulSoup(response.content, "lxml")
    if soup.select_one(expect_selector):
        return None
    title = soup.title.get_text(strip=True) if soup.title else "(no title)"
    return f"HTTP {response.status_code}, {len(response.content):,} bytes, title={title!r}"

Called twice against the same successful response, once with a selector that exists and once with one that doesn't:

looks_blocked(r, 'article.product_pod')   -> None                    # fine
looks_blocked(r, 'div.checkout-form')     -> HTTP 200, 51,294 bytes, title='All products | Books to Scrape - Sandbox'

Assert on the thing you need, not on the status code. Log the title and byte count when the assertion fails, because together they identify the block immediately.

Here's what 3 real refusals looked like when we sent a bare python-requests/2.34.2 user agent at them:

https://www.g2.com/       HTTP 403    1,685 bytes   50 chars of text   title='g2.com'
https://www.zillow.com/   HTTP 403    5,776 bytes   35 chars of text   title='Access to this page has been denied'
https://www.indeed.com/   HTTP 403   25,724 bytes   77 chars of text   title='Blocked - Indeed.com'

That output contradicts the usual assumptions about block pages in 3 ways. The block pages aren't small: Indeed returned nearly 26 KB, which is heavier than plenty of real pages, so "the response was tiny" isn't a reliable test on its own. The ratio of markup to readable text is the real signal instead. All 3 carry under 80 characters of readable text, so a page that's kilobytes of markup wrapped around one sentence is often a challenge page. And Indeed's byte count moved between 2 runs an hour apart, from 27,666 to 25,724, because the challenge page is generated per request. Don't assert on an exact length.

Cheap signals worth checking: readable text far below the page's normal volume, expected container missing, a title that contains "just a moment", "denied" or "blocked" once you lowercase it, or a text/html body when you asked for JSON. A check for the exact phrase "Access Denied" would have missed both of those titles, while lowercasing and looking for "denied" or "blocked" caught them. The response headers help too, since server: cloudflare alongside a 403 names the thing that stopped you.

A challenge page often says "Please enable JavaScript", and that message sends people off to install a headless browser when what they actually hit was a block. Check for the block first.

Proxies when scaling

Rate limits and bans can arrive long before parsing becomes the bottleneck, and no amount of selector tuning fixes a 429. They can be triggered by several signals, including IP reputation, request volume, session behaviour and authentication state.

Which session format you want depends on whether anything needs to be kept between requests:

• Rotating sessions: each request leaves through a different IP from your allocated pool, which fits large one-shot crawls where no state carries between pages.
• Sticky sessions: one session ID returns the same IP for a fixed window, up to 24 hours on Live Proxies plans, which is what a flow depending on cookies, a cart or a logged-in view requires.

Rotating residential proxies come from real home connections, which is why they clear more often than datacenter ranges: the addresses sit in ISP-issued consumer ranges rather than in a hosting provider's block. How much that helps still depends on the individual IP's history and on how you behave once you're through.

The rotation isn't a scheduler cycling through a list. The addresses change because the home connections behind them change, when an ISP reassigns a router's address, when someone reboots, or when a peer leaves the network and is replaced.

Live Proxies allocates those IPs privately per plan and doesn't put them on the same targets for another customer. That reduces overlap on your target rather than guaranteeing a clean address, because an IP still carries whatever history it arrived with and the target's own detection still applies. The guide to rotating proxies covers the mechanics, and the guide to residential, datacenter and mobile proxies compares the trade-offs.

Changing your IP fixes only the address you arrive from. It does nothing about the layer underneath your headers: the TLS handshake. Plain requests speaks HTTP/1.1 only, and its handshake doesn't look like a browser's, so a site can classify you before you send a single header. No header setting can fix that, because the handshake happens first.

So if you're blocked no matter what headers you set, the usual next step is a client that does match a browser's TLS fingerprint, such as curl_cffi. It's close to a drop-in. Run pip install curl_cffi, swap the import, add one argument, and the parsing code below it doesn't change:

from curl_cffi import requests   # in place of `import requests`

response = requests.get("https://example.com/", impersonate="chrome", timeout=25)
soup = BeautifulSoup(response.content, "lxml")

impersonate="chrome" sends Chrome's real TLS and HTTP/2 handshake instead of the one that requests sends. In a check it cleared Zillow, while plain requests on the same machine still got 403. It did not clear G2.

Matching the fingerprint is only the minimum now, not the whole job. By 2026 the large anti-bot services score several signals at once and check that they agree: IP reputation, the TLS and browser fingerprints, and how the session behaves. So a browser-shaped handshake from a flagged datacenter IP still fails.

With that layer accounted for, adding the proxy changes nothing about the parsing code you've already written. It's one argument on the request:

import os
import requests
from bs4 import BeautifulSoup

# Reuses the HEADERS set you already defined. Keep the proxy URL in
# the environment rather than in the file. It carries your password, the file ends
# up in version control, and requests prints the URL in full inside connection
# errors, so a hardcoded one leaks into every traceback and log:
#   export LP_PROXY="http://USERNAME-ACCESS_CODE-1:PASSWORD@PROXY_HOST:PORT"
# The trailing number on the username is the sticky session ID. Keep it to
# hold one IP, drop it to rotate through the pool.
PROXY = os.environ["LP_PROXY"]
proxies = {"http": PROXY, "https": PROXY}

with requests.Session() as session:
    response = session.get(
        "https://books.toscrape.com/",
        headers=HEADERS,
        proxies=proxies,
        timeout=20,
    )
    response.raise_for_status()

soup = BeautifulSoup(response.content, "lxml")
print(len(soup.select("article.product_pod")))

With the proxy in place, test a small sample of exits before you point a large crawl at a new pool. The proxy tester runs one request through an IP and reports the exit country and whether it came back good, slow or failing. Use it to check connectivity, latency and geographic targeting across several exits, so a bad or mis-targeted pool shows up before you scale the crawl.

The same URL isn't the same page. There's a second reason to route through a proxy, and it has nothing to do with getting blocked: your exit IP can decide which document the server hands you.

We requested https://www.nike.com/ 3 times through exits in 3 countries and followed the redirects:

country           ISP                            page actually served
United States     AS46690 Verizon Business       https://www.nike.com/
Canada            AS855 Bell Canada              https://www.nike.com/ca/
United Kingdom    AS5607 Sky UK Limited          https://www.nike.com/gb/

That's 1 URL, 3 documents. The prices are in 3 currencies, the availability differs, and nothing in the request said which one we wanted except where the packet came from. If you scrape that URL from a datacenter in Virginia and report the numbers as "the prices", you've quietly published US prices under a global label. That class of error doesn't raise an exception and doesn't show up in a missing-value count. It survives all 7 validation checks, because the data is complete and well formed. It's just answering a different question than the one you asked.

So geography is a scraping parameter, not a networking detail. Decide which market you're collecting, then pick exits in it. Live Proxies supports 55+ locations worldwide, with a particularly strong presence in the United States, Canada, and the United Kingdom. The exit country is part of the proxy URL, so the same script can collect data across different markets by swapping the LP_PROXY value. B2B Enterprise extends geo-targeting beyond the standard set.

One practical note from those runs: the same property that makes these exits work is what makes their latency uneven. They're genuine home connections, not datacenter ranges. An individual request occasionally times out where a datacenter IP wouldn't. A retry lands you on a different exit and generally succeeds, which is the retry-with-backoff helper doing 2 jobs at once. That only works while you're rotating. If you keep the session ID digit on the username, the retry leaves through the same IP for up to an hour, which is exactly the wrong behaviour when the exit itself is what failed. Check whether the string you copied carries a trailing session digit before you use it for bulk collection. Drop the digit for bulk collection and keep it only for the flows that genuinely need one address.

How to Make Your BeautifulSoup Scraper Faster

Before optimising, check where the time goes. Parsing a 51 KB page took about 5 ms. One HTTP request to the same page took over 1,500 ms. Network dominates, so reducing requests matters more than micro-optimising selectors.

That same imbalance points to the other option: overlap the requests instead of waiting on each one in turn. requests and BeautifulSoup are synchronous, so a sequential loop sits idle on the network for almost all of its wall-clock time.

# Reuses HEADERS and the urls list of catalogue pages (page-1 to page-50).
import threading
from concurrent.futures import ThreadPoolExecutor

thread_local = threading.local()

def worker_session():
    # One Session per worker thread. requests.Session isn't guaranteed
    # thread-safe, so sharing one races on its connection pool and cookies.
    if not hasattr(thread_local, "s"):
        thread_local.s = requests.Session()
        thread_local.s.headers.update(HEADERS)
    return thread_local.s

def count_products(url):
    response = worker_session().get(url, timeout=20)
    response.raise_for_status()
    return len(BeautifulSoup(response.content, "lxml").select("article.product_pod"))

with ThreadPoolExecutor(max_workers=4) as pool:
    counts = list(pool.map(count_products, urls[:8]))

Against the first 8 pages, run both ways:

sequential, 1 at a time              3.98s   160 products
4 workers, same 8 pages              2.10s   160 products

Those 4 workers bought roughly twice the speed, not 4 times, and the ratio moved between 1.5 and 2.0 across 6 runs. The sequential loop was already reusing one connection, so each page after the first cost about what a session request costs rather than the 1,500 ms a cold one does. Most of the win was taken before any thread started. Plan from your own measurement rather than from the worker count. An async client such as httpx overlaps the same way if you'd rather not manage threads.

The cap is politeness, not Python's speed. Those same 4 workers doubled the request rate from a single address, which is the number a rate limiter counts, so cap the pool per domain and keep a delay between requests to the same host. When you want that scheduling as a default rather than something you build yourself, that's where Scrapy becomes worth using.

Reduce requests

Cache aggressively while developing:

CACHE = Path("cache")
CACHE.mkdir(exist_ok=True)     # write_bytes will not create the directory for you

def cached_get(url, session, max_age=3600):
    key = hashlib.sha256(url.encode()).hexdigest()[:16]
    path = CACHE / f"{key}.html"
    if path.exists() and time.time() - path.stat().st_mtime < max_age:
        return path.read_bytes(), True
    response = session.get(url, timeout=10)
    response.raise_for_status()
    path.write_bytes(response.content)
    return response.content, False

Running it 3 times for books.toscrape.com/catalogue/page-2.html, the first cold and the rest served from disk:

  run 1: 50,877 bytes from network  1524.2 ms
  run 2: 50,877 bytes from cache       0.2 ms
  run 3: 50,877 bytes from cache       0.1 ms

Iterating on selectors against a cached file is thousands of times faster and sends zero traffic to the site. The other way to cut requests is to skip detail pages unless the list page genuinely lacks the field you need.

Once you're running through residential proxies, that cache stops being a convenience and starts saving real money, because residential bandwidth is billed by the gigabyte. Do the arithmetic with that 50,877-byte page:

• A 100,000-page crawl moves about 5.1 GB.
• Re-running a 500-page crawl 20 times while you tune selectors moves about 0.51 GB, and every byte of it is spent re-downloading markup you already had on disk.
• Following a detail page for every product on those 100,000 pages moves about 35 GB, roughly 7 times the list-page crawl. A detail page here averages about 18 KB against the list page's 51 KB, but there are 20 of them per list page.

Multiply by your own per-GB rate: re-running that 500-page crawl 20 times bills 0.51 GB, and served from disk it bills none. This is also the argument for stream=True, which hands you the headers before the body downloads, so you can check the type or size and skip the fetch, or pull only the first chunk of a huge file with iter_content. And it's the argument for never fetching images you aren't going to use, since a single product photo can outweigh the entire HTML document that links to it.

Parse efficiently

Use lxml. We benchmarked all 3 parsers on the same 51,294-byte page, 20 runs, median:

The speed gap is real but modest at this size, and the ratio holds as documents grow. On a 600 KB page html5lib still costs about 2.7 times what lxml does, which is 100 ms of difference rather than 9. Correctness is the better reason to choose lxml. Parsers disagree on markup that the HTML Standard explicitly allows:

markup = "<ul><li>Alpha<li>Beta<li>Gamma</ul>"    # li end tags omitted, which is legal

for parser in ("html.parser", "lxml", "html5lib"):
    items = BeautifulSoup(markup, parser).find_all("li")
    print(f"{parser:<12} count={len(items)}  {[i.get_text() for i in items]}")

The counts agree. The text doesn't:

html.parser  count=3  ['AlphaBetaGamma', 'BetaGamma', 'Gamma']
lxml         count=3  ['Alpha', 'Beta', 'Gamma']
html5lib     count=3  ['Alpha', 'Beta', 'Gamma']

There's no exception and no empty list, just wrong values in your CSV. This is why the parser argument isn't optional.

Parser choice is the big win here. The other 2 are smaller: search inside the card you already found rather than re-querying the whole document, and extract every field from a card before moving on:

for card in soup.select("article.product_pod"):
    title = card.select_one("h3 a")["title"]     # scoped to the card
    price = card.select_one("p.price_color")     # not soup.select_one(...)

Each scoped call searches one card's subtree, so the saving grows with the length of the page.

That leaves one parser to explain. html5lib earns its place in a single case: it's the only one of the 3 that reproduces browser behaviour exactly, including synthesising . Version 1.1 from June 2020 was still the newest release on PyPI, so treat it as a specialist tool rather than a default.

When parsing itself becomes the cost, usually on re-parses of pages you already have on disk, the fast option is selectolax. It wraps a browser-grade C engine, and it parsed the same page about 7 times faster than BeautifulSoup does with lxml. Bare lxml.html, without the BeautifulSoup layer on top, is faster still.

Run incremental updates

On a schedule, only fetch what changed. Keep the IDs you've seen and skip them:

import json, requests
from pathlib import Path
from urllib.parse import urljoin
from bs4 import BeautifulSoup

SEEN_FILE = Path("seen.json")
# Reuses the HEADERS set you already defined.

URL = "https://books.toscrape.com/"
response = requests.get(URL, headers=HEADERS, timeout=10)
response.raise_for_status()          # a 403 here would look like "no new items"
soup = BeautifulSoup(response.content, "lxml")

seen = set(json.loads(SEEN_FILE.read_text())) if SEEN_FILE.exists() else set()
# Key on the resolved URL, not href.split("/")[0]. Page 1 hrefs start with
# "catalogue/", so splitting collapses every product on the page to one id.
ids = [urljoin(URL, a["href"]) for a in soup.select("article.product_pod h3 a")]
new = [i for i in ids if i not in seen]

print(f"{len(ids)} items on the page, {len(new)} not seen before")
SEEN_FILE.write_text(json.dumps(sorted(seen | set(ids))))

The last line is the one that makes it incremental. Without writing the set back, every run is a first run. Running it twice against an unchanged page:

first run : 20 items on the page, 20 not seen before
second run: 20 items on the page,  0 not seen before

For freshness, persist a date per id instead of the flat seen set. Then re-scan any id older than your chosen window, less often than you check for new items.

How to Point This at Your Own Site

The path that works: pick one page type, build a minimal scraper, add headers and timeouts, then pagination and dedupe. Move to detail pages only when the list page genuinely lacks a field. Clean and validate before you trust any of it, and add pages only while your missing rates stay flat.

By now that one file should hold 8 pieces, in the order you built them:

• parse_card() for list-page rows
• the HEADERS set
• the fetch() retry helper
• crawl() with its 3 stop rules
• get_soup()
• parse_detail() for the richer record
• parse_price() with its PriceError
• validate() for the report you read before trusting the rows

The fixture test lives beside them.

Most of what separates a scraper that survives from one that breaks isn't clever parsing. It's the basics: passing .content instead of .text, naming your parser, checking that a 200 contains what you asked for, and capping your crawl.

The step-by-step build ran against a sandbox, so that you can run it yourself and get the same numbers back. The failures came from real sites instead: the prices that defeat a one-line regex from a live German page, the variants from IKEA, and 3 live 403s from G2, Zillow and Indeed. The crawler rules came from the Guardian's robots.txt, and the geo redirects from nike.com. A sandbox is the honest way to teach the basic steps and the wrong way to finish. So run it against the site you actually came here for, in this order:

1. Open one of its list pages and press Ctrl+U, or Cmd+Option+U on a Mac. If your field isn't in view source, check the script tags for a JSON blob first, since the rest of these steps assume the data is in the HTML.
2. Find the smallest repeating block that holds one record, or the largest part of one that a single block holds, and name one field inside it.
3. Fetch that page once with a timeout, a header set and .content, then check the count you get back against what you can see on the page.
4. Add pagination only when that number is right, and add a stop rule after that.

Expect the first real site to behave worse than this one. Its class names are likely to be generated, its prices won't look like £51.77, and it may give you a 403 before it gives you anything else. When the 403 arrives, the parser you built doesn't change. Only the fetch layer does: a complete header set, a reused session, a slower pace, and a proxy string that also decides which country you arrive from.

Scrape one list page, export a CSV with source_url and scraped_at on every row, and read 3 of those rows yourself.

Further reading: How to Scrape Dynamic Content from a Website in 2026 and How to Scrape Data from an Ecommerce Website in 2026.

The Complete BeautifulSoup Scraper

parse_price sits above parse_card, its caller, so this is not the order you built it in:

"""books.toscrape.com: list pages to a validated CSV."""

import csv
import random
import re
import time
from datetime import datetime, timezone
from decimal import Decimal
from urllib.parse import urljoin

import pandas as pd
import requests
from bs4 import BeautifulSoup

START = "https://books.toscrape.com/"
WORDS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
RETRY_STATUSES = {429, 500, 502, 503, 504}

HEADERS = {
    # Copy this from your own browser (chrome://version) rather than reusing
    # the string printed here. A User-Agent that stopped moving years ago is
    # itself a signal, which is the opposite of what a header set is for.
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept-Encoding": "gzip, deflate",
}

SPACES = " \xa0\u202f\u2009"  # space, nbsp, narrow nbsp, thin space
NUMBER = re.compile(rf"\d[\d.,{SPACES}]*\d|\d")
SYMBOLS = {"£": "GBP", "$": "USD", "€": "EUR", "¥": "JPY", "₹": "INR"}
ZERO_DECIMAL = {"JPY", "KRW", "ISK"}  # currencies with no minor unit


class PriceError(ValueError):
    pass


def parse_price(text, decimal_hint=None):
    """Return (Decimal, currency) or raise. Never guess."""
    text = text or ""
    numbers = NUMBER.findall(text)
    if len(numbers) != 1:  # a range, or a sale price pair
        raise PriceError(f"{len(numbers)} numbers in {text!r}, expected 1")

    body = numbers[0].strip(SPACES)
    currency = next((c for s, c in SYMBOLS.items() if s in text), None)
    # A leading minus or accounting parentheses make it negative. Without this the
    # sign is silently dropped and a refund of -5.00 is stored as income of 5.00.
    start = text.find(numbers[0])
    negative = bool(re.search(r"[-−]\s*[£$€¥₹]?\s*$", text[:start])) or (
        "(" in text[:start] and ")" in text[start + len(numbers[0]) :]
    )
    dot, comma = body.rfind("."), body.rfind(",")

    if dot > -1 and comma > -1:
        sep = "." if dot > comma else ","  # the later one is the decimal
    elif dot > -1 or comma > -1:
        found = "." if dot > -1 else ","
        tail = len(body) - max(dot, comma) - 1
        if tail == 3 and body.count(found) == 1:
            # "1.299" is 1299 in Berlin and 1.299 in Boston. Genuinely ambiguous.
            if currency in ZERO_DECIMAL:
                sep = "," if found == "." else "."
            elif decimal_hint is None:
                raise PriceError(f"{body!r} is ambiguous, pass decimal_hint")
            else:
                sep = decimal_hint
        elif body.count(found) > 1:
            sep = "," if found == "." else "."  # repeated, so it groups
        else:
            sep = found
    else:
        sep = "."

    group = "," if sep == "." else "."
    cleaned = body.replace(group, "").translate({ord(c): None for c in SPACES})
    value = Decimal(cleaned.replace(sep, "."))
    return (-value if negative else value), currency


def parse_card(card, source_url):
    link = card.select_one("h3 a")
    price_text = card.select_one("p.price_color").get_text(strip=True)
    try:
        price, currency = parse_price(price_text)
    except PriceError as exc:
        # Record the gap rather than dropping the row. A row you never wrote is
        # invisible, so a broken price selector would show up as a shorter file
        # instead of as a column the missing-rate check can count.
        print(f"  unparsed price {price_text!r} on {source_url}: {exc}")
        price, currency = None, None

    # "star-rating Three" -> 3. The rating is in the class, not the text.
    classes = card.select_one("p.star-rating")["class"]
    rating = next((WORDS[c] for c in classes if c in WORDS), None)

    return {
        "title": link["title"],
        "price": price,
        "currency": currency,
        "rating": rating,
        "in_stock": "In stock" in card.select_one("p.availability").get_text(),
        "url": urljoin(source_url, link["href"]),
        "source_url": source_url,
        "scraped_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    }


def fetch(session, url, *, attempts=4, timeout=10):
    for attempt in range(1, attempts + 1):
        try:
            response = session.get(url, headers=HEADERS, timeout=timeout)
        except requests.RequestException as exc:
            if attempt == attempts:
                raise
            delay = 2**attempt + random.uniform(0, 1)
            print(
                f"  [{attempt}/{attempts}] {type(exc).__name__}, sleeping {delay:.1f}s"
            )
            time.sleep(delay)
            continue

        if response.status_code in RETRY_STATUSES and attempt < attempts:
            # Honour a numeric Retry-After, back off otherwise. The HTTP-date form
            # is not parsed here, so it falls through to the exponential delay.
            retry_after = response.headers.get("Retry-After")
            # Cap it. Retry-After is the remote server's number, and an unbounded
            # one parks a scheduled run for as long as that server likes.
            delay = (
                min(float(retry_after), 300)
                if (retry_after or "").isdigit()
                else 2**attempt
            )
            delay += random.uniform(0, 1)
            print(
                f"  [{attempt}/{attempts}] HTTP {response.status_code}, sleeping {delay:.1f}s"
            )
            time.sleep(delay)
            continue

        response.raise_for_status()
        return response

    raise RuntimeError(f"gave up on {url} after {attempts} attempts")


def get_soup(session, url):
    return BeautifulSoup(fetch(session, url).content, "lxml")


def parse_detail(soup, url, source_list_url):
    specs = {}
    for row in soup.select("table.table-striped tr"):
        if row.th is None or row.td is None:
            continue
        key = re.sub(r"[^\w]+", "_", row.th.get_text(strip=True).lower()).strip("_")
        specs[key] = row.td.get_text(strip=True)

    price, currency = parse_price(specs.get("price_incl_tax", ""))
    available = re.search(r"\((\d+) available\)", specs.get("availability", ""))
    heading = soup.select_one("#product_description")
    para = heading.find_next_sibling("p") if heading else None

    return {
        "title": soup.select_one("h1").get_text(strip=True),
        "category": [a.get_text(strip=True) for a in soup.select("ul.breadcrumb a")][
            -1
        ],
        "price": price,
        "currency": currency,
        "units_available": int(available.group(1)) if available else None,
        "description": re.sub(r"\s+", " ", para.get_text(strip=True)) if para else "",
        "url": url,
        "source_list_url": source_list_url,
    }


def crawl(start=START, max_pages=3):
    rows, seen = [], set()
    url, page = start, 0

    with requests.Session() as session:
        while url and page < max_pages:  # stop rule 1
            page += 1
            # fetch() carries the retries; a transient 503 no longer aborts the crawl.
            soup = BeautifulSoup(fetch(session, url).content, "lxml")

            cards = soup.select("article.product_pod")
            if not cards:
                # 0 cards is a block or a renamed class, not an empty page.
                # Without this, stop rule 2 reads it as a clean finish.
                raise RuntimeError(
                    f"no cards on {url}, title={soup.title and soup.title.get_text(strip=True)!r}"
                )

            new = 0
            for card in cards:
                row = parse_card(card, url)
                if row["url"] in seen:  # stable key, not the title
                    continue
                seen.add(row["url"])
                rows.append(row)
                new += 1

            print(f"  page {page}: {len(rows):>3} total, {new:>2} new")

            if new == 0:  # stop rule 2
                break

            next_link = soup.select_one("li.next a")
            if not next_link:  # stop rule 3
                break
            url = urljoin(url, next_link["href"])

            time.sleep(
                1 + random.uniform(0, 0.5)
            )  # a fixed delay is a machine-perfect rhythm
    return rows


def validate(rows):
    df = pd.DataFrame(rows)
    prices = df["price"].dropna()  # unparsed prices are None, not dropped rows
    report = {
        # Count "" as missing too, or a selector that returns an empty string
        # reports 0% missing while the column is quietly blank.
        "missing_rates": {
            c: f"{(df[c].isna() | (df[c].astype('string') == '')).mean() * 100:.1f}%"
            for c in df.columns
        },
        "duplicate_urls": int(df["url"].duplicated().sum()),
        "price_min": float(prices.min()) if len(prices) else None,
        "price_max": float(prices.max()) if len(prices) else None,
        "price_nonpositive": int((prices <= 0).sum()),
        # The check that a range test cannot make for you.
        "currencies": sorted(df["currency"].dropna().unique()),
        "currency_mixed": df["currency"].nunique() > 1,
    }
    return df, report


if __name__ == "__main__":
    rows = crawl()

    with open("books.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)
    print(f"\nwrote {len(rows)} rows to books.csv")

    df, report = validate(rows)
    print(f"\nmissing rates   {report['missing_rates']}")
    print(f"duplicate_urls  {report['duplicate_urls']}")
    print(
        f"price range     {report['price_min']} to {report['price_max']} "
        f"{report['currencies']}, mixed={report['currency_mixed']}"
    )

    # The 7th check, which no rule can make for you. Read these 3 rows yourself.
    print("\nspot check, 3 random rows:")
    for row in random.sample(rows, 3):
        print(
            f"   {row['title'][:38]:<40}{row['currency']}  {row['price']}  {row['rating']} stars"
        )

    # Stage 2. Only worth it when the list page lacks a field you need.
    with requests.Session() as session:
        print("\nstage 2: visiting 2 detail pages")
        for row in rows[:2]:
            try:
                detail = parse_detail(
                    get_soup(session, row["url"]), row["url"], row["source_url"]
                )
            # parse_detail raises more than network errors: PriceError on an unreadable
            # price, LookupError on a missing breadcrumb, AttributeError on a missing h1.
            except (
                requests.RequestException,
                LookupError,
                AttributeError,
                PriceError,
            ) as exc:
                print(f"   skipped {row['url']}: {type(exc).__name__}: {exc}")
                continue
            print(
                f"   {detail['title'][:30]:<32}{detail['category']:<20}"
                f"{detail['units_available']} units  {len(detail['description'])} char description"
            )
            time.sleep(1 + random.uniform(0, 0.5))

Against the sandbox that prints:

  page 1:  20 total, 20 new
  page 2:  40 total, 20 new
  page 3:  60 total, 20 new

wrote 60 rows to books.csv

missing rates   {'title': '0.0%', 'price': '0.0%', 'currency': '0.0%', 'rating': '0.0%', 'in_stock': '0.0%', 'url': '0.0%', 'source_url': '0.0%', 'scraped_at': '0.0%'}
duplicate_urls  0
price range     12.84 to 57.31 ['GBP'], mixed=False

spot check, 3 random rows:
   America's Cradle of Quarterbacks: West  GBP  22.50  3 stars
   The Elephant Tree                       GBP  23.82  5 stars
   Foolproof Preserving: A Guide to Small  GBP  30.52  3 stars

stage 2: visiting 2 detail pages
   A Light in the Attic            Poetry              22 units  1017 char description
   Tipping the Velvet              Historical Fiction  20 units  1029 char description

Run that against the sandbox first, so you have a known-good baseline to compare against. Then change what is site-specific: START, the card selector and the next-link selector in crawl(), and the field selectors in parse_card().

Frequently Asked Questions

How to do web scraping using BeautifulSoup if the site changes often?

Write selectors that survive redesigns: prefer IDs, data attributes and semantic containers over generated class names, and add a fallback chain so a missing element degrades to None rather than crashing. Save a raw HTML sample from each run, so you can replay a failure or re-test the parser after a redesign without touching the site again. Track missing rates per field, since a jump from 0% to 100% is often a sign of a broken selector. If your main cost is selectors that keep breaking, Scrapling is built for exactly this: it stores a fingerprint of each element you match and relocates the closest match after a redesign moves it.

How to use BeautifulSoup for web scraping on dynamic websites?

Confirm the problem before solving it. Open view source and search for a value you want. If it's absent, check the script tags, because many client-rendered pages embed their data as JSON in the HTML, and parsing that's faster and more stable than driving a browser. If the data genuinely isn't there, the page is loading it from an API you can often call directly.

How to build a web scraper python project that runs daily?

Wrap your scraper in a single entry point and schedule that. On Linux a crontab line like 0 6 * * * cd /srv/scraper && .venv/bin/python run.py >> run.log 2>&1 runs it at 6am every day, and a task runner or a cloud scheduler does the same on other systems. Make each run idempotent so a retry doesn't duplicate rows. Store scraped_at on every record, keep a set of seen IDs so you only fetch new items, and log row counts and missing rates to a file you actually read. Alert on zero rows, because silence often means a block rather than no new data.

How to avoid getting blocked when web scraping using BeautifulSoup and requests?

Send a complete, consistent header set rather than just a User-Agent, reuse a Session so your connection and cookies persist, keep concurrency low per domain, and back off exponentially on 429 and 5xx. Only ask for Accept-Encoding values you can actually decode. Treat a 429 as an instruction to wait and a 403 as a refusal that retrying won't fix.

How to scrape multiple websites with the same BeautifulSoup scraper?

Separate the selectors from the logic. Keep a per-site config mapping field names to selectors, give each field a fallback selector list, write one generic parser that reads that config, and normalise everything into a single output schema. Then downstream code doesn't care which site a row came from.

How to tell whether a BeautifulSoup scraper is quietly returning wrong data?

Assume it is, and build the checks that would catch it. Track missing rates per field and alert when one moves, since a broken selector reports 100% missing rather than raising. Assert that the element you came for is present rather than trusting a 200. Keep the currency beside every price and confirm the column holds a single value, because a number that's wrong but plausible passes a range check. Then read 3 random rows yourself on every run, which is the only check that catches text sitting in the wrong field.

How to scrape a site that uses Cloudflare or heavy bot protection?

Recognise it first: almost no readable text in the response, a title like "Just a moment", a vendor header such as cf-mitigated or a page telling you to enable JavaScript. Slow down, since aggressive retries make things worse. Check whether the data is available through an official API before engineering around the protection. If you still need it, a client that matches a real browser's TLS fingerprint such as curl_cffi handles the client side, and residential proxies handle the IP side.