BeautifulSoup web scraping combines 2 Python libraries: _requests_ downloads a page, and BeautifulSoup presents that HTML as a tree you can query by tag, class or CSS selector. BeautifulSoup doesn't do the parsing itself. That job goes to a parser you choose. Give _<ul><li>Alpha<li>Beta<li>Gamma</ul>_ to _html.parser_ and the first item reads _AlphaBetaGamma_. Give it to _lxml_ and it reads _Alpha_. Both report 3 items, and neither raises. That markup is legal, because the[ HTML Standard](https://html.spec.whatwg.org/multipage/grouping-content.html#the-li-element) lets you drop an _li_ end tag when another _li_ follows. Tools built to feed pages to language models ship BeautifulSoup as a hard dependency: crawl4ai, scrapegraphai and llama-index-readers-web all pin _beautifulsoup4_. This guide builds one scraper file from a single page to a validated CSV, which is where an [AI web scraping](https://liveproxies.io/blog/ai-web-scraping-and-data-collection) pipeline starts. ## TL;DR BeautifulSoup web scraping fails quietly, so the skill is building the checks that catch it. - Check view source first. The browser shows things that _requests_ never receives. - Name the parser and pass _response.content_. Defaults differ by machine, and _.text_ corrupts non-ASCII. - Test for _[]_ after every search, or a selector matching nothing gives an empty dataset and no error. - Parse prices with something that keeps the currency: the usual regex reads _€ 357,76_ as 35776. - Cap page counts, confirm a 200 holds your element, and add rotating proxies once one IP limits you. ## What Is Web Scraping Using BeautifulSoup in Python? [Web scraping](https://liveproxies.io/blog/what-is-web-scraping) using BeautifulSoup is the process of downloading a page's HTML with an HTTP client, then using BeautifulSoup to turn that HTML into a searchable tree you can pull values from. A fetch problem gives you the wrong bytes, and a parse problem gives you the wrong values out of the right bytes. If you're new to the wider ecosystem, the [Python web scraping tutorial](https://liveproxies.io/blog/python-web-scraping) covers how these pieces fit alongside Scrapy and Playwright. The pages that work on day one usually share a shape: many similar records, each in its own repeating block of HTML. - **Directory listings**, where each row is a company or a person, common in lead research. Rows describing identifiable people are personal data under the GDPR and the CCPA, so that use case needs a lawful basis before you collect, not after. - **Blog headlines**, where you want the title, link and date - **Product listing pages**, where you want name, price and availability, which is the basis of much [price monitoring](https://liveproxies.io/blog/what-is-dynamic-pricing) work For records like these you're aiming for a table like this one: | **title** | **price** | **currency** | **rating** | **in_stock** | **url** | **source_url** | **scraped_at** | | --- | --- | --- | --- | --- | --- | --- | --- | | A Light in the Attic | 51.77 | GBP | 3 | True | …/a-light-in-the-attic_1000/index.html | [books.toscrape.com/](http://books.toscrape.com/) | 2026-07-19T02:13:31+00:00 | | Tipping the Velvet | 53.74 | GBP | 1 | True | …/tipping-the-velvet_999/index.html | [books.toscrape.com/](http://books.toscrape.com/) | 2026-07-19T02:13:31+00:00 | | Soumission | 50.10 | GBP | 1 | True | …/soumission_998/index.html | [books.toscrape.com/](http://books.toscrape.com/) | 2026-07-19T02:13:31+00:00 | The URLs are shortened here for width, though the real CSV carries them in full. When a price looks wrong later, _source_url_ and _scraped_at_ are what let you re-fetch that exact page and tell a site redesign apart from a parser bug. ### How BeautifulSoup fits in a Python web scraper BeautifulSoup sits in the middle of a pipeline: **Request → Parse → Extract → Validate → Save** BeautifulSoup handles only the parse and extract steps, and it runs the parse through the parser you chose. It never touches the network. If you hand it a URL string instead of HTML, it warns you rather than fetching anything: ``` BeautifulSoup("https://books.toscrape.com/", "lxml") # MarkupResemblesLocatorWarning: The input passed in on this line looks # more like a URL than HTML or XML. ``` That warning exists because passing a URL is such a common first mistake. ### What BeautifulSoup cannot do BeautifulSoup doesn't execute JavaScript. It reads the HTML you hand it and nothing more, which with _requests_ is whatever the server sent. If a site fetches its data after the page loads, that data isn't in what the server sent you, so BeautifulSoup never sees it. The practical test takes 2 requests. Both of these URLs show a list of quotes in a browser: ``` import requests from bs4 import BeautifulSoup for url in ["https://quotes.toscrape.com/", "https://quotes.toscrape.com/js/"]: response = requests.get(url, timeout=10) soup = BeautifulSoup(response.content, "lxml") print(f"{url:<32}-> {len(soup.select('div.quote'))} matches") ``` The 2 URLs answer differently: ``` https://quotes.toscrape.com/ -> 10 matches https://quotes.toscrape.com/js/ -> 0 matches ``` Same data, same site, same selector. The second page builds its list in the browser, so BeautifulSoup sees an empty container. That doesn't mean you need a headless browser. On pages like this one the data is usually still in the response, sitting in a script tag as JSON. ## When Should You Use BeautifulSoup for Web Scraping? Use BeautifulSoup when the data is already in the HTML that the server sends you. BeautifulSoup is the right tool when: - The values you want appear in **view page source** - The markup is reasonably stable between visits - You want something cheap and fast, since parsing a 51 KB page takes about 5 ms, without the memory and startup cost of a browser It isn't enough when: - Content appears only after a scroll, click or login - The page assembles itself from API calls that the browser makes later **The decision rule:** open the page and view its source, with Ctrl+U on Windows and Linux or Cmd+Option+U on a Mac, then search for a value you want. Use view source, not Inspect Element. Inspect shows you the live DOM after JavaScript has run, which isn't what _requests_ receives. If your value is in view source, you don't need a browser. **The question that comes up first: why not just hand the HTML to a language model?** Because the two aren't alternatives. Several of the tools built for exactly that job ship BeautifulSoup as a hard dependency: | **Package** | **Version** | **Released** | **Requires** | | --- | --- | --- | --- | | crawl4ai | 0.9.2 | 15 July 2026 | _beautifulsoup4~=4.12_ | | scrapegraphai | 2.1.6 | 20 July 2026 | _beautifulsoup4>=4.14.3_ | | llama-index-readers-web | 0.6.0 | 12 March 2026 | _beautifulsoup4<5,>=4.12.3_ | The reason is arithmetic. Models are billed per token, and raw HTML is mostly markup. On the 9 pages this guide scrapes, reducing a page with BeautifulSoup first removed 82 to 94% of the tokens that a model would otherwise read, measured with _tiktoken_ on the _cl100k_base_ encoding. Parse deterministically, and spend model tokens only where determinism genuinely fails: reconciling many sites that describe the same thing differently, classifying text you've already extracted, or writing the selectors in the first place. A model proposes the selector once, you check it, and then you run it a million times without paying the model again. One failure mode is worth knowing because it looks like a model problem and isn't. Hand a model the HTML of a JavaScript-rendered page and ask for selectors, and it will often invent plausible ones for elements that were never in the document you sent. It's answering from what a page like that usually looks like, because the values genuinely aren't there. That's the same condition the view source test detects, and it's cheaper to catch in view source than with an API bill. A model can't see markup that the server didn't send you, and neither can BeautifulSoup. ### BeautifulSoup vs. Selenium Selenium drives a real browser, so JavaScript runs and the DOM fills in. BeautifulSoup only parses HTML you already have. The split depends on whether the content exists before the browser touches it: - **A job board that loads more roles when you click "show more"** usually needs Selenium or a similar tool, because those roles don't exist until the click happens. If that click only fires an API call though, you can make the call yourself. - **A static blog archive** needs BeautifulSoup, because every post title is already in the HTML. Running a browser here just makes the job slower and heavier for no benefit. If you need the browser route, the [Selenium web scraping guide](https://liveproxies.io/blog/selenium-web-scraping) covers driver setup, pagination and proxy authentication. ### BeautifulSoup vs Playwright Playwright is a newer browser automation tool than Selenium. Against BeautifulSoup though, the comparison is the same one as Selenium, because the question isn't which browser tool is nicer. It's whether you need a browser at all. Answer that first. If the data is in view source, neither browser tool is worth its cost. If it isn't, you need a [headless browser](https://liveproxies.io/blog/what-is-headless-browser), and Selenium or Playwright runs one. ### BeautifulSoup vs. Scrapy Scrapy is a crawling framework. It handles scheduling, concurrency, retries, pipelines and export. BeautifulSoup is a parser and nothing else, so the two aren't competitors. You can use both together. Scrapy comes with its own selector API, but nothing stops you from passing a response body to BeautifulSoup inside a spider callback when you want its more forgiving search API. A reasonable rule: reach for Scrapy once you're crawling thousands of pages across many sites and want that machinery handled for you. For the middle ground, the difference between a [web crawler](https://liveproxies.io/blog/what-is-web-crawler) and a scraper is worth understanding before you pick. ## How to Do Web Scraping Using BeautifulSoup in 10 Minutes In 10 minutes you go from an empty folder to a CSV of 20 products, in 3 steps: install, fetch and parse, then write the file. The target is _books.toscrape.com_, a sandbox built for this purpose whose own banner reads "We love being scraped!". It has pagination, detail pages and spec tables, which is everything the step-by-step build needs. ### Quick setup Create a virtual environment and install the packages: ``` python3 -m venv .venv # Windows: py -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install beautifulsoup4 requests lxml html5lib pandas ``` _requests_ fetches the page and _beautifulsoup4_ gives you the tree to search. _lxml_ is the one worth explaining: it's a separate parser that BeautifulSoup can use, faster than the built-in one and, more importantly, it handles broken markup close to the way a browser does. For a wider look at the options, see how [data parsing](https://liveproxies.io/blog/what-is-data-parsing) works. _html5lib_ is a third parser, installed here only because later sections compare it against the other 2. _pandas_ is used once, much later, for the validation report. Confirm what you actually installed, because version drift is a big reason why old tutorials stop working: ``` import bs4, requests, lxml.etree print(bs4.__version__, requests.__version__, lxml.etree.__version__) ``` Yours will differ, and that's the point of checking: ``` 4.15.0 2.34.2 6.1.1 ``` The code here runs on beautifulsoup4 4.15.0, soupsieve 2.9, requests 2.34.2, lxml 6.1.1, html5lib 1.1, pandas 3.0.3 and Python 3.14.5, except where a section deliberately runs an older version to show what changed. The measurements are ours and the proxy and geo-targeting tests ran through Live Proxies [residential exits](https://liveproxies.io/products/rotating-residential). Keep everything from here in one file, adding to the same script rather than replacing it. The _parse_card_ function you write in a moment is the one the crawler calls on every page. Not everything belongs in that file though. The short blocks that show a warning or an error in a comment are demonstrations to run on their own, and a few others parse a snippet of markup written inline to make one point. **Troubleshooting the install.** If you see _bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml_, you asked BeautifulSoup for a parser you didn't install. Run _pip install lxml_. On macOS, an _SSL: CERTIFICATE_VERIFY_FAILED_ error often means you never ran _Install Certificates.command_ from your Python install folder. ### Minimal working scraper A working scraper is one fetch, one parse and one loop: ``` import requests from bs4 import BeautifulSoup URL = "https://books.toscrape.com/" response = requests.get(URL, timeout=10) response.raise_for_status() # Pass .content (bytes), not .text. This site sends no charset in the # Content-Type header, so requests would fall back to ISO-8859-1 and the # pound sign would come out as "Â£". BeautifulSoup reads the meta tag instead. soup = BeautifulSoup(response.content, "lxml") for card in soup.select("article.product_pod")[:5]: link = card.select_one("h3 a") print({ # the visible text is truncated ("A Light in the ..."), # the full title lives in the title attribute "title": link["title"], "price": card.select_one("p.price_color").get_text(strip=True), "url": link["href"], }) ``` That prints 5 dictionaries, one per card: ``` {'title': 'A Light in the Attic', 'price': '£51.77', 'url': 'catalogue/a-light-in-the-attic_1000/index.html'} {'title': 'Tipping the Velvet', 'price': '£53.74', 'url': 'catalogue/tipping-the-velvet_999/index.html'} {'title': 'Soumission', 'price': '£50.10', 'url': 'catalogue/soumission_998/index.html'} {'title': 'Sharp Objects', 'price': '£47.82', 'url': 'catalogue/sharp-objects_997/index.html'} {'title': 'Sapiens: A Brief History of Humankind', 'price': '£54.23', 'url': 'catalogue/sapiens-a-brief-history-of-humankind_996/index.html'} ``` In that output, 2 things aren't ready to use. The URLs are relative, and the prices are strings. ### Export to CSV Exporting is _csv.DictWriter_ over a list of dictionaries, and it's the point where the rows change shape in 3 ways. URLs resolve to absolute, the price becomes a number with its currency stored beside it, and every row records where it came from and when. The card already carries a rating and a stock flag, so you get those at no extra cost. The code is reorganised too. Extraction moves into a _parse_card()_ function, because the crawler reuses it unchanged, and the loop now covers all 20 cards rather than the first 5: ``` import csv import re from datetime import datetime, timezone from decimal import Decimal from urllib.parse import urljoin import requests from bs4 import BeautifulSoup URL = "https://books.toscrape.com/" WORDS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5} CURRENCY = {"£": "GBP", "$": "USD", "€": "EUR"} def parse_card(card, source_url): link = card.select_one("h3 a") price_text = card.select_one("p.price_color").get_text(strip=True) # Every price on this page reads "£51.77", so the symbol is one character # and the rest is the number. Store the currency beside the amount rather # than putting it in the column name. Real sites write prices in formats # that break this split, which parse_price handles. symbol, amount = price_text[0], price_text[1:] price, currency = Decimal(amount), CURRENCY.get(symbol, symbol) # "star-rating Three" -> 3. The rating is in the class, not the text. classes = card.select_one("p.star-rating")["class"] rating = next((WORDS[c] for c in classes if c in WORDS), None) return { "title": link["title"], "price": price, "currency": currency, "rating": rating, "in_stock": "In stock" in card.select_one("p.availability").get_text(), "url": urljoin(source_url, link["href"]), "source_url": source_url, "scraped_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), } response = requests.get(URL, timeout=10) response.raise_for_status() soup = BeautifulSoup(response.content, "lxml") rows = [parse_card(c, URL) for c in soup.select("article.product_pod")] with open("books.csv", "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=rows[0].keys()) writer.writeheader() writer.writerows(rows) print(f"wrote {len(rows)} rows to books.csv") ``` The file has a header row and 20 records. Here are the first 2: ``` wrote 20 rows to books.csv title,price,currency,rating,in_stock,url,source_url,scraped_at A Light in the Attic,51.77,GBP,3,True,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html,https://books.toscrape.com/,2026-07-19T02:13:31+00:00 Tipping the Velvet,53.74,GBP,1,True,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html,https://books.toscrape.com/,2026-07-19T02:13:31+00:00 ``` There is one more encoding trap on the way out. Excel on Windows can read a plain _utf-8_ CSV as cp1252, depending on version and locale. When it does, a price like _€40,96_ or a name like _Café_ comes back as _â‚¬40,96_ and _CafÃ©_, the same _Â£_ mojibake from the other direction. If a person will open the file in Excel, write it with _encoding="utf-8-sig"_, which adds a byte-order mark that Excel recognises. If a program will read it, keep plain _utf-8_, because that mark can confuse a naive parser. Most of the sandbox titles are ASCII, but 9 of the 1,000 aren't: one quotes the poem _Pangur Bán_, another names a _Café_, and 7 carry a curly apostrophe. So the trap is live even here, and the moment your rows carry a _£_, a _ç_ or a _ü_, that one argument is the difference between clean data and a column of garbage. ### Common quick start errors Just 7 errors account for the bulk of failed first runs. | **Symptom** | **Cause** | **Fix** | | --- | --- | --- | | _AttributeError: 'NoneType' object has no attribute 'get_text'_ | _find()_ matched nothing and returned _None_ | Check for _None_ before chaining | | Empty list, no error | _find_all()_ matched nothing and returned _[]_ | Print the length before you loop | | _£_ shows as _Â£_ | Passed _response.text_ when the server sent no charset | Pass _response.content_ | | _bs4.FeatureNotFound_ | Asked for _lxml_ without installing it | _pip install lxml_ | | Script hangs forever | No timeout on the request | Always pass _timeout=_ | | Works locally, wrong data on a colleague's machine | No parser specified, so each machine picked its own | Always pass a parser explicitly | | _DeprecationWarning: Call to deprecated method findAll_ | Copied a pre-2012 tutorial | Use _find_all_ | That last row deserves a warning. BeautifulSoup 4.15.0, is the release its own [changelog](https://git.launchpad.net/beautifulsoup/plain/CHANGELOG) calls the last to tolerate the old camelCase API. The changelog is unusually direct about what happens next: > In a subsequent point release, the DeprecationWarning issued when you use these obsolete features will be replaced by NotImplementedError, giving you a final chance to change your code before the implementations are removed entirely. Once the features are removed, code that tries to use them will start behaving strangely, since Beautiful Soup will generally interpret the method and attribute names as tag names. So _soup.findAll_ will eventually be read as "find a tag called _findAll_", which gives you _None_, so calling it raises _TypeError: 'NoneType' object is not callable_ and your error message never mentions _findAll_ at all. Affected names include _findAll_, _findNext_, _findParent_, _replaceWith_, _nextSibling_ and _renderContents_. Rename them now while the warnings still tell you where they are. ## How to Inspect a Page Before Web Scraping Using BeautifulSoup Open DevTools, with F12 on Windows and Linux or Cmd+Option+I on a Mac, click the element picker, and hover over one item in the list. View source told you whether the data is there at all, and DevTools is for the next question: what to select. You're looking for the smallest repeating container that holds one complete record. On the target page that container is _article.product_pod_, and there are 20 of them. Find the repeating block first, then extract fields inside it. Scraping all prices and all titles as 2 separate lists works until one item is missing a price. After that, every row is off by one. Sometimes no container holds the whole record. Hacker News puts a story's title in one table row and its score and comment count in the next, as siblings, so the repeating block you find contains only half a record. Table layouts do this often, and so do _dt_ and _dd_ pairs. When it happens, anchor on the half that always exists, then step sideways to the rest with _find_next_sibling()_. On markup shaped like that story list, the pattern looks like this: ``` html = """"""

120 points

""" row = BeautifulSoup(html, "lxml").select_one("tr.athing") # the half that always exists meta = row.find_next_sibling("tr") # the half that might not if meta and not meta.select_one("td.subtext"): meta = None # that row is the next story, not this one's metadata row.select_one("td.title a").get_text(strip=True) # 'First story' meta.select_one("span.score").get_text(strip=True) if meta else None # '120 points' ``` The anchor row carries the title, so it's the one to loop over. That _td.subtext_ check is the safeguard: _find_next_sibling("tr")_ returns the next row whatever it holds, so on a story with no score it gives you the next story's title row, and reading a score out of that raises _AttributeError_ on _None_. Confirm the sibling is the row you meant before you read it. ### Find stable selectors A selector tends to survive a redesign when it describes what an element _is_ rather than where it currently sits or how it currently looks. Prefer these in order, most durable first: 1. **Semantic containers**: _article_, _main_, _nav_ 2. **IDs and data attributes**: _#product_description_, _[data-testid="price"]_ 3. **Single meaningful classes**: _.price_color_ 4. **Structural position**, only as a last resort: _div > div:nth-child(3)_ Avoid long chains of generated class names like _.css-1x7f2p9_. Those can change whenever the site rebuilds its stylesheet. When plain CSS can't express what you need, the answer isn't XPath, which BeautifulSoup doesn't support at all. A _find()_ with a small function does what a selector can't, because it runs your own Python on each tag: _soup.find(lambda t: t.name == 'tr' and t.th and t.th.get_text(strip=True) == 'Availability')_ picks the row whose first cell reads Availability. _:has()_ and the soupsieve extension _:-soup-contains()_ reach the same row without leaving _select()_. If you do reach for structural position, check which soupsieve you have. BeautifulSoup delegates every _select()_ call to that library, and version 2.9 fixed _:nth-child(An+B)_ patterns that had been matching incorrectly. Running the same selectors against a list of 8 items on each version: ``` selector soupsieve 2.8.4 soupsieve 2.9 li:nth-child(3) ['3'] ['3'] li:nth-child(2n+1) ['1','3','5','7'] ['1','3','5','7'] li:nth-child(2n-2) [] ['2','4','6','8'] li:nth-child(n-1) [] all eight ``` The old version didn't raise on those selectors but returned an empty list, which is the same signal you get from a selector that's simply wrong. Plain positions like _:nth-child(3)_ were never affected. Formulas with a positive step were, though, whenever the list was short enough that the sequence landed on its last item. That comparison uses 8 items, which is why _2n+1_ looks safe in it. On the old version, the same _2n+1_ came back empty against a list of 1, and _:nth-child(n+5)_ returned nothing against a list of 5. Check with _python -c "import soupsieve; print(soupsieve.__version__)"_ before you lose hours debugging a selector that was right all along. Watch for 2 selector traps: First, copying a selector from your browser's dev tools often gives you _table > tbody > tr_. Browsers insert a __ element that isn't in the source HTML. Most parsers don't: ``` table = "

" for parser in ["html.parser", "lxml", "html5lib"]: soup = BeautifulSoup(table, parser) copied = soup.select("table > tbody > tr") # what DevTools hands you works = soup.select("table tr") # what you should write print(f"{parser:<12} copied={len(copied)} works={len(works)}") ``` The markup has 2 rows, so _works=2_ is the right answer everywhere: ``` html.parser copied=0 works=2 lxml copied=0 works=2 html5lib copied=2 works=2 ``` html5lib is the exception because it synthesises the __ a browser would add. Write _table tr_ instead. Second, _class_=_ matches the full class string exactly, in order: ``` btn = BeautifulSoup('Go', "lxml") btn.find_all(class_="btn btn-primary lg") # 1 match btn.find_all(class_="btn-primary btn lg") # 0 matches, same classes reordered btn.find_all(class_="btn") # 1 match, single class works btn.select(".btn.btn-primary") # 1 match, order independent ``` When you need to match several classes at once, CSS selectors are the safer choice. ### Confirm content is not dynamic Fetch your own target and count what your selector matches. If it returns 0, don't reach for a browser yet. Look at the script tags first, because many client-rendered pages ship their data as JSON right there in the HTML. The browser shows 10 quotes. The HTML that _requests_ receives contains none of them: the surrounding markup ends at line 26, and the whole dataset sits in a script tag just below it. ![quotes-to-scrape](https://strapi.liveproxies.io/uploads/quotes_to_scrape_bc9f3211ee.png) That _var data_ is the whole dataset, already in JSON, sitting in the HTML you were about to give up on. So the fix is to find the script tag and parse it, not to start a browser: ``` import json, re import requests from bs4 import BeautifulSoup response = requests.get("https://quotes.toscrape.com/js/", timeout=10) soup = BeautifulSoup(response.content, "lxml") match = None for script in soup.find_all("script"): body = script.string or "" if "var data" in body: match = re.search(r"var data\s*=\s*(\[.*?\]);", body, re.S) break # Check before you chain. Without this, a page that changed shape raises # AttributeError on None rather than telling you the block is gone. if match is None: raise SystemExit("no 'var data' block on this page") quotes = json.loads(match.group(1)) print(f"parsed {len(quotes)} quotes straight out of the script tag") for quote in quotes[:3]: print(f" {quote['author']['name']:<20} {quote['text'][:52]}...") ``` That's 10 quotes, none of which were ever in an HTML element: ``` parsed 10 quotes straight out of the script tag Albert Einstein “The world as we have created it is a process of our... J.K. Rowling “It is our choices, Harry, that show what we truly a... Albert Einstein “There are only two ways to live your life. One is a... ``` The first place to look is the markup itself: _", "lxml") noisy.get_text() # 'Real' # the same markup under html5lib, which does not get this protection: BeautifulSoup("

Real

", "html5lib").get_text() # 'Realvar x=1;' ``` That protection comes from storing the contents as _Script_ and _Stylesheet_ objects, which the html5lib tree builder doesn't do. It also still matters if you walk the tree yourself. ### Extract attributes like links and images Text is often the wrong field. Look at one card from the target site: ```

...

A Light in the ...

``` That small block of markup already contradicts 3 assumptions that people bring to a first scrape: 1. The **rating** is a word in the class attribute, not text anywhere on the page. 2. The **visible title can be truncated**. _get_text()_ gives you _"A Light in the ..."_ here, and half the cards on page 1 are cut the same way. The _title_ attribute has the real one. 3. The **href is relative**, so it's useless on its own. Reading those 3 from the card looks like this: ``` link = card.select_one("h3 a") link["title"] # 'A Light in the Attic' link.get_text(strip=True) # 'A Light in the ...' <- truncated link["href"] # 'catalogue/a-light-in-the-attic_1000/index.html' card.select_one("p.star-rating")["class"] # ['star-rating', 'Three'] ``` Always resolve links with _urljoin_ against the page you found them on: ``` from urllib.parse import urljoin urljoin("https://books.toscrape.com/", link["href"]) # https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html ``` This isn't cosmetic. On this site, page 1 links look like _catalogue/the-book_1/index.html_ while page 2 links look like _the-book_2/index.html_, because the pages sit at different depths. Hardcoding a base URL works on page 1 and produces 404s from page 2 onward. _urljoin_ handles both. ### Handle missing elements safely One card missing one field shouldn't kill the run. Use _.get()_ for attributes and check before chaining: ``` def safe_field(card, selector, attr=None, default=None): node = card.select_one(selector) if node is None: return default return node.get(attr, default) if attr else node.get_text(strip=True) # The same pattern on 3 fields. In parse_card, wrap the reads this way # rather than replacing the keys it already returns: row = { "title": safe_field(card, "h3 a", attr="title"), "price": safe_field(card, "p.price_color"), "image": safe_field(card, "img", attr="src"), } ``` Store _None_ rather than skipping the row. A field you recorded as missing shows up in the missing rate, and a jump in that rate is how you detect a broken selector. A row you never wrote is invisible, so a missing-rate check has nothing to count. ## How to Scrape Multiple Pages with BeautifulSoup Pagination Just 3 patterns cover most pagination you'll meet: query parameters (_?page=2_), path segments (_/page/2_), and a "next" link you follow. Following the next link is the one that doesn't need you to guess how many pages exist. ### Build page URLs safely Before you write any pagination loop, check _/sitemap.xml_ and the _Sitemap:_ line in _robots.txt_. Plenty of sites publish every URL you want in one file, which can turn a fragile multi-page crawl into one fetch and a list of links. When there's no sitemap and the URLs are predictable, generate them, but always cap the range: ``` urls = [f"https://books.toscrape.com/catalogue/page-{n}.html" for n in range(1, 51)] ``` Record the list URL on every row, which _parse_card_ already does as _source_url_. When something looks wrong later, that column is what tells you which page it came from. ### Find and follow next links When the URLs aren't predictable, let the site tell you where to go next by reading its own next link: ``` next_link = soup.select_one("li.next a") if next_link: url = urljoin(url, next_link["href"]) ``` When the last page is reached, the site drops the next element, so _select_one_ returns _None_ and the _if_ never fires. Without that check, reading _next_link["href"]_ would raise a _TypeError_ instead. ### Deduplicate items across pages Dedupe on something stable, like the item URL or an ID, rather than on a title, because 2 different books can share a name: ``` seen, unique = set(), [] for row in rows: if row["url"] in seen: continue seen.add(row["url"]) unique.append(row) ``` Dedupe as you go rather than at the end, so a page that repeats itself doesn't quietly double your row count first. ### Stop rules that prevent runaway crawls A crawler with no stop condition runs until something else stops it, and that something is often the site you're crawling. Set all of the following, because they fail independently and any one of them can be the only thing that catches a bug: 1. **Max pages.** A hard ceiling, so a bug can't become a thousand requests. 2. **No new items.** If a page only returns items you already have, you're probably looping. 3. **No next link.** The natural end. Here's the crawler with all 3: ``` # Continues the scraper, reusing parse_card and the fetch() retry helper. START = "https://books.toscrape.com/" def crawl(start=START, max_pages=3): rows, seen = [], set() url, page = start, 0 with requests.Session() as session: while url and page < max_pages: # stop rule 1 page += 1 # fetch() carries the retries; a transient 503 no longer aborts the crawl. soup = BeautifulSoup(fetch(session, url).content, "lxml") cards = soup.select("article.product_pod") if not cards: # 0 cards is a block or a renamed class, not an empty page. # Without this, stop rule 2 reads it as a clean finish. raise RuntimeError( f"no cards on {url}, title={soup.title and soup.title.get_text(strip=True)!r}" ) new = 0 for card in cards: row = parse_card(card, url) if row["url"] in seen: # stable key, not the title continue seen.add(row["url"]) rows.append(row) new += 1 print(f" page {page}: {len(rows):>3} total, {new:>2} new") if new == 0: # stop rule 2 break next_link = soup.select_one("li.next a") if not next_link: # stop rule 3 break url = urljoin(url, next_link["href"]) time.sleep( 1 + random.uniform(0, 0.5) ) # a fixed delay is a machine-perfect rhythm return rows rows = crawl() ``` It stops after the third page because _max_pages_ says so, not because the site ran out of pages: ``` page 1: 20 total, 20 new page 2: 40 total, 20 new page 3: 60 total, 20 new ``` One thing this version doesn't do is survive its own crash. It holds every row in a list and only writes at the end, so a network error on page 40,000 loses the 39,999 pages before it. On a long crawl, write each page's rows straight to disk inside the loop instead of holding them in a list: open the CSV in append mode and _writerows_ after each page. Call _writeheader_ only when the file is new, or a rerun writes a second header row into the middle of your data. A _.jsonl_ file avoids that entirely, because one _json.dumps(row, default=str)_ line per record has no header to repeat, which is why it's the easier format to append to and resume. The _default=str_ is not optional, since _json_ refuses to serialise the _Decimal_ that _parse_price_ returns. Persist which pages you've finished too, so a rerun skips them instead of starting over. The in-memory dedupe set only prevents duplicates within one run, so a set that survives a restart has to live on disk. ## How to Build a Python Web Scraper Using BeautifulSoup for Item Detail Pages A 2-stage crawl costs 21 requests where the list page alone costs 1, and buys 3 fields the list page never carries. ### List page to detail page flow Stage 1 collects the links off the list page. Stage 2 visits each one in turn, reusing the same session so the connection isn't rebuilt every time. The loop stays readable with 2 helpers. _get_soup_ wraps the fetch and parse pair you've written several times already, and routes it through the _fetch()_ retry helper so every detail request inherits the backoff rather than aborting on the first transient error: ``` def get_soup(session, url): return BeautifulSoup(fetch(session, url).content, "lxml") ``` ### Parse key value specs Spec tables are often _th_/_td_ pairs. Normalise the keys so downstream code isn't dealing with _"Price (excl. tax)"_: ``` # soup here is one product's detail page: soup = get_soup(session, product_url) specs = {} for row in soup.select("table.table-striped tr"): if row.th is None or row.td is None: # header and spacer rows have neither continue # \w keeps non-ASCII letters, so a Cyrillic or Greek label still gets a key. # [^a-z0-9] would delete the whole label and collapse every row onto "". key = re.sub(r"[^\w]+", "_", row.th.get_text(strip=True).lower()).strip("_") specs[key] = row.td.get_text(strip=True) ``` _specs_ then holds the whole table, keyed for downstream code: ``` {'upc': 'a897fe39b1053632', 'product_type': 'Books', 'price_excl_tax': '£51.77', 'price_incl_tax': '£51.77', 'tax': '£0.00', 'availability': 'In stock (22 available)', 'number_of_reviews': '0'} ``` _pandas.read_html_ is the obvious alternative, and it's the right tool when a page carries a wide data table you want as a DataFrame anyway. It's less useful here. A 2-column table of keys and values comes back with unnamed columns, so you still combine the 2 halves and still normalise the keys. And reading it from _response.text_ rather than _io.BytesIO(response.content)_ reproduces the same mojibake shown earlier. Whichever route you take, keep the raw string alongside anything you parse from it. When _availability_ changes format, having the original means you can re-parse without re-scraping: ``` match = re.search(r"$(\d+) available$", specs.get("availability", "")) units = int(match.group(1)) if match else 0 ``` The list page only says "In stock", while the detail page says how many. ### Clean product descriptions The description on this site sits _after_ the _#product_description_ heading, not inside it: ``` heading = soup.select_one("#product_description") description = "" if heading: para = heading.find_next_sibling("p") if para: description = re.sub(r"\s+", " ", para.get_text(strip=True)) ``` _find_next_sibling()_ is the tool for "the thing after this thing". Then collapse runs of whitespace with _\s+_, and keep the raw HTML alongside it when the structure has downstream value. That's every piece of _parse_detail_ except 2. Here it's assembled, with the title read from the page's _h1_ and the category from the last link in the breadcrumb, the only place this site names it: ``` def parse_detail(soup, url, source_list_url): specs = {} for row in soup.select("table.table-striped tr"): if row.th is None or row.td is None: continue key = re.sub(r"[^\w]+", "_", row.th.get_text(strip=True).lower()).strip("_") specs[key] = row.td.get_text(strip=True) available = re.search(r"$(\d+) available$", specs.get("availability", "")) heading = soup.select_one("#product_description") para = heading.find_next_sibling("p") if heading else None return { "title": soup.select_one("h1").get_text(strip=True), "category": [a.get_text(strip=True) for a in soup.select("ul.breadcrumb a")][-1], "price": specs.get("price_incl_tax", ""), # still a string until parse_price replaces this "units_available": int(available.group(1)) if available else None, "description": re.sub(r"\s+", " ", para.get_text(strip=True)) if para else "", "url": url, "source_list_url": source_list_url, } ``` Turning that raw _"£51.77"_ string into something you can sort and sum is its own problem. With _parse_detail_ defined, the driver is short. The _[:4]_ keeps the demo to 4 requests: ``` with requests.Session() as session: soup = get_soup(session, START) urls = [urljoin(START, a["href"]) for a in soup.select("article.product_pod h3 a")][:4] print(f"stage 1: collected {len(urls)} product URLs from {START}\n") print("stage 2: visiting each product page") for url in urls: try: detail = parse_detail(get_soup(session, url), url, source_list_url=START) # parse_detail raises more than network errors: LookupError on a missing # breadcrumb, AttributeError on a missing h1. except (requests.RequestException, LookupError, AttributeError) as exc: print(f" skipped {url}: {type(exc).__name__}: {exc}") # one bad page must not kill the batch continue print(f" {detail['title'][:32]:<33}{detail['category']:<22}" f"{detail['price']:<8}{detail['units_available']} left " f"{detail['description']} chars") time.sleep(1 + random.uniform(0, 0.5)) ``` Over the first 4 products that prints: ``` stage 1: collected 4 product URLs from https://books.toscrape.com/ stage 2: visiting each product page A Light in the Attic Poetry £51.77 22 left 1017 chars Tipping the Velvet Historical Fiction £53.74 20 left 1029 chars Soumission Fiction £50.10 20 left 1093 chars Sharp Objects Mystery £47.82 20 left 1635 chars ``` Category, unit count and description length are all fields that the list page never carried. ### Handle variants and multiple offers Variants are often separate URLs rather than options inside one page. The tutorial site can't demonstrate that, so the example is a real commercial page. We fetched IKEA's BILLY bookcase on their UK site, checking _robots.txt_ first, which allows _/p/_ and disallows _/cart/_. The page is worth more than any invented example for 2 reasons. The first is that it publishes its own structured data, so you don't have to guess at selectors: ``` import json # soup here is a product page you've fetched, in this case IKEA's BILLY bookcase. for tag in soup.find_all("script", type="application/ld+json"): try: data = json.loads(tag.string or "") except json.JSONDecodeError: continue # one malformed block must not hide a valid one below it # A page can carry several of these blocks, and a block can be a list rather # than an object, so neither "the first one" nor "it is a dict" is safe. for node in (data if isinstance(data, list) else [data]): if isinstance(node, dict) and node.get("@type") == "Product": # offers is a list whenever a product has several sellers or sizes. # Taking the first is a choice, not a fix: it drops the rest # without saying so. Loop over the list when you need every offer. offer = node.get("offers") offer = offer[0] if isinstance(offer, list) else (offer or {}) print(node.get("sku"), node.get("color"), offer.get("price"), offer.get("priceCurrency")) ``` That page carries 2 blocks, a _BreadcrumbList_ and a _Product_, and only the second one matches: ``` 002.638.50 white 55 GBP ``` Look for _application/ld+json_ before you write a single CSS selector. Plenty of commercial pages carry a _Product_ block with SKU, price and currency already parsed, and it tends to survive redesigns that break selectors. The 2 guards matter: if you drop the list check, a site whose block is a JSON array gives you _AttributeError: 'list' object has no attribute 'get'_. At multi-site scale, _extruct_ reads JSON-LD, microdata, RDFa and OpenGraph in one call, with those shapes already handled. The second is how the variants are actually modelled. The colours aren't options inside one page. Each is **a separate URL with its own SKU**, linked from the others: ``` colour sku price currency size white 002.638.50 55 GBP 80 x 28 x 202 cm black 404.773.40 70 GBP 80 x 28 x 202 cm brown 505.086.52 70 GBP 80 x 28 x 202 cm beige, brown 105.089.32 70 GBP 80 x 28 x 202 cm ``` Identical dimensions, 4 SKUs, and on that day the white one was £15 cheaper than its siblings. If you scrape the URL you happened to open, you get "the price of a BILLY bookcase" when there isn't one, just the price of whichever colour you landed on. So the variant problem is often a crawling problem before it's a parsing one: find the sibling links, follow them, and key your rows on SKU rather than on product name. Real attribute values aren't tidy enumerations either, as _beige, brown_ in that colour column shows. Anything downstream that expects one word per colour has a problem on day one. Keep variants as their own rows or a separate table keyed by SKU. Flattening to one row per product with a single _price_ column can't answer "which colours were out of stock", and you can't reconstruct that information afterwards. ## How to Scrape Only Visible Webpage Text with BeautifulSoup For article scraping and summarisation you want readable text rather than navigation and footers, and you want it to keep some shape. The same navigation and footers on every page become tokens you pay for over and over, and a single flat block of text loses the structure that makes a long document usable later. The examples run against one product page, so fetch it into _soup_ first: ``` DETAIL = "https://books.toscrape.com/catalogue/sharp-objects_997/index.html" soup = BeautifulSoup(requests.get(DETAIL, headers=HEADERS, timeout=10).content, "lxml") ``` Asking for all of its text at once gives you everything on the page, navigation included: ``` 1. soup.get_text(separator="\n", strip=True) on the whole document 2,291 chars over 42 lines first 5 lines: ['Sharp Objects | Books to Scrape - Sandbox', 'Books to Scrape', 'We love being scraped!', 'Home', 'Books'] ``` The first 5 lines are the page title, the site name, a banner and 2 breadcrumbs. None of it's the content. ### Remove noise tags On lxml and _html.parser_, _get_text()_ already skips the text inside _script_ and _style_, but it leaves elements like _nav_ and _footer_ in the tree, and you rarely want them in extracted text: ``` for tag in soup(["noscript", "template", "svg", "nav", "footer"]): tag.decompose() ``` _decompose()_ destroys the element. _extract()_ removes it and returns it if you want it later. On this product page that loop changes nothing, because the only tag from the list it carries is an empty _

How To

BeautifulSoup Web Scraping: How to Build a Web Scraper With Python (Step-By-Step Tutorial)

Selenium Web Scraping With Python: Full 2026 Guide