If your web scraper keeps breaking every few weeks, you already know the pattern. I call it the Selector Cycle of Death. You spend an hour tuning CSS selectors, the short address strings that tell a script which part of a page to grab, until everything extracts perfectly. Then the site ships a minor frontend update, a renamed class or a restructured wrapper div, and your script quietly stops working. No error, no warning, just empty results.
The root problem is how most scrapers are designed. I tell the script exactly where to look instead of what to look for. Hard-coded selectors are brittle by nature, because they assume a page's structure will stay frozen in time. It rarely does.
Why a web scraper keeps breaking when the site looks unchanged
Class names like product-card or listing-row exist for styling. No site owner ever promised they would stay put. They are internal plumbing rather than a published interface, and a designer can rename all of them in an afternoon without changing a single thing a human visitor sees. Your scraper, meanwhile, sees a completely different document.
That is why the breakage feels random. The page looks identical in the browser and the script returns nothing.
Broken CSS selectors fail silently
The worse half of the problem is how the failure shows up, which is to say it doesn't. Ask the BeautifulSoup parsing documentation for every element matching div.product-card and, if nothing matches, you get an empty list back. Not an exception. Not a warning. An empty list, which the rest of the script then processes perfectly happily into an empty CSV.
A web scraper that keeps breaking loudly would at least be honest about it. Instead the scheduled job runs on time, the file lands exactly where it should, and the row count is zero. That gap between "ran successfully" and "produced anything" is what makes this failure mode so easy to miss.
A different approach: repeated topology
Lately I have been experimenting with what I am calling a Repeated Topology approach. Instead of targeting specific IDs or class names, the scraper looks for structural patterns that tend to show up regardless of how a site is styled.
Here is the insight. Whether you are looking at a news homepage, a real estate listings page, or an e-commerce category, the heart of the page is almost always a list of similar-looking items: articles, listings, or products repeated in a row. The visual design changes from site to site, but the underlying structure, a container with several children that all look alike, tends to stay consistent.
Names change constantly. Shape changes rarely.
The strategy: topology mapping
The approach breaks down into three steps.
Step one: kill the noise
Headers, footers, navigation bars, scripts, and other boilerplate get stripped out early. None of that is the main dish, so it goes before any analysis happens. This step matters more than it sounds, because a navigation menu is itself a list of similar items and will otherwise compete with the real content for attention.
Step two: find the echo
The script scans through the remaining HTML looking for containers whose children share a structural signature, for example five div elements in a row that all use the same tag and class combination. That repetition is a strong signal: it is very likely a product grid, article list, or news feed, even without knowing anything about the site's naming conventions.
Step three: auto-harvest
Once a repeating cluster is found, the script pulls out text, links, headings, and images from each item automatically, without needing to be told in advance which tags hold which type of content.
The result is not a scraper tied to one site's markup. It is a more general way of locating the pulse of a page based on its shape rather than its labels.
A working BeautifulSoup scraper
Here is a basic Python implementation using the requests HTTP library, which fetches the raw HTML, and BeautifulSoup, which parses that HTML into something you can search through:
python
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
def detailed_cluster_scrape(url):
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')
base_url = url
# 1. Cleanup Noise: Strip the fluff
for noise in soup(['script', 'style', 'nav', 'footer', 'header', 'svg', 'noscript']):
noise.decompose()
clusters = []
# 2. Topology Mapping: Find repeated patterns
for container in soup.find_all(['div', 'ul', 'section', 'ol']):
children = container.find_all(recursive=False)
if len(children) < 3:
continue
# Generate a "signature" for each child based on tag and class
child_signatures = [f"{c.name}.{'.'.join(c.get('class', []))}" for c in children]
# Heuristic: If most children look the same, we found a content cluster
if len(set(child_signatures)) <= 2:
item_blocks = []
for child in children:
block_data = {
"text": " ".join(child.get_text(separator=" ", strip=True).split()),
"links": [urljoin(base_url, a['href']) for a in child.find_all('a', href=True)],
"images": [urljoin(base_url, img['src']) for img in child.find_all('img', src=True)],
"headings": [h.get_text(strip=True) for h in child.find_all(['h1', 'h2', 'h3', 'h4'])]
}
if block_data["text"]:
item_blocks.append(block_data)
if item_blocks:
clusters.append({
"container": f"{container.name}.{'.'.join(container.get('class', []))}",
"items": item_blocks
})
# 3. Sort by the largest cluster (usually the main feed)
clusters.sort(key=lambda x: len(x['items']), reverse=True)
return clusters
What the script hands back
Each cluster comes back as a dictionary: the container it was found in, then a list of items, each carrying its cleaned-up text, absolute links, image URLs, and any headings. Clusters are sorted largest first, on the assumption that the biggest repeating group is the main feed.
That output is deliberately raw. Turning it into something a business can use, deduplicated rows in a CSV, a SQLite table, a Google Sheet that refreshes overnight, is a separate step, and in client work it is usually where most of the real effort sits.
Why this holds up better over time
The biggest advantage of this approach is resilience. A traditional scraper breaks the moment a site renames a class or restructures a wrapper element, because it is looking for that exact name. A topology-based scraper does not care about names at all. It cares about repetition and shape. As long as a site keeps presenting its main content as a list of similarly structured items, which is true for the overwhelming majority of content-driven sites, the scraper keeps working through cosmetic redesigns.
It is also a useful starting point for exploring an unfamiliar site. Instead of opening dev tools and manually hunting for the right selectors, I run this kind of cluster detection first to surface where the real content lives, then refine from there if I need more specific fields.
Where it falls short
This is not a cure-all, and I would rather you know the failure modes before building anything on top of it.
False positives on other repeated blocks
The heuristic, meaning a rule of thumb that is usually right rather than a guarantee, can produce false positives on pages with other repeated structures. Sidebars full of similar-looking widgets, related-post strips, and stacked ad blocks all share the same shape as a genuine content feed. Sorting by cluster size helps, since the main feed is usually the longest, but it is not foolproof. In practice I add a filter or two: a minimum text length per item, or a rule that every item must contain at least one link.
JavaScript-rendered content needs a browser
This will not handle JavaScript-rendered content out of the box. Requests only fetches the HTML the server sends, and on plenty of modern sites that first response is close to empty, with the feed assembled in the browser afterwards. Those sites need a headless browser, meaning a real browser engine running without a visible window, such as the Playwright library for Python or Selenium, to render the page first. The cluster analysis then runs on the rendered HTML instead of the empty shell. I write more about that trade-off across my other write-ups on scraping and automation.
Scraping responsibly
Scraping should be done responsibly. Check a site's robots.txt, the file at the root of a domain that states which paths automated clients are asked to leave alone, along with its terms of service. Google's introduction to robots.txt explains how the file is structured. Avoid hammering servers with rapid requests, and respect any rate limits or access restrictions in place. Whether a particular site is appropriate to scrape comes down to those terms and your own judgement, not something I decide on your behalf.
Cutting scraper maintenance down to a quick check
Most of the frustration in this work comes from scrapers built on details a site owner never promised to keep stable. Shifting the question from "where is this data?" to "what does this data look like structurally?" produces something noticeably more durable. Not perfect, but a lot less likely to need a fix every time a site gets a facelift.
Scraper maintenance never drops to zero. It can drop to a monthly glance at a row count.
If your web scraper keeps breaking and you would rather not be the one debugging it at eleven at night, that is the work I do: custom scrapers, browser automation, and the cleaning and scheduling that turns raw output into a file someone can actually open. You can see how I scope and run projects, or start a custom scraper build.