Web scraping done properly: what's realistic, reliable and within the rules

What scraping is good at — and what it is not

Web scraping means having software read public web pages and pull out structured data — prices, listings, contact details, specifications — and hand it back as a clean spreadsheet or feed. It is genuinely useful for research, monitoring competitors’ public prices, gathering public records, and turning scattered information into something you can sort and analyse.

It is the wrong tool for anything behind a login you are not allowed past, anything explicitly forbidden by a site, and any personal data you have no lawful reason to collect. Just because a page loads does not mean the data on it is yours to take.

The part that breaks: sites change

A scraper written today reads the shape of a page as it looks today. When the site is redesigned — and it will be — the scraper quietly starts returning nothing, or worse, the wrong thing. This is the real cost of scraping, and it is why a weekend script and a dependable data feed are not the same product.

A build that lasts assumes the source will change. It checks that what it collected looks sane, it flags when a source goes quiet instead of failing in silence, and it is written so that fixing a broken source is a small job rather than a rewrite.

Staying on the right side of the line

Responsible data collection is not just about what is technically possible. A build done properly respects some clear boundaries:

• Public data only. Information anyone can see without logging in or defeating a barrier — not private accounts, not paywalled content.
• Read the site’s rules. Its robots.txt and terms of use say what is welcome. Where they say no, that is the answer.
• Be gentle. Sensible rate limits so the source is never hammered. Good scraping is close to invisible; aggressive scraping is a nuisance and gets you blocked for a reason.
• Personal data is different. Names, emails and anything identifying a person fall under data-protection law such as UK GDPR. Collecting it needs a lawful basis, not just the ability to grab it.

If a project needs data that these boundaries rule out, the honest answer is to say so and find a licensed source or an official API instead.

Getting clean data out

Raw scraped data is messy: inconsistent formats, duplicates, half-empty rows, three different ways of writing the same date. The value is in the cleaning, not the grabbing. A proper job normalises the fields, removes duplicates, fills or flags the gaps, and delivers the result in whatever your team actually uses — CSV, a spreadsheet, a database table or a scheduled feed.

Delivered well, you get data you can trust and use immediately, not a pile you still have to tidy by hand.

When an API beats a scraper

If a site offers an official API — a supported way to request its data directly — that is almost always the better route. It is stable, it does not break on a redesign, and it is explicitly permitted. A good developer checks for an API first and only reaches for scraping when there is no supported alternative and the data is genuinely public.

The goal is reliable data you are allowed to have, by the most durable route available — not cleverness for its own sake.