Web scraping and data extraction.

Scraping is straightforward to start and difficult to keep running. The site changes its markup, adds rate limiting, moves content behind JavaScript, or blocks your requests entirely. A scraper that worked last month and silently returns empty results this month is worse than no scraper, because you may not notice. I build these to detect their own failure and tell you.

What’s included

• Structured extraction into JSON, CSV or a database
• Scheduled monitoring and change detection
• JavaScript-rendered sites via headless browser where needed
• Rate limiting and polite crawling that will not get you blocked
• Validation that alerts when output looks wrong or empty
• De-duplication and normalisation across sources
• Aggregation from many sources into one clean dataset

Building for change, not for today

Selectors break. I structure extraction so the parsing rules are separate from the plumbing, which makes a site redesign a configuration fix rather than a rewrite. I also validate output shape on every run, so a change that produces empty or malformed results raises an alert instead of quietly poisoning your dataset.

Being a good citizen

Aggressive scraping gets you blocked and creates problems for the site you are pulling from. I rate limit, identify the crawler honestly, respect robots directives, and cache so the same page is not fetched repeatedly. This is also the pragmatic choice — polite scrapers survive much longer.

A worked example

I built a news aggregation engine that pulls around a hundred RSS feeds from international publishers on a schedule, normalises and de-duplicates them, and serves seven categories from a flat-file cache with no database. It runs on an hourly cycle and self-heals if the cache goes stale.