Dataharvest 2024 - the European Investigative Journalism Conference

Scraped data can often be the backbone of an investigation, but some websites are more difficult to scrape than others. This could be because of the sheer volume of data you need, or the way the site is built - either accidentally or deliberately making it hard to scrape. This session will cover best practices for dealing with tricky sites, including coping with captchas, using proxy and other scraping services, and best ways to scale up your scraping by using the cloud. This is an advanced session aimed at people who already have experience of writing code to scrape websites and want to move up to the next level: participants will leave with an understanding of how to approach hard-to-scrape websites, plus the tradeoffs and costs.