Dataharvest 2024 - the European Investigative Journalism Conference

Despite the increasing deployment of risk assessment algorithms, journalists have struggled to access and test these systems. In March 2023, a team of journalists managed to access the source code, machine learning model file, and training data of a machine learning welfare fraud algorithm deployed in the Dutch city of Rotterdam. This access enabled them to execute an experiment to test for bias and discrimination.

This hands-on, in-person workshop will have participants work directly with the Rotterdam algorithm in order to learn how to develop and test hypotheses on risk assessment algorithms. Unlike additive models, more complex non-linear ML models like those used in Rotterdam can create unpredictable effects that make them difficult to test. We have built a frontend app for the Rotterdam model that allows participants to test it without writing any code.

This workshop will have the following structure:
-How to investigate the full lifecycle of an algorithm (training data, model type, performance, fairness) and the questions you can ask with differing levels of technical access.
-How to develop a data-driven hypothesis for a non-linear machine learning model
-Executing hypothesis on frontend app and interpreting results.