Algorithmic impact assessment: a case study in healthcare
From automated diagnostics to personalised medicine, the healthcare sector has seen a surge in the application of data-driven technologies (including AI) to deliver health outcomes. To increase the rate of innovation in this space, public-health agencies have sought to make health data they control more accessible to researchers and private sector firms.
But while data-driven technologies have the potential to bring enormous benefits to healthcare, they also come with serious risk of harm to the people, the environment and society, including perpetuating algorithmic bias, impeding transparency and public scrutiny and creating risks to individual privacy.
There is therefore a pressing need to understand and mitigate the potential impacts of AI and data-driven systems in healthcare before they are developed and deployed, and to address serious risks, including, but not limited to:
- the perpetuation of ‘algorithmic bias’, exacerbating health inequalities by replicating entrenched social biases and racism in existing systems;
- inaccessible language or lack of transparent explanations can make it hard for clinicians, patients and the public to understand the technologies undermining public scrutiny and accountability;
- the collection of personal data, tracking and the normalisation of surveillance, creating risks to individual privacy.
Understanding the potential impacts of data-driven systems before they are developed is the best way to mitigate these potential harms. While there is a growing academic literature around how public-sector agencies can conduct algorithmic impact assessments (AIAs),1 there remains a lack of case studies of these frameworks in practice, particularly in a healthcare setting.
Building on Ada’s existing work on assessing algorithmic systems, this project identifies a case-specific impact assessment model for use in the NHS AI Lab‘s National Medical Imaging Platform (NMIP), a proposed large-scale dataset of high-quality chest X-rays (CXRs), MRIs, skin, ophthalmology and other images, made available to researchers and private companies to test, train and validate medical AI products.
The NHS is set to trial the use of this assessment as part of the work of the NHS AI Lab. The framework will be used in a pilot to support researchers and developers in assessing the possible risks of an algorithmic system before they are granted access to NHS patient data. By implementing this research, the NHS in England is set to be the first health system in the world to use this new approach to the ethical use of AI.
We have developed actionable steps on how to implement the NMIP AIA for the NHS AI Lab, as well as helped to inform wider AIA research and practice at the intersection of health and data science, and in the public and private sectors.
The NMIP provides a novel case study as it sits at the intersection of two audiences for impact assessment: industry developers and the public sector. Existing research has tended to focus on either public sector procurers of systems (such as the Canadian AIA), or the developers of new technology themselves (such as use of human rights impact assessments in industry). This case study offers a new lens through which to examine and develop algorithmic impact assessment at the intersection of public sector and industry development.
As the outputs of this project, we have built a detailed, step-by-step algorithmic impact assessment model for the NHS AI Lab to use as a requirement of access to the NMIP, including the artefact of the AIA, the NMIP algorithmic impact assessment template. To support the implementation of this AIA, we also designed an end-to-end walkthrough of the process from the perspective of an NMIP applicant, the NMIP algorithmic impact assessment user guide.
In our report Algorithmic impact assessment: a case study in healthcare we introduce the AIA process, a synthesis of the algorithmic impact assessment literature and the healthcare AI context and ‘Seven operational questions’ that policymakers and researchers interested in building, adopting or mandating AIAs to consider. This study provides a missing piece of the puzzle in practical evidence on the use of algorithmic impact assessments.
This research is being funded by a grant of £66,000 from NHS AI Lab and is governed by a memorandum of understanding (MOU), which is available here.
Further work and opportunities
Image credit: poba
This report sets out the first-known detailed proposal for the use of an algorithmic impact assessment for data access in a healthcare context.
Identifying common language for algorithm audits and impact assessments