Evaluation of foundation models

Project background

Foundation models are a form of artificial intelligence (AI) designed to produce a wide variety of outputs. They are capable of a range of possible tasks and applications such as text synthesis, image manipulation or audio generation. These models can be standalone systems or can be used as a ‘base’ model to build and ‘fine tune’ other applications. Notable examples are OpenAI’s GPT-3 and GPT-4, which underpin the conversational tool ChatGPT, and image generators like Midjourney. These recent developments have led to a surge in policy and media attention on foundation models.

To address the risks and maximise the benefits of foundation models, regulatory and governance approaches are being developed in the private sector (e.g. OpenAI’s systems card), civil society (e.g. Partnership on AI’s protocols for foundation models) and public sector (e.g. the UK’s AI Safety Institute) .

A major focus of these governance approaches has been on the evaluation and testing of models through methods like red-teaming, piloting and auditing. These approaches are still emerging and are largely being led by private-sector labs. Some of these approaches may raise serious ethical and legal questions, such as the ethics of making an AI system accessible to the public to refine/train products like ChatGPT.

As technology companies seek to deploy, test and evaluate powerful AI technologies, it is critical for policymakers and regulators to understand what kinds of evaluation methods exist, and the potential limitations of these evaluation practices.

Project overview

This project aims to provide policymakers with evidence to support foundation model governance. This includes where different evaluation obligations should sit in the supply chain of foundation models, the limitations of evaluations, and an exploration of potential consequences of evaluation.

It will draw on literature on auditing and evaluation of foundation models, and interviews with developers and designers of AI-powered systems.

The first output of this work will be an explainer on the meaning of evaluation and related concepts (such as audits, red-teaming and impact assessments) in the context of AI and foundation models in particular.

This will be followed by a paper summarising:

• the current state of foundation model evaluations
• theoretical and practical limitations of those evaluations
• what evaluations can tell policymakers
• actions to take on the basis of evaluation results.

This project is supported via UKRI by the DCMS Science and Analysis R&D Programme. It was developed and produced according to UKRI’s initial hypotheses and output requests. Any primary research, subsequent findings or recommendations do not represent DCMS views or policy and are produced according to academic ethics, quality assurance and independence.

Image credit: FG Trade