Quality of Dutch Government - Project
Context: The Institute of the Quality of the Dutch Government (working title) aims to build a high- quality longitudinal database of the quality and performance of the Dutch government. For that purpose, it seeks to compile a wide range of sources of data, and translate them into easy-to-comprehend, reliable indicators of how well the government is performing on a series of important process and outcome aspects. There is a wealth of available data which could inform the database, collected regularly by various governmental and non- governmental organizations. Some of this data is in ready-to-use format: quantitative and longitudinal. However, there is also plenty of regularly collected potentially highly informative data that is presented in reports, usually taking a qualitative format.
Main question: How can qualitative reports on the performance of the government be reliably translated into quantitative indicators using LLMs?
Sub-questions: • What kind of indicators pertaining the quality and performance of the government (and its various organizations) could be extracted from the regularly published reports by the Dutch Court of Audit (external auditor) the Auditdienst Rijk (internal auditor) and/or Directie Internationaal Onderzoek en Beleidsevaluatie (IOB)? • What considerations must be taken into account in order to ensure the validity and reliability of the extracted quantitative indicators? • What are the opportunities to build a prototype to scale the extraction of data from the provided organizations, so that the extraction of data could be done continuously, and potentially using other qualitative data sources?
Objective: The aim of this research project is to explore the opportunities for translating qualitative data on the quality of the Dutch government into quantitative indicators. The goal is to develop a tool which would allow for the continuous extraction of data from regularly published reports.
Elaboration of the project: • Analysis of the possibilities: Familiarize yourselves with the Dutch Court of Audit, Auditdienst Rijk and/or IOB reports: what kinds of reports do they publish, about what government organizations, and what kind of information do they contain? This will provide the grounds for identifying what kinds of indicators would be possible to extract from the reports. The goal is to have systematic and longitudinal data collection process. • Reliability and validity considerations: The extracted quantitative data would be useful only if its quality is high. Here it is important to consider key measurement issues: how can the validity and reliability of the extracted data be maximized? • Build a tech demo: Build a tool that relies on an LLM, which could consistently extract quantitative data from qualitative reports, that builds on the analysis performed in the previous two steps. • Critical reflection on risks and opportunities: Analyze the performance of the tool in terms of the quality of the data it produces. Offer a critical reflection on the possibilities provided by the tool, but also its limitations.
Deliverables: • A tech demo of a natural language system (e.g. LLM) which could be used to extract quantitative indicators from published governmental and non- governmental reports. • A report containing: o Critical discussion on the opportunities and obstacles for using LLMs for the outlined task. o Discussion on how the validity and reliability of the extracted data could be ensured using LLMs as an extraction tool. o Recommendations on when such tool could be best employed for the given task, and when the risks outweigh the benefits.