Warwickshire County Council: “Access to Food” – exploring LLMs for citizen needs

Warwickshire County Council's Access to Food project uses large language models to understand citizens' access to healthy food options.

Cyber

Artificial Intelligence

03 Feb 2025

Overview

Warwickshire Council has been a pioneer in applying artificial intelligence (AI) within local government. Their journey began in 2018 with a project utilising convolutional neural networks to analyse video data for traffic management. This initiative, alongside subsequent projects like email clustering using machine learning, demonstrated the value of AI for gleaning insights and improving service delivery.

Warwickshire's latest project, "Access to Food," tackles a pressing issue in the face of the rising cost of living. It aims to understand citizens' access to healthy food options, particularly in deprived areas where supermarkets might not be accessible. The project goes beyond simply identifying shops; it seeks to classify businesses based on factors such as size and product availability. This approach reveals potential areas with limited access to affordable and nutritious food.

While traditional algorithmic methods could achieve this task, Warwickshire opted to explore the potential of Large Language Models (LLMs). Instead of relying on pre-defined rules, they wanted to assess LLMs' ability to learn and classify businesses based on a general overview. This innovative approach involved utilising the council's in-house expertise to download and implement the open-source LLM, Lama 2. This ensured complete data sovereignty and minimised risks associated with third-party data handling, making the proof-of-concept a low-risk venture.

The project prioritised developing a robust data infrastructure to optimise LLM performance. This emphasis on in-house expertise and data security reflects Warwickshire's commitment to responsible AI adoption while exploring its potential for citizen services.

Governance and internal engagement

To ensure responsible AI development, in-house projects undergo a dedicated review process. This includes checking against a Data Protection Impact Assessment (DPIA), an Equality Impact Assessment (EIA), and a data ethics framework, depending on the project's nature. Additionally, a data readiness scoping exercise identifies potential data quality issues, ensuring the project uses viable data.

This project involved collaboration with wider Business Intelligence (BI) teams. An evaluation report summarising the findings was shared with the BI service in strategic research and further discussed with other service areas for potential integration.

A usage policy for Enterprise CoPilot provides guidance on responsible AI use. Data ethics and algorithmic transparency are key priorities on the council's data roadmap for this year. To further strengthen governance, oversight boards are planned for implementation by year-end. Additionally, the Head of Data will assume the role of AI Lead, overseeing AI governance and implementation across the council. Finally, the council plans to engage its Citizens Panel on the use of AI within local government.

Proof of concept: promising results but room for improvement

While the initial AI implementation showed promise, a simpler data science approach proved more accurate for this specific task. The AI model achieved 63 per cent accuracy based solely on prompt engineering, a more straightforward approach using string concatenation (without AI) achieved 95 per cent. However, the AI performed well at understanding the general task, with 96 per cent accuracy, but struggled with the specific classification challenge. Despite limitations in accuracy, the model shows promise for saving staff time by automating multiple manual searches through further fine-tuning.

Challenges and learnings

Several challenges arose from the council's approach to develop and manage their own LLM:

The council lacked a dedicated funding stream for AI solutions. As a workaround, they used the most powerful local laptop at their disposal to run the LLM.
The chosen LLM (Llama2-12b) required control through human reinforcement and efficient performance with data compression (quantisation). They encountered challenges configuring it for their hardware, leading to slow processing times (up to 1.5 hours per output). A separate cost analysis revealed the significant expense of dedicated LLM hardware, with a used graphics card costing around £41,000.
Security concerns arose during the project. Despite instructions to only provide classifications, the AI generated irrelevant information (hallucinations) in 3.8 per cent of cases. Eight instances presented a potential security risk by including business names in the responses. This highlights the need for robust security measures when deploying LLMs. Since this was a proof-of-concept with internal data management, risks were mitigated within the council.

Data quality was satisfactory for this project, focusing solely on business properties and types. Despite the challenges, the development cost was minimal, requiring mostly time investment. Overall, the project demonstrated the potential of LLMs but also highlighted the need for careful consideration of accuracy, security, and hardware requirements before full-scale deployment.

Overall and what next

Warwickshire Council's "Access to Food" project serves as a bold experiment in exploring open-source LLMs for local government applications. While simpler methods achieved superior accuracy for the specific task of business classification, the project yielded valuable learnings.

The council's innovative decision to leverage an open-source LLM (Lama 2) demonstrated a cost-effective and data-sovereign approach. This strategy minimised risks associated with third-party data handling and empowered the council to explore the technology independently. However, the project also highlighted the challenges associated with open-source LLMs. Configuring Lama 2 for their hardware proved complex, and processing times were slow.

Overall, Warwickshire Council's approach provides valuable insights for local governments considering LLMs. Their exploration in leveraging open-source models, and identifying limitations like security risks and hardware demands, paves the way for further responsible exploration of open-source LLMs in local government services. The council continues to explore cost-effective ways to run LLMs in-house for future applications.