Home Artificial Intelligence AI-based proof-of-concept cuts ‘manual effort’ in climate data gathering: BIS

AI-based proof-of-concept cuts ‘manual effort’ in climate data gathering: BIS

Climate change: those involved in the BIS Innovation Hub-led experiment point out that although financial companies increasingly disclose climate-related information in corporate reports, the data is often ‘not easily accessible’; inset: BIS Innovation Hub Eurosystem head Raphael Auer | Credit (main photo): Jody Davis, Pixabay

Findings of a proof-of-concept project based around using artificial intelligence (AI) to accelerate the extraction of climate-related data from corporate reports have been presented today by European central bank innovation experts.

The Bank for International Settlements (BIS) Innovation Hub has been working with the Bank of Spain, Deutsche Bundesbank and European Central Bank (ECB) on ‘Project Gaia’, which has focused on using AI – in particular large-language models (LLMs) – to enable ‘comprehensive’ analysis of climate-related risks in the financial system by ‘automatically extracting’ climate-related indicators from publicly available corporate documents.

Experimentation co-ordinated by the BIS Innovation Hub’s Eurosystem centre aimed to help analysts and supervisors search corporate climate-related disclosures and extract data ‘quickly and efficiently’ on indicators such as total emissions, green bond issuance and voluntary net-zero commitments.

‘Using AI and, in particular, LLMs, Gaia delivered a proof-of-concept demonstrating it is possible to automate the task of identifying such indicators across a large set of reports, significantly reducing manual effort in climate assessments,’ the BIS announces today (19 March).

Project researchers have ‘broken new ground by integrating LLMs into an application and using it for the extraction of data on climate-related financial risks’, BIS explains, adding that its ‘flexible design may serve as a model for AI-enabled applications in a broader range of use cases for central banks and the financial sector.’

Potential ‘beyond financial industry’

The 42-page ‘Project Gaia: enabling climate risk analysis using generative AI’ report sets out the rationale for the initiative by stating that although financial companies increasingly disclose climate-related information in their corporate reports, ‘these data are not easily accessible today due to the heterogeneity of regulations, frameworks and standards, among other challenges’.

Project Gaia used AI and LLMs to extract climate-related key performance indicators (KPIs) from corporate reports, ‘open[ing] up the possibility of analysing climate-related financial risks at a scale that was previously unimaginable.’

‘The traditional manual approach of collecting KPIs for analysis requires dedicated effort to add each additional KPI and each additional institution,’ the report explains. ‘However, once the platform [was] available [to the project researchers], adding new KPIs or new institutions comes at near-zero costs and with very little delay.’

‘As is the case in other projects close to the edge of technological development, some of the findings may be used in a new context and some design choices may become obsolete after the project. But the learnings from hands-on usage and integration into real-life processes will help shape expectations and pave the way forward in the field,’ it continues.

‘The use of LLM for data extraction is an emerging technology with great potential and Project Gaia is an early example of demonstrating its feasibility and exploring its capabilities at scale,’ it goes on to state, adding that the project’s significance ‘goes beyond climate-related KPIs or, indeed, the financial industry’.

‘The Gaia proof-of-concept demonstrates the power of creating AI-enabled intelligent tools to automate existing workflows,’ the report continues. ‘This approach has the potential to change the way we work in the financial industry and beyond. Project Gaia is one of the first comprehensive studies investigating how this can be done in practice.’

RELATED READING ECB explores ChatGPT-style large-language models in ‘careful embrace’ of AI – a news story (6 October 2023) on a blogpost by the ECB’s chief services officer, Myriam Moufakkir, in which she provided examples of how the ECB is currently using AI across its functions

LLMs’ ‘breakthrough’ required redesign

The report’s executive summary explains the broader context of the project by stating that central banks, supervisory authorities and financial institutions require ‘higher quality and more accessible’ data to model the financial risks posed by climate change.

‘In financial institutions’ corporate reports, climate-related data are buried among other financial and non-financial information and, in many cases, information pertaining to one company is split across multiple reports, and relevant information is contained in texts, tables, footnotes and figures. These challenges constrain the usability of climate-related information,’ it states.

The Gaia project’s vision is to create an ‘open web-based tool’ that helps analysts and supervisors search corporate climate-related disclosures and extract data, ‘thereby reducing the manual effort involved in climate assessments’. Work to date (as being presented today) is described as (just) ‘phase one’.

The profile of AI has exploded since the launch of ChatGPT less than 18 months ago. ‘The evolution of Project Gaia is testimony to recent rapid development in the field of AI,’ the report notes. ‘At the time of project planning, LLMs had limited availability. The breakthrough in LLMs opened up new possibilities and a redesign of the proof-of-concept was therefore required.’

RELATED ARTICLE ECB aims for AI-fired SupTech ‘cockpits’ in ‘smartphone for supervisors’ – a news story (3 October 2023) on ECB Supervisory Board member Elizabeth McCaul’s speech at an ECB-hosted ‘Supervision Innovators Conference 2023’

Non-English language documents

The tool developed by the Gaia project team works by extracting ‘structured’ information from ‘unstructured’ PDF documents, combining all the information elements, such as text, tables and figures.

It was evaluated using a test-set of 2,328 publicly available documents from 187 financial institutions from across the world.

While most of the documents were in the English language, a ‘small number’ of Spanish and German language reports are also included.

‘Tests have shown that Gaia is able to obtain correct values from the non-English documents, which underlines another important advantage of LLM-based text analysis: it can be made, to a large degree, language independent,’ the report states. But it adds that ‘it is important to note that relying on LLMs does not, in itself, make the solution capable of handling multiple languages.’

It goes on to state that ‘other design choices must also be made with language independence in mind, in particular, the search for the relevant context inside the documents needs to be language independent. The semantic searches based on Open AI embeddings used in Gaia fulfil this criterion, but a simple keyword search would fail.’ Open AI is the company behind ChatGPT.

RELATED ARTICLE BIS Innovation Hub highlights tokenisation ambition in 2024 work priorities – our news story (23 January 2024) on the BIS Innovation Hub’s current work programme

Proejct Gaia’s next steps

Turning towards practicalities with the Gaia proof-of-concept, the report states that ‘adding new KPIs or new institutions is quick and easy’, making it possible to ‘extract and analyse a multitude of KPIs from a large number of institutions, opening up the possibility of climate risk analysis at a scale that was previously unimaginable’.

It also offers ‘harmonised metrics’ despite the mishmash of naming and definitions across different jurisdictions and companies.

Within the financial sector alone, the report states that AI-based KPI extraction from large bodies of textual documents ‘can be a game-changer, for example, in regulatory and supervisory use-cases’.

In respect of next steps, the report concludes by stating that the proof-of-concept was constructed with ‘enterprise-grade components, in an architecture designed for high scalability, to meet the needs of a future open tool’.

‘One possible continuation is to make the solution publicly available as an open web-based service for climate-related financial risk analysts and support the growing demand for climate-related data.

Another natural next step is to expand into use cases beyond green finance,’ the report states. It adds that ‘LLMs as a technology are rapidly evolving, increasing performance and expanding with new capabilities, such as internet search or image recognition’. It adds further here that, ‘for example, online search greatly extends an LLM’s knowledge base, enabling responses to current events and potentially covering information that was not part of the model’s training’.

‘Future phases of Gaia will need to investigate and adopt these new developments to continue to harness the full power of cutting-edge AI,’ it concludes.

RELATED ARTICLE COP28 sustainable finance TechSprint winners announced in UAE – a news story (5 December 2023) on a hackathon-style competition that climaxed at the COP28 climate summit in which one of three entry fields (‘problem statements’) was AI solutions for sustainable finance reporting, verification and disclosure in the financial services industry

Further considerations

Technology neither exists nor develops in a regulatory vacuum, and the report also gives a nod to governance matters.

‘Deployment of AI-enabled tools involves several non-technical challenges, which will need to be addressed in a potential future practical application of the Gaia technology,’ the report states, pointing out that there are a growing number of initiatives addressing regulatory and policy issues to mitigate AI-related risks.

So, ethical and legal considerations need to be taken into account and ‘proper safeguards put in place to ensure privacy, security and accountability’, the report states. It adds that users of AI systems ‘must be able to understand them and be comfortable using them’ and, in addition, the environmental impact of large AI models ‘needs to be monitored and minimised’.

‘To ensure that these and other considerations are properly accounted for, a future real-life application of the Gaia technology will need to be surrounded by proper governance structures and processes,’ the report cautions.

The report acknowledges the involvement of 11 team members at the BIS Innovation Hub’s Eurosystem centre, including centre head Raphael Auer; as well as six and four team members at the Bank of Spain and Bundesbank, respectively, and two ECB team members. The project has also been informed by members a ‘Green Finance AI Working Group’, among other groupings.