How the data pipeline works

This page covers the data and software side of the Climate Extremes AI Agent. It shows how precipitation data are pulled from five public data services, each with its own access method. It then follows how they become a small compressed demo store, and how an AI agent runs fixed, deterministic diagnostics on that store without ever doing the climate maths itself.

5 data servicesCDS API, ESGF search, OPeNDAP, DestinE STAC, NOAA HTTPS
14 MBcompressed NetCDF demo store shipped with the app
11 toolsregistered for the agent, schemas built from the code
≤ 3 roundsof tool calls per question, then a forced summary
60 shard time limit for the sandboxed code tool
~5,400 linesof Python in scripts, library and app

1. The pipeline at a glance

Data flows from left to right. Each box is one stage, and the blue text names the code that does it. The access scripts run once. The deployed app works only from the demo store.

1 · SERVICES Copernicus CDS ERA5, cdsapi request queue ESGF CMIP6, search API + OPeNDAP NOAA PSL THREDDS CPC, OPeNDAP + HTTPS DestinE HDA Climate DT, STAC search 2 · ACCESS Access scripts ① search the catalogue (Solr JSON, STAC POST) ② subset on the server (OPeNDAP, CDS area) ③ log in where needed (CDS key, DESP token) ④ wait for queued orders ⑤ retry on 5xx errors scripts/download_*.py 3 · PREPARE Demo store units to mm/day regrid to a common 0.5° grid cut to regions, monthly means for 1995–2014 zlib-compressed NetCDF, 6 files, 14 MB create_demo_data.py utils/regrid.py · data/demo/ 4 · AGENT ReAct loop Gemini or Groq picks tools from the registry TF-IDF lookup of dataset, region and index facts Python runs the tool, the LLM only reads the result and explains it max 3 rounds, then a text-only summary ai/agent.py · ai/knowledge.py 5 · RESULTS Diagnostics RX1day, RX5day, R95p, bias, trends, correlation, distribution fits diagnostics/ · ai/stats.py Streamlit app figures + JSON sidecar with metadata, activity trace, interpretation dashboard/streamlit_app.py What the demo store holds CMIP6 MPI-ESM1-2-HR historical precipitation: global monthly 2013, and Germany monthly 1995–2014. NOAA CPC gauge-based precipitation: Europe and South Asia daily 2013, global monthly 2013 regridded, and Germany monthly 1995–2014. The ERA5 and DestinE scripts show how those services are reached. Their data are not part of the shipped demo store.

2. Five services, five ways in

Each data service has its own way of finding data, logging in and delivering it. The scripts in scripts/ are small, readable examples of each one.

ServiceHow data are foundHow data are deliveredLoginScript
Copernicus Climate Data Store (ERA5)Named dataset plus a request dictionary: variable, dates, hoursThe request is queued on the server and cut to a Europe box there, then returned as NetCDFCDS API keydownload_era5_precip.py
ESGF search (CMIP6)Search API with facets (project, source_id, experiment_id, table_id, member_id) returning Solr JSONFile records are filtered by the date range in the file name, then fetched over HTTPSnonedownload_cmip6_precip.py
ESGF and NOAA THREDDS (OPeNDAP)Known dataset URLs per year or per 5-year fileOPeNDAP subsetting: only the Germany box travels over the network, never the global filenonedownload_germany_20yr.py
DestinE Harmonised Data Access (Climate DT)STAC search (POST /stac/search) on collection DT_CLIMATE_ADAPTATION, filtered by model, experiment, resolution and parameterAn order is placed, the script polls while the server answers 202 (queued), then streams the fileDestinE platform account, bearer token, Climate DT access checkdownload_destine_precip.py
NOAA PSL (CPC)Fixed file pattern per yearPlain HTTPS downloadnonedownload_cpc_precip.py

3. Problems the pipeline had to solve

transfer size

Don't download the globe

Twenty years of daily global data would be large. For Germany, OPeNDAP cuts the box on the server, so only the subset is sent, and the CDS request asks for a Europe area instead of the whole globe.

conventions

Different grids and units

CMIP6 comes in kg m⁻² s⁻¹ and is converted to mm/day. Longitudes run 0–360° in one source and −180–180° in another, and latitude order differs. The subset code checks the latitude direction before slicing, and fields are regridded to a common 0.5° grid.

regridding

A safe fallback

xESMF is used for regridding when it is installed. Otherwise the code falls back to xarray interpolation, so the app still runs on a host without the heavier library.

asynchronous services

Queued orders

DestinE and the CDS do not hand over files at once. The DestinE script follows the order's Location header and polls every five seconds until the file is ready. Requests are retried automatically on 500–504 errors.

credentials

Secrets stay out of the code

DestinE credentials come from environment variables, a local .env file or a password prompt. LLM keys come from Streamlit secrets. None of them is stored in the repository.

deployment

Small enough to ship

The demo files are re-saved with zlib compression (level 5), without losing data, so the whole store is 14 MB and the free app host needs no download at start-up.

deterministic

The LLM never does the maths

Extreme indices, biases, trends and fits are checked-in Python, xarray and SciPy functions. The JSON schema for each tool is built automatically from the function's signature and docstring, so the agent's view of a tool always matches the code.

guard rails

Bounded and sandboxed

The agent stops after three rounds of tool calls and must then summarise. The fallback code tool checks scripts for imports and unsafe calls before running them. It then runs them in a separate process with a 60-second limit and read access to the project data folders only.

4. How the code is organised

Access layer scripts/download_*.py · create_demo_data.py compress_germany_files.py One script per service, plus the steps that turn downloads into the demo store. Science layer diagnostics/ · utils/ · plotting/ · ai/stats.py ETCCDI indices, climatologies, bias, trends, pattern correlation, distribution fits, maps. Each figure is saved with a JSON sidecar holding its metadata. Agent and app layer ai/agent.py · tools.py · knowledge.py code_sandbox.py · dashboard/ Tool registry, the TF-IDF knowledge base built from the NetCDF attributes, the ReAct loop and the Streamlit interface.
SafeguardWhat it does
Fixed tool registryThe agent can call only the 11 registered functions. An unknown tool name comes back as an error, and when a tool fails the agent stops instead of answering from incomplete results.
Activity traceEvery question shows which tools ran, with which arguments and with what result, next to the answer.
Fallback for broken tool callsIf a model returns malformed tool-call JSON, the agent falls back to a plain text answer instead of crashing.
Metadata from the filesThe knowledge base reads dimensions, variables and attributes straight from the NetCDF files, so answers about the data match the data.
Pinned environmentenvironment.yml and requirements.txt pin versions, including numpy < 2 for Cartopy, and packages.txt lists the system libraries for the cloud host.

5. What could come next

StepWhy
Automated tests and CIThe extreme indices and statistics are good candidates for small offline tests on GitHub Actions.
Km-scale DestinE dataThe STAC access route already exists. Adding Climate DT precipitation to the store would move the extremes to km-scale models.
Cloud-native storeSaving the prepared data as chunked Zarr would let the app read larger, multi-year datasets lazily.

6. In one sentence

Precipitation data are found and fetched through five public services: an API request queue, a search API, OPeNDAP server-side subsetting, a STAC catalogue with token login and plain HTTPS. They are harmonised and compressed into a 14 MB store. A tool-bounded AI agent then runs only deterministic, checked-in diagnostics on that store and shows every step it took.