Skip to content

Show errata of the datasets in a recipe - #18

Merged
bouweandela merged 16 commits into
mainfrom
dataset-errata
Nov 4, 2026
Merged

bouweandela merged 16 commits into
mainfrom
dataset-errata

Conversation

@bouweandela

@bouweandela bouweandela commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Shows the errata from the ESGF errata service that affect the datasets in a recipe, on the line of the dataset, linked to the erratum. Errata are shown as information, to distinguish them from errors and warnings about the recipe itself, and the message gives their status (new, on hold, resolved, or won't fix). Resolved errata are only shown if the recipe pins an affected version, because a resolved erratum usually no longer affects the latest version. If an erratum only affects some of the variables of a dataset, the message names them.

The errata are looked up in the background after the problems in the recipe have been shown, shown as they are found, and the answers of the service are kept on disk for a day with requests-cache. The service is searched by project, model, and experiment, and the affected datasets are matched in the server. The errata service API is not documented, so this may need to change if the service changes. The setting esmvaltool.languageServer.errata (or initializationOptions.errata in other editors) turns this off.

Files that are not ESMValTool recipes get no errata. tests/test_links.py checks that the erratum page the diagnostics link to resolves.

Tested with pixi run test, pixi run pre-commit run --all-files, and against the real service with recipes with CESM2 and GFDL-CM4 datasets.

🤖 Generated with Claude Code

bouweandela and others added 2 commits November 2, 2026 13:20
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Nov 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.68%. Comparing base (04feee2) to head (8cf000e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #18      +/-   ##
==========================================
+ Coverage   92.56%   93.68%   +1.12%     
==========================================
  Files          20       21       +1     
  Lines        2031     2328     +297     
  Branches      311      350      +39     
==========================================
+ Hits         1880     2181     +301     
+ Misses        111      110       -1     
+ Partials       40       37       -3     
Flag Coverage Δ
default 95.65% <100.00%> (+0.98%) ⬆️
py312 95.65% <100.00%> (+0.98%) ⬆️
vscode-1.93.0 82.30% <100.00%> (+0.58%) ⬆️
vscode-stable 82.30% <100.00%> (+0.58%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bouweandela and others added 9 commits November 3, 2026 09:05
Use the renamed intake_esgf_mapping, check the erratum link in test_links.py,
and test that other YAML files get no errata.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Find the location of datasets that come from a YAML merge key, instead of
leaving out their errata, and show an erratum once if an alias or merge key
makes a dataset part of several paths. Test that a failure to find errata
keeps the other diagnostics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dataset that is shared through an alias or merge key now gets one message
for each erratum, naming the variables of all its paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Expand ranges such as r(1:5)i1p1f1 like ESMValCore does, instead of treating
them as any ensemble member, so an erratum of a member outside the range is
not shown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Publish the errata after looking up each source, instead of when all have
been looked up, and stop looking them up when a new check starts. Only get
resolved errata for datasets with a version, because the search tells which
errata are resolved and they do not apply to other datasets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Search for errata by experiment, if the service lists that facet for the
project, so fewer errata need to be fetched. Fetch them one at a time, because
fetching them concurrently is hardly faster: the service handles about 4.6
requests per second however many are sent at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use requests-cache instead of a cache in memory, so restarting the server does
not look up all errata again. Expired answers are removed when the cache is
opened, and errata can still be looked up if the cache cannot be used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

bouweandela and others added 5 commits November 4, 2026 19:22
- Keep the errata on the screen until a new lookup has finished, or if the
  service fails, and warn the user once if errata may be missing.
- Look up errata in their own thread, and do not ask the service for a minute
  if it cannot be reached, so waiting for it does not hold up recipe checks.
- Keep the errata that were found if fetching another one fails.
- Remember the answers in memory as well, for repeated checks of a recipe.
- Compare derived variables with the variables they are computed from.
- Mention the experiment in the description of the errata setting, and test
  that the setting is sent to the language server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An erratum is not a problem with the recipe, so errata that are new or on hold
should not look like warnings about the recipe. Add the status to the message,
to keep it visible which errata are still open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Notice when the API of the errata service or another web service that the
tests use changes, also when no changes are pushed to the repository.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bouweandela
bouweandela merged commit af510d0 into main Nov 4, 2026
5 checks passed
@bouweandela
bouweandela deleted the dataset-errata branch November 4, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants