Skip to content

Replace datasets with glob patterns by the datasets with data - #22

Open
bouweandela wants to merge 8 commits into
mainfrom
populate-datasets
Open

bouweandela wants to merge 8 commits into
mainfrom
populate-datasets

Conversation

@bouweandela

@bouweandela bouweandela commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Adds a code lens above datasets: and additional_datasets: lists with glob patterns, e.g. {dataset: '*', project: CMIP6, exp: historical, ensemble: 'r*i1p1f1'}, that replaces those entries by the datasets with data for every variable group that uses the list. The same command is offered as a code action on that line, for editors without code lenses such as Emacs with Eglot. Neither is offered in files that are not ESMValTool recipes.

  • The values are found with the facet search used for completion, one facet at a time for all variable groups at once. ensemble and sub_experiment are expanded last, which reduces the number of searches: all CMIP6 historical r*i1p1f1 datasets for two variables take about 40 seconds instead of 3 minutes.
  • Ensemble members are written as ranges with esmvalcore.dataset.datasets_to_recipe, and the entries are written as format_recipe writes them. Only the entries with glob patterns are edited; anchored entries and aliases are left alone.
  • Progress is shown while searching, errors (e.g. no data, or a facet that cannot be searched) are shown as messages, and the edit is only applied if the recipe has not changed in the meantime.
  • Needs a newer ESMValCore commit than the one in pixi.lock on main (for format_recipe), so pixi.lock is updated.

Known issue, not fixed here: the ESGF searches share intake-esgf's sqlite-backed requests cache between threads, and one large run logged sqlite3.InternalError: cannot reset statement.

Tested with pixi run test, pixi run test-vscode, pixi run pre-commit run --all-files, and against ESGF.

🤖 Generated with Claude Code

Add a code lens above lists of datasets with glob patterns, and a code
action for editors without code lenses, that replaces these datasets by
the datasets that have data for every variable group that uses the
list. The values are found with the facet search used for completion,
and ensemble members are written as ranges with datasets_to_recipe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Nov 2, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.78788% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.04%. Comparing base (4397523) to head (6acb533).

Files with missing lines Patch % Lines
src/esmvaltool_language_server/populate.py 98.33% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #22      +/-   ##
==========================================
+ Coverage   93.71%   94.04%   +0.33%     
==========================================
  Files          21       22       +1     
  Lines        2339     2504     +165     
  Branches      350      371      +21     
==========================================
+ Hits         2192     2355     +163     
- Misses        110      111       +1     
- Partials       37       38       +1     
Flag Coverage Δ
default 95.89% <98.78%> (+0.24%) ⬆️
py312 95.89% <98.78%> (+0.24%) ⬆️
vscode-1.93.0 82.94% <ø> (ø)
vscode-stable 82.94% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bouweandela and others added 5 commits November 3, 2026 09:09
Only offer the code lens and code action for ESMValTool recipes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the time range can be searched for, `timerange: '*'` would be
replaced by a copy of the dataset for each time range found. ESMValCore
resolves these patterns itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bouweandela and others added 2 commits November 4, 2026 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants