Repository navigation
Conversation
Implement HydraToPywrV2Network exporter and `export-v2` CLI command to
translate Hydra network data into pywr-next (v2) JSON format.
Key v2 schema differences handled:
- Network wrapped in top-level `"network"` key; parameters as array
- Node names in meta.name; scalars/refs wrapped as Metric objects
- Edges as dicts with from_node/to_node
- Timestepper: integer days -> {"type":"Days"} or {"type":"Frequency"}
- Profile values wrapped as {"type":"Literal","values":[...]}
- Aggregated: parameters->metrics, agg_func string->{"type":"Sum"} etc.
- ControlCurve family: storage_node->storage_metric Node ref,
control_curves and values items wrapped as Metrics
- Reservoir/Storage: initial_volume uses Absolute/Proportional envelope;
evaporation/rainfall use {"data": metric} wrapping; level/area dropped
- Turbine: f64 fields kept as raw scalars, field renames applied,
unknown Hydra attributes dropped
- Empty scenarios list omitted entirely (schema rejects [])
- Recorders, dataframe/interpolatedvolume params skipped
- v1 optimisation fields (is_variable, lower/upper_bounds) stripped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace hand-rolled v1→v2 parameter conversion logic with a clean pass-through for v2-format data. V1-style type names (e.g. 'monthlyprofile', 'ControlCurveInterpolatedParameter') now raise V1FormatError with the offending parameter name, the v2 equivalent type, and instructions to use pywr.convert_model_from_v1_json_string(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The published schema differs from the older pywr-next checkout the exporter
was written against. Exported models were rejected by ModelSchema:
- the top-level "timestepper" key is now "time"
- metric sets and outputs are identified by "meta": {"name": ...} rather than
a top-level "name"
Found by validating and running an exported model against the 2.0.0b11 wheel.
The existing test modules import classes that were removed long ago (PywrHydraImporter, PywrHydraExporter) and fail at collection, so none of the suite could run. Exclude them in tests/conftest.py until they are ported, and add tests against the current API: - test_v1_roundtrip.py: import the v1 parent template, import a v1 model into Hydra and export it again, for four models. - test_v2_exporter.py: build a small network from the v2 parent template in Hydra, export it with HydraToPywrV2Network, validate it with the Pywr v2 schema and run it, checking the flows. Skipped when Pywr v1 is installed, since v1 and v2 cannot share an environment. - local_client.py: an in-process client that translates the remote API calling conventions hydra-pywr uses, and imports a template from a JSON file (register_template depends on functions that no longer exist). - tests/data/templates: copies of the v1 and v2 parent templates. test_model_results.py is skipped when Pywr v2 is installed. All tests use temporary SQLite databases.
hydra_pywr imports scipy, numpy, click, python-dateutil, requests, s3fs and pymongo directly but did not declare them, so a clean install failed to import (scipy was not pulled in by anything else). Add v1 and v2 extras to choose the Pywr runtime. Pywr 1.x and 2.x both install as the `pywr` module and cannot share an environment. hydra_client, hydra_network_utils and pywrparser are imported but not listed: they are not installable from PyPI under usable names (the `hydra-client` project on PyPI is an unrelated package). Their git locations are noted in a comment.
Runs the test suite in two jobs, one per Pywr runtime, on pushes to master and pull requests. Both use temporary SQLite databases only. hydra-base is installed from the branch carrying the template cache fix until it is released. The v1 job deselects test_hydropower_results, which needs hydropower nodes not present in Pywr 1.x from PyPI.
5 tasks
Conflict in pyproject.toml: keep master's requires-python (>=3.10) and its git dependencies (pywrparser, hydra-client, hydra-network-utils) together with this branch's added PyPI dependencies and the v1/v2 extras. Drop the comment that said the git packages were deliberately not declared.
pywrparser, hydra-client and hydra-network-utils are now declared in pyproject.toml, so only hydra-base (pinned to a branch until its fix is released) still needs installing first.
test_losslink_aggregated_companion_recorder_is_ignored builds a model with pywr.model, which does not exist in Pywr v2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #62.
What this PR contains
The branch already had the Pywr v2 JSON exporter (
exporter_v2.py,export-v2) and the change to raise a clear error for v1 parameter types. This adds:4e9d62d). The exporter was written against an older pywr-next checkout, and its output was rejected by the publishedpywr2.0.0b11 wheel. The top-leveltimestepperkey is nowtime, and metric sets and outputs usemeta: {name: ...}instead ofname. Found by validating and running an exported model.e3d341a). None of the existing test modules could be collected: seven import classes that were removed long ago (PywrHydraImporter,PywrHydraExporter), and the eighth fails under Pywr 1.31.1 from PyPI. Those seven are excluded intests/conftest.pyuntil they are ported. New tests use the current API:test_v1_roundtrip.py: imports the v1 parent template and four v1 models into Hydra and exports them again.test_v2_exporter.py: builds a small network from the v2 parent template, exports it, validates it against the v2 schema, runs it, and checks the flows. Skipped when Pywr v1 is installed.local_client.pyandtests/data/templates/: an in-process client (hydra-pywr is written for the remote API's calling conventions) and copies of the v1 and v2 parent templates.register_template()can't be used for the tests because the functions it calls no longer exist.pyproject.toml(10e69c2). Declares dependencies thathydra_pywrimports directly but did not list (scipywas missing, so a clean install could not import the package), and addsv1andv2extras. Pywr 1.x and 2.x both install aspywrand cannot share an environment.7ded018). A GitHub Actions workflow that runs the tests once per Pywr runtime.Things to know
HYDRA_BASE_REFin the workflow tomasteronce that is merged, otherwise CI will break when the branch is deleted.test_hydropower_results. It fails withKeyError: 'turbine'under Pywr 1.x from PyPI, so hydropower is not covered by CI.hydra_client,hydra_network_utilsandpywrparserare not declared inpyproject.toml. They are not installable from PyPI under usable names (thehydra-clientproject on PyPI is an unrelated package), so the workflow installs them from git.PYWR2_*datatypes.