[MaoMengww]

Which issue or RFC does this PR close?

Closes #1975.

Implements the Source authority and verification prerequisite in RFC #1652, while preserving the Source access and lineage_only semantics in RFC #1458.

Rationale for this change

Source generation eligibility does not describe the authority or verification of its material. Atomic Memory needs explicit, versioned evidence declarations that remain inspectable after registration changes, restarts, and projection rebuilds.

This change records those facts without introducing trust scores, ranking bonuses, or additional write authority.

What changes are included in this PR?

• Add MemoryEvidenceDeclaration and exact Source material bindings through the existing Source Definition registry.
• Accept declarations only from trusted native adapters or deployment-owned configuration for exact remote Definition manifest identities.
• Retain historical Definition manifests, Source declaration bindings, and evidence snapshots keyed to exact Atomic Memory revisions.
• Add an authorized Python evidence-read API and reuse the existing projection rebuild path to verify retained historical material.
• Resolve missing, invalid, or unverifiable declaration records to neutral untrusted / unknown values. Missing or corrupt authoritative Source material and database failures remain errors.
• Add the additive p0005 evidence schema migration for SQLite and MySQL-compatible backends, without backfilling historical trust.
• Add compact SQLite/OceanBase acceptance scenarios covering revision history, write rollback, registration drift, cross-Scope isolation, missing-snapshot neutrality, synthetic model trust claims, and unchanged recall consumers.

Are there any user-facing changes?

This adds optional Source Definition evidence declarations and a Python API for inspecting an exact Atomic Memory revision's evidence.

It does not change Atomic Memory bodies or content hashes, Artifact revision identities, HTTP/OpenAPI search schemas, public search scores, or citation identities. It introduces no quality ranking or new generation eligibility rules.

Persistence gains additive evidence tables. Existing Sources and Memory revisions without accepted declarations remain neutral; current configuration does not retroactively attest them.

How was this change tested?

• make check: passed, including lock consistency, pre-commit checks, generated guidance checks, and type checking.
• make contract-test: 52 passed.
• make opendal-test: 8 passed; connector lint, formatting, and type checks passed.
• Full suite with doctests and the required DSH configuration runtime, using Python 3.12.13 / SQLite 3.50.4:
  • 5294 passed, 385 skipped, 0 failures.
  • Command: uv run --frozen --python 3.12 python -m pytest -n 6 --doctest-modules --require-dsh-config-runtime
• Focused Source, persistence, migration, lifecycle, and recall regressions: 128 passed.
• Real OceanBase CE 4.4.2.1 evidence acceptance: 3 passed, 0 skipped.
• Wheel and source distribution builds passed; the wheel contains the new evidence migration resources.

The initial full run under the local Python 3.11 / SQLite 3.37.2 environment exposed an existing Topic Memory vector-prefilter compatibility failure. A standalone SQLite reproducer isolated the cause: sqlite-vec's rowid IN KNN prefilter requires SQLite 3.38 or newer. The original test and full suite passed with the newer SQLite runtime, without changing code, assertions, or skip conditions.

The complete GitHub Actions Python-version matrix has not been run locally.

AI usage statement

AI assistance was used through Oh My Pi with gpt-6.1sol for implementation, test design and consolidation, code inspection, and validation analysis.

[hidb4ai]

Please address the two native Source evidence issues below before merging. Both reproduce through public SQLite flows and break evidence inspection and projection rebuild after successful Memory generation.

[hidb4ai]

Native Source evidence inspection and projection rebuild pass the input-alias and model-default scenarios. Remote observation ingestion still fails for serialization aliases; please address the inline P2 before merging.

[hidb4ai]

The three reported reproduction scenarios now pass. One remote Source regression remains when aliases apply to inherited envelope fields; please address the inline P2 before merging.

[hidb4ai]

Re-reviewed 25d9e7a7. The previous two P2s are fixed. One remote-ingestion regression remains for recursive Source schemas with identity aliases; please address the inline P2 before merging.

[hidb4ai]

Re-reviewed 2b95df67. The recursive Source transport regression is fixed. Two P2s remain: an unchanged aliased remote Definition conflicts with its existing manifest after upgrade, and version-2 canonical history still fails evidence inspection and rebuild under a newer aliased Definition. See the inline finding and my follow-up in the existing historical-identity thread. Please address both before merging.

[hidb4ai]

[P2] Preserve registered remote manifest identities when upgrading

An unchanged remote Definition with model_config = ConfigDict(serialize_by_alias=True) and body: str = Field(alias="wire_text") registers and submits successfully on base 0da4436d. After upgrading the same SQLite database, forcing by_alias=False changes its schema from wire_text to body and changes its fingerprint without changing the Definition name/version. register_source_definition() then raises SourceConflictError against the previously accepted manifest, blocking the worker. Preserve existing supported manifest/payload identities or provide an explicit version transition rather than silently redefining the same version.

[MaoMengww]

Fixed in 13517fdd.

The transport contract now derives both sides from the same alias policy:

• _manifest_for_definition(transport=True) uses model_json_schema(mode="serialization", by_alias=bool(model_config.get("serialize_by_alias")));
• project_source_for_transport() uses model_dump(mode="json"), whose default by_alias=None follows the same serialize_by_alias config.

For the reported case, serialize_by_alias=True with body: Field(alias="wire_text") now keeps wire_text in both the schema and the payload, and the fingerprint matches the base model_json_schema() result. Definitions that only declare serialization_alias still use field names, also matching base.

test_alias_serialization_remote_manifest_and_payload_remain_upgrade_stable reproduces the upgrade path: register on a fresh SQLite database, reopen the same database, re-register the unchanged manifest, and submit the observation. It asserts the schema equals the base-generated schema, the fingerprint is recomputed from it, and the payload retains wire_text.

Affected tests: 63 passed, 10 skipped; make check passed. OceanBase parameterizations were skipped locally because no instance is available.

[hidb4ai]

[P2] The original serialize_by_alias=True case is fixed on 3c5c0ecc, but fingerprint compatibility still fails for an ordinary amount: Decimal field with no aliases. On base 2d11abd6, public manifest registration and observation submission succeed: the validation schema allows number/string, and the payload contains "1.25". On this head, mode="serialization" at observations.py:178 produces a string-only schema and changes the fingerprint for the unchanged Definition name/version. Re-registering it on the upgraded SQLite database raises SourceConflictError, blocking the worker even though the payload is identical. Preserve supported pre-upgrade manifest identities or provide an explicit version transition.

[MaoMengww]

Fixed in 3a8b38be.

The transport schema now keeps the pre-upgrade validation schema unless an alias renames a required payload key, checked recursively for nested models. Optional aliased fields keep the base schema because the canonical key is an allowed extra property while the alias key is not required, which matches pre-upgrade behaviour and preserves the fingerprint. AliasChoices resolves to its first string choice; AliasPath keeps the field name.

Regressions:

• test_decimal_source_manifest_fingerprint_survives_upgrade asserts the Decimal Definition keeps the base model_json_schema() and fingerprint, and that re-registration and observation submission still succeed after reopening the same SQLite database.
• test_optional_alias_keeps_the_preexisting_manifest_fingerprint covers an aliased optional version field and asserts the same fingerprint preservation.

[hidb4ai]

[P2] The Decimal reproduction now passes on 3a8b38be, but manifest compatibility still fails for nullable nested models. A Source with detail: Detail | None = None, where Detail.label: str = Field(alias="wire_label"), registers and submits detail: null successfully on base 2d11abd6. The recursion at observations.py:264-266 sees the required alias inside Detail and switches the entire manifest to serialization mode, changing its nested schema from wire_label to label and changing the fingerprint. On this head, re-registering the unchanged Definition on the upgraded SQLite database raises SourceConflictError, although the payload is identical. Preserve the registered contract for previously accepted payloads, or require an explicit Definition version transition when changing nested serialization.

[MaoMengww]

Fixed in 4a7f41ff.

Manifest compatibility now ignores nested models that the payload is not required to instantiate:

• only a required, non-nullable, directly nested BaseModel can influence the schema mode;
• Detail | None, optional fields, and list/dict containers keep the pre-existing validation schema and fingerprint;
• required top-level fields still fall back to the serialization schema when an alias renames them.

Added test_nullable_nested_alias_keeps_the_preexisting_manifest_fingerprint, which asserts the base model_json_schema() and fingerprint, then re-registers on the same SQLite database and submits detail: null successfully.

Also merged upstream 1da99747 and resolved the mysql schema and observation-validation import conflicts.

Verification: 152 passed, 17 skipped across the source/evidence/migration suites; make check passed.

[hidb4ai]

Re-reviewed 3c5c0ecc. The original alias-configuration and version-2 historical-read reproductions now pass. One P2 remains: changing transport schema mode changes the fingerprint of an unchanged Decimal-bearing Definition and prevents re-registration after upgrade. Added the reproduction to the existing manifest-compatibility thread. Please address this before merging.

[hidb4ai]

Re-reviewed 3a8b38be. The Decimal upgrade reproduction now passes. One manifest-compatibility P2 remains: a nullable nested model changes the unchanged Definition's fingerprint and blocks re-registration after upgrade, even when the payload retains detail: null. Added the reproduction to the existing manifest-compatibility thread. Please address this before merging.