TRAPI Object Modeling: translator_tom

A library for statically typed, fast serialize/deserialize TRAPI in Python, for Translator-wide use.

Models based on Pydantic provide deserialize with basic validation, serialize, and statically-typed construction with very reasonable performance, as well as utility methods based on architectural descisions, such as message/result/kg/etc. merging and standard TRAPI manipulation.

Allows for easy FastAPI standup.

Model Usage

The main ways you interact with a Model are as follows:

• Model.from_json() and Model.to_json()
• Model.from_dict() and Model.to_dict()
• Model.from_msgpack() and Model.to_msgpack()
• Validated instantiation: Model()
• Direct construction: Model.model_construct()

JSON Validation

These models can be used for validation of straight JSON:

from translator_tom import Query

query_json = """
{
  "submitter": "TOM tester",
  "message": {
    "query_graph": {
      "nodes": {
        "n0": { "ids": [ "PUBCHEM.COMPOUND:726218" ] },
        "n1": { "ids": [ "NCBIGene:3778" ] }
      },
      "edges": {
        "e0": {
          "subject": "n0",
          "object": "n1",
          "predicates": [ "biolink:related_to" ]
        }
      }
    }
  }
}
"""

query = Query.from_json(query_json)

# Access is now statically typed and editor provides hints + completions
query_graph = query.message.query_graph
assert query_graph is not None
assert len(query_graph.nodes) == 2  # True

Similarly, you can validate from JSON with a FastAPI endpoint:

from fastapi import FastAPI
from translator_tom import Query

app = FastAPI()

@app.post("/query")
def query(body: Query) -> str:
    return f"Got {len(body.message.query_graph.nodes)} query nodes!"

Dict Validation

You can also validate dicts:

from translator_tom import Query

query_dict = {
  "submitter": "TOM tester",
  "message": {
    "query_graph": {
      "nodes": {
        "n0": { "ids": [ "PUBCHEM.COMPOUND:726218" ] },
        "n1": { "ids": [ "NCBIGene:3778" ] }
      },
      "edges": {
        "e0": {
          "subject": "n0",
          "object": "n1",
          "predicates": [ "biolink:related_to" ]
        }
      }
    }
  }
}

query = Query.from_dict(query_dict)

query = Query(**query_dict)  # Also works (less clear, not recommended)

Construction

There are two ways to construct instances within Python:

The first is just calling the model like any class. This ensures everything you pass is validated (meaning you don't need to construct every model and can just pass dicts, though static type checkers will complain).

from translator_tom import Query

query = Query(
    submitter="TOM tester",
    # Could use types, or just pass a dict (static checkers will complain, though!)
    message={
        "query_graph": {
            "nodes": {
                "n0": {"ids": ["PUBCHEM.COMPOUND:726218"]},
                "n1": {"ids": ["NCBIGene:3778"]},
            },
            "edges": {
                "e0": {
                    "subject": "n0",
                    "object": "n1",
                    "predicates": ["biolink:related_to"],
                }
            },
        }
    },
)

Another way is to use Model.model_construct().

Warning

This does no validation, so it's faster, but you have to pass correct construction for everything. No types will be coerced to their correct models. Only use it for internal construction where you know everything is already valid (a static type checker such as ty or pylance is highly recommended!). An example:

from translator_tom import Biolink, Curie, Message, QEdge, QNode, Query, QueryGraph

# Using each type provides hints and type checking, making internal TRAPI construction
# safer.
query = Query.model_construct(
    submitter="TOM tester",
    message=Message(
        query_graph=QueryGraph(
            nodes={
                # Used a helper function to ensure curie formatting (optional)
                "n0": QNode(ids=[Curie("PUBCHEM.COMPOUND", "726218")]),
                "n1": QNode(ids=["NCBIGene:3778"]),
            },
            edges={
                "e0": QEdge(
                    subject="n0",
                    object="n1",
                    # Used a helper function to ensure biolink prefix formatting (optional)
                    predicates=[Biolink("related_to")],
                )
            },
        )
    ),
)

Convenience Methods

TOM provides many convenience methods, similar to those in reasoner-pydantic.

from translator_tom import KnowledgeGraph

my_kg = KnowledgeGraph.new()  # Init an empty knowledge graph

other_kg = get_some_kg()  # Imagine a function returns another KG with data...
_old_new_mapping = other_kg.normalize()  # Normalize edge IDs (keeps a mapping of old->new)

my_kg.update(other_kg, pre_normalized="other")  # Handles merging appropriately, can skip redundant normalization

There are many more, it's recommended to look at the models themselves as they are self-documenting (every model has docstrings equivalent to the descriptions in the original spec!). Common examples are .<field>_list and .<field>_dict for optional container fields for easy iteration without None-guarding, .new() for quick instantiation with sensible defaults (mostly of container-like models, but also for LogEntry with automatic timestamping), etc.

More in-depth utility methods include .normalize() for Message/KnowledgeGraph/Result/AuxiliaryGraph, .prune() for KnowledgeGraph, etc.

TypedDict Usage

This library also provides TypedDict models, which can be used for internal static typing without class instantiation overhead, at the cost of some code verbosity.

• *DictUtil.from_json() and *DictUtil.to_json()
• *DictUtil.from_msgpack() and *DictUtil.to_msgpack()
• Direct construction: *Dict()

JSON Reading

Unlike with models, the model_dicts don't validate by default.

from translator_tom.model_dicts import QueryDictUtil, QNodeDictUtil


query_json = """
{
  "submitter": "TOM tester",
  "message": {
    "query_graph": {
      "nodes": {
        "n0": { "ids": [ "PUBCHEM.COMPOUND:726218" ] },
        "n1": { "ids": [ "NCBIGene:3778" ] }
      },
      "edges": {
        "e0": {
          "subject": "n0",
          "object": "n1",
          "predicates": [ "biolink:related_to" ]
        }
      }
    }
  }
}
"""

query = QueryDictUtil.from_json(query_json)  # returns type QueryDict

# These key accessors now have hints+completions in type-aware editors
query_graph = query["message"]["query_graph"]
assert query_graph is not None  # Type narrowing
assert len(query_graph["nodes"]) == 2  # True
n0_ids = query_graph["nodes"]["n0"].get("ids") or []  # `ids` is optional
assert n0_ids == ["PUBCHEM.COMPOUND:726218"]  # True

# DictUtils also provide safe accessors:
n0 = query_graph["nodes"]["n0"]
assert QNodeDictUtil.ids_list(n0) == ["PUBCHEM.COMPOUND:726218"]  # True


# A 'lite' version of validation may be optionally used
# This doesn't mutate the parsed dict, but throws ValidationError if it fails.
# Significantly faster than model validation; but not as thorough
query = QueryDictUtil.from_json(query_json, validate=True)

Casting and direct instantiation

Oftentimes you'll just want to cast a model_dict:

from typing import cast

from translator_tom.model_dicts import QueryDict

query_plain = {
    "submitter": "TOM tester",
    "message": {
        "query_graph": {
            "nodes": {
                "n0": {"ids": ["PUBCHEM.COMPOUND:726218"]},
                "n1": {"ids": ["NCBIGene:3778"]},
            },
            "edges": {
                "e0": {
                    "subject": "n0",
                    "object": "n1",
                    "predicates": ["biolink:related_to"],
                }
            },
        }
    },
}

# cast is free at runtime; it only tells the type checker to treat query_plain as a QueryDict.
query = cast("QueryDict", query_plain)

You can also just pass an already-existing dict to the dict constructor, although it produces a shallow copy:

from translator_tom.model_dicts import QueryDict

# An existing dict you've annotated as a QueryDict (checked against it here).
query_plain = {
    "submitter": "TOM tester",
    "message": {
        "query_graph": {
            "nodes": {
                "n0": {"ids": ["PUBCHEM.COMPOUND:726218"]},
                "n1": {"ids": ["NCBIGene:3778"]},
            },
            "edges": {
                "e0": {
                    "subject": "n0",
                    "object": "n1",
                    "predicates": ["biolink:related_to"],
                }
            },
        }
    },
}

query = QueryDict(**query_plain)

Direct construction

You can also use the model_dicts directly as construction guides:

from translator_tom.model_dicts import (
    MessageDict,
    QEdgeDict,
    QNodeDict,
    QueryDict,
    QueryGraphDict,
)

# Each constructor provides key hints and type checking
query = QueryDict(
    submitter="TOM tester",
    message=MessageDict(
        query_graph=QueryGraphDict(
            nodes={
                "n0": QNodeDict(ids=["PUBCHEM.COMPOUND:726218"]),
                "n1": QNodeDict(ids=["NCBIGene:3778"]),
            },
            edges={
                "e0": QEdgeDict(
                    subject="n0",
                    object="n1",
                    predicates=["biolink:related_to"],
                )
            },
        )
    ),
)

Semantic Validation (WIP)

A very WIP item is Semantic Validation:

from translator_tom.validation import semantic_validate

warnings, errors = semantic_validate(some_model)  # Any TOM model

This returns a list of warnings and errors with clear descriptions and tuples describing their locations.

Warning

This feature is WIP and does not do every bit of semantic validation you might expect.

Design Decisions

There are a view caveats to using TOM, listed below:

Implicit Enums

In some sections (such as knowledge_type), only certain values are used by existing systems, despite the field being an open string. In these cases TOM explicitly defines enums, as this may help to catch early validation errors. Some areas where the TRAPI spec defines short-codes that are not well-policed do not define enums.

Literal over Enum

Python Literals are slightly faster for serialization, so internally, literals are used. Enums are still provided, largely for documentation access. All literals use the standard names you'd expect from TRAPI, while enums have Enum as a suffix.

Using None where None is allowed, despite default

In TRAPI, some properties are non-required, but default to an empty list. TOM defaults these to None, to save serialized space. This is in-line with intended TRAPI 2.0 changes, and doesn't break interoperability.

Hash calc + representation

Hashing is used in several cases, including KG normalization. TOM uses stablehash to ensure hashes are stable, and outputs hashes as unpadded base64url, with 120 bits truncation, by default. This offers a nice tradeoff of collision safety and hash shortening; all hashes are exactly 20 characters long.

Differences from reasoner-pydantic

• Extra fields do not contribute to hashes
• Knowledge Node hash does not take attributes or categories into account
• MetaAttribute hash does not take into account name fields
• Message does not auto-normalize, and results do not auto-merge. You have to manually call the appropriate methods.
• BiolinkEntity, BiolinkPredicate, and BiolinkQualifier are now sub-types on the Biolink utility class.
  • This causes one issue: BiolinkPredicate and BiolinkEntity don't show up the JsonSchema generated from these models (but the patterns are preserved )