Skip to content

serialize: a builtin module's cache hash covers its content - #4190

Merged
borisbat merged 3 commits into
masterfrom
aleksisch/strengthen-serialization
Oct 5, 2026
Merged

borisbat merged 3 commits into
masterfrom
aleksisch/strengthen-serialization

Conversation

@aleksisch

@aleksisch aleksisch commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Behavior change: an existing module cache misses once per builtin module whose hash now covers more, and rewrites itself on that run.

Why. A cache record pinned each builtin module by a hash of its function mangled names only. A changed result type, argument name, default, side effect, constant, enum or handled-type layout served a stale record. A hash miss was also treated as drift and never rewritten, so a real change left the cache partial forever.

What changes.

  • The serializer hashes each builtin module's content: function signatures, results, argument names and defaults, flags, globals, enums, structures, aliases, handled-type layout and traits - through the existing semantic hashes where one exists.
  • dasbind hashes its C++ content once at construction, before a registrar's compile adds [extern] proxies, whose count grows during a run.
  • A hash miss resumes like any failed record, and the writeback repairs the stream.

Observable behavior.

  • C++ signature change -> before: stale record served; after: one reparse, then clean.
  • Module compiled between two [extern] registrars -> before: reparsed every run; after: served.

Where to look. Module::getOwnSemanticHash in ast_module.cpp, dasbind's override in module_builtin_dasbind.cpp, AstSerializer::builtinHash in module_builtin_ast_serialize.cpp.

Cost. The full hash is computed once per builtin module per serializer, then served from the serializer's cache. Measured on a script requiring 26 daslib modules (60 records, 10 builtin modules), warm run, Release, Linux x64:

builtin module functions full hash
ast_core 526 ~1.15 ms (mostly handled types)
$ 2343 ~0.57 ms
rtti_core 225 ~0.18 ms
sqlite 237 ~0.13 ms
math 455 ~0.074 ms
strings, fio_core, pugixml, debugapi, jobque 110-155 each 0.05-0.07 ms each
all 10 ~2.4 ms per run

Each record after that pays a cache hit per builtin module in its library: about 0.1 us per hit, 0.55 us per record. Warm wall time 0.39-0.41 s.

#nightly

Validation, claims, ledger

Validation

  • tests/module_cache 73/73, modules/dasLLVM/tests/llvm_cache_selfheal.das, tests-cpp-small env-serializer and dasbind cases 6/6, test_aot_subset on tests/language 1839/1839 (the enumeration hash move keeps every AOT hash).
  • Negative-controlled: the $ constant test fails with globals dropped from the hash; test_extern_between_registrars_served fails with isDasbindProxy disabled (run with an earlier fixture set; the final fixtures pass, their control run was cut short by a concurrent build).

Not done

  • ast_core's handled types are half the cost: each annotation's semantic hash walks its field types again with fresh dependency sets. Sharing the walk, or using the seal-time ownSemanticHash, would cut it.
  • [extern] registrars are still reparsed every run, as on master. Serving them (the JIT's llvm_func.das has 1154 externs) needs their proxies replayed from the record, which is separate work.
  • Not hashed: C++ behavior behind an unchanged declaration (annotation hooks such as makeFieldType), builtin macros, keywords and options.

@aleksisch
aleksisch force-pushed the aleksisch/strengthen-serialization branch from a3c74d7 to 2af965c Compare October 2, 2026 19:57
@aleksisch aleksisch changed the title serialize: a builtin module's cache hash covers its content; a module's compile-time state rides in the record serialize: a builtin module's cache hash covers its content Oct 2, 2026
@aleksisch
aleksisch force-pushed the aleksisch/strengthen-serialization branch 5 times, most recently from d09228a to 2aca2b8 Compare October 5, 2026 10:28
@aleksisch
aleksisch requested a review from borisbat October 5, 2026 10:33
TypeDecl::getSemanticHash and TypeDecl::getOwnSemanticHash each hashed an enumeration type
inline - its mangled name, base type, and every entry's name and folded value. Both call
Enumeration::getOwnSemanticHash now, which feeds the same bytes in the same order, so every
hash built on a TypeDecl keeps its value.
A record pinned each builtin module by a hash of its function mangled names only, so a changed
signature, default, flag, constant, enum, structure or handled type served a stale record.
Module::getOwnSemanticHash hashes all of it, computed once per serializer. dasbind hashes its
C++ content once at construction, before a registrar's compile adds [extern] proxies. A hash
miss now resumes like any failed record and the cache rewrites itself.
@aleksisch
aleksisch force-pushed the aleksisch/strengthen-serialization branch from 2aca2b8 to 0fc9e80 Compare October 5, 2026 15:28
@borisbat
borisbat merged commit 17c7393 into master Oct 5, 2026
64 checks passed
@borisbat
borisbat deleted the aleksisch/strengthen-serialization branch October 5, 2026 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants