The module-cache record is a third of its size and writes twice as fast: per-record tables for strings, types, files, modules and expression classes, delta-coded LineInfo, adaptive-size integers, an inline write path

[borisbat]

Rebuild required: the module-cache stream format changes (version 211) - every binary that includes ast_serializer.h (daslang, daslang-live, the C API, external modules) must be rebuilt together; older cache records are rejected and rewritten on first run.

Why. A module-cache record was a third strings, a fifth 8-byte file offsets and two fifths type payloads, most of them repeats: the JIT module graph behind hello_world -jit took 9.2 MB and 37 ms to write.

What changes.

• Strings, types, files, modules and expression classes stream as per-record first-mention numbers; the string table points at the AST's own strings, the type table keys the live type and compares its content, and the reader deep-copies a type per reference so no two nodes share one.
• A LineInfo is coded against the previous one in the record; 32-bit integers stream as adaptive sizes.
• The writer appends through an inline path on a doubling buffer; flush() trims it, and every consumer calls it.
• The reader accounts finalize (annotations, macro simulate, gc) and program setup apart from the decode, through two rtti externs and the -log-compile-time cache line.
• DAS_SERIALIZE_PROFILE, off by default, prints a size and time breakdown of a stream per frame, with string and type histograms and per-record sizes.

Observable behavior.

• hello_world -jit record: 9.2 MB, write 37 ms, decode 26 ms -> 2.4 MB, 16 ms, 19 ms; plain hello_world 396 KB -> 157 KB; the CI ser/deser sweep stream 265 MB -> 133 MB.
• A cache written by an older binary reads as stale and is rewritten; nothing else a user sees changes.

Where to look. src/builtin/ARCHITECTURE.md sections 6 and 7 carry the encoding; the type table's numbering after the payload and the fresh-entry count apart from the map in operator<<(TypeDeclPtr&) are the two spots that bit during development, both pinned by the warm AOT generator test.

Validation, claims, ledger

Validation

• The CI ser/deser sweep (--ser then --deser over tests/) run locally: 1404 programs, 12650 tests, 0 failed, on daslang and daslang-live rebuilt from this tip.
• The REVIEW.das gates ran under this tree's bin/daslang, not the PATH binary.
• Woodpecker round on the pre-audit tip: no findings, trace covered every changed file; not re-run after the audit batch (renames, enums, constants, a flush guard, tests).
• The eviction test's fixture requires strings_boost, fio and ast so a record measures 386 KB on this box - two fit the 1 MB test limit, three do not; retune the ballast when a record moves again.
• hash_combine64 for the type key measured 4 ms slower than the FNV mix on the JIT-graph write (19.9 vs 15.9 ms); the FNV mix stays.

Claims - stated, not tested

• A type carrying fixedDimExpr or typeMacroExpr streams inline (number 1); no fixture counts how often the sweep reaches that arm.
• The record header's locals in trySerializeProgramModule go through serializeTemp; a plain read there is a use-after-free only an ASAN lane would show.
• A module first mentioned under ignoreEmptyExternal and mentioned again outside it fails the record; no fixture produces that order.

Not done

• A corrupt-stream fixture table for the seven new rejection guards (the existing corrupt-stream case covers them generically).
• A nightly compile with DAS_SERIALIZE_PROFILE=1; the rail is compile-time and no lane defines it.
• Folding typeTableable into TypeDecl::isExprType (different question; isExprType drives inference), the first-mention skeleton into one template (three payload hooks on a hot path) and sameTypeContent into isSameExactType (the serializer's relation counts module, inference's must not).
• Self-review findings on pre-existing rules the diff did not touch (src/misc job_que wording, tests/REVIEW.md dasbind probe wording, tests/REVIEW.md opening block) and the lint candidates the audits named: a REVIEW.das check that every per-record table clears in clearNodeIds, a token scan for the libc float classifiers in luau_float2string.cpp, a flush()-before-buffer check in daScriptC.cpp.

[Copilot]

🟡 Changes recommended

The review identified a few correctness issues in newly-added/modified code paths (test census value truncation, potential empty-string read UB, and flush behavior when writePos==0) that should be fixed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR updates daScript’s module-cache serialization format (version bump to 211) to significantly reduce record size and improve write performance by introducing per-record dedup tables (strings/types/files/modules/expression classes), delta-coding LineInfo, and adaptive-size integer encoding, plus an optional compile-time serialization profiling rail.

Changes:

• Redesign module-cache record encoding/decoding with per-record tables, varint/zigzag for 32-bit ints, and delta-coded LineInfo; add explicit flush() points to ensure byte-exact buffers.
• Add serialization profiling infrastructure (DAS_SERIALIZE_PROFILE) and expose deserializer phase timings via RTTI (ast_serializer_finalize_usec, ast_serializer_setup_usec), with documentation updates.
• Extend/retune module-cache tests and review checklists to cover the new invariants (record size ballast, position census, flush requirements, string lifetime rules).

File summaries

File	Description
tests/REVIEW.md	Updates checklist routing language for module resolution/loading/cache tests.
tests/module_cache/test_rtti_serializer.das	Adds RTTI serializer round-trip test with phase-timer assertions.
tests/module_cache/test_default_cache_path.das	Retunes eviction test fixture by adding ballast requires and clarifying assertion text.
tests/module_cache/REVIEW.md	Refines module-cache test checklist wording (assertion literals, helper comparison fields, LINT030 policy).
tests-cpp/small/test_env_serializer.cpp	Adds a position census to validate LineInfo/type at stability across cold vs warm cache runs.
src/misc/REVIEW.md	Extends review rules around float classification and adds a flush-before-buffer handoff rule for C API serialization.
src/misc/daScriptC.cpp	Flushes serialization storage before returning pointer/size to callers.
src/builtin/REVIEW.md	Updates and expands serializer-related review rules (version bump, string lifetime, per-record table clearing, flush discipline).
src/builtin/module_builtin_rtti.cpp	Exposes new RTTI externs for deserializer finalize/setup timing accessors.
src/builtin/module_builtin_ast_serialize.cpp	Implements new encoding: per-record tables, varint decode fast-path, delta-coded LineInfo, profiling frames, and added flushes.
src/builtin/ARCHITECTURE.md	Documents the new module-cache record byte encoding and the profiling rail.
src/ast/REVIEW.md	Adjusts checklist text and adds a rule about serializeTemp for header-read string locals.
src/ast/ast_parse.cpp	Adds finalize-time accounting and per-record profiling rows; uses serializeTemp for header string reads.
src/ast/ARCHITECTURE.md	Documents the decode/finalize/setup timing split and how it’s reported.
skills/internal/build_and_debug.md	Adds instructions for enabling and interpreting serialization profiling output.
skills/daslang/references/everything.md	Documents new RTTI functions for finalize/setup microsecond counters.
include/daScript/ast/REVIEW.md	Clarifies checklist phrasing around bindings/field offsets and JIT cache versioning.
include/daScript/ast/ast_serializer.h	Adds flush()/asVector(), inline append path, per-record table state, profiling structs/macros, and bumps serializer version to 211.
doc/source/stdlib/handmade/function-rtti-ast_serializer_setup_usec-0xb312e3fc052a000e.rst	Adds stdlib doc stub for ast_serializer_setup_usec.
doc/source/stdlib/handmade/function-rtti-ast_serializer_finalize_usec-0x4aa223d6ebfe0792.rst	Adds stdlib doc stub for ast_serializer_finalize_usec.
doc/reflections/das2rst.das	Ensures RTTI docs generator groups the two new RTTI functions.

Review details

• Files reviewed: 21/21 changed files
• Comments generated: 4
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

The new position census test currently packs LineInfo fields into a single 64-bit word without masking (risking collisions for larger columns/lines), and there is a verified typo in a new/modified diagnostic string.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

tests-cpp/small/test_env_serializer.cpp:56

• PositionCensus::record packs line/column ranges into a single 64-bit word without masking, so if any field exceeds its implicit bit budget (e.g., column/last_column >= 1024) the values overlap and the census can miss real mismatches. Store full 32-bit fields (or mask+assert) so the test reliably detects LineInfo corruption.

    void record ( const LineInfo & at ) {
        positions.push_back(uint64_t(at.line) << 40 | uint64_t(at.last_line) << 20 | uint64_t(at.column) << 10 | uint64_t(at.last_column));
        positions.push_back(uint64_t(hash_blockz64((const uint8_t *) (at.fileInfo ? at.fileInfo->name.c_str() : ""))));
    }

• Files reviewed: 21/21 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It changes the core module-cache serialization format and timing/accounting across multiple subsystems, requiring careful human verification of compatibility and correctness beyond automated review.

Review details

• Files reviewed: 21/21 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

The changes substantially alter a core serialization format and its performance-critical implementation, so a final human review is warranted despite added tests and documentation.

Review details

• Files reviewed: 21/21 changed files
• Comments generated: 0 new
• Review effort level: Lite