Conversation
aleksisch
force-pushed
the
aleksisch/bench-jit-native
branch
16 times, most recently
from
September 21, 2026 16:18
d01a104 to
d7c320b
Compare
… its files
run_all drives the interpreter, jit and aot lanes over tests/ at once, so up to four copies of one
dasHV test run together - and every fixture bound a single fixed port. The losers reported "bind:
Address already in use" and went on to serve from a server that never started: one run took SIGSEGV
at address 0x120 inside the tick that followed.
A server now asks for port 0 and reports the port it got, which is what every other framework does
- Go's net.Listen(":0"), Rust's TcpListener::bind("…:0"), Java's new ServerSocket(0). libHV serves
nothing for port 0, since it listens only above zero, so start() learns a free port from a socket of
its own, releases it, and binds that number; bound_port reads it back. No band, no walk, no fixed
port to lose.
The fixture publishes that port over a one-capacity channel, whose pop blocks until the value lands
- one primitive where a wait group plus an atomic used to carry the wait and the number separately.
The static and upload directories are named for the port, so two copies never share a tree - two
rewriting one 17 MB big.bin made a SERVE_FILE answer 500 from a file another run had truncated - and
they are removed with the tree, not with remove(), which never deleted a directory.
… alone Inference rewrote try/recover into builtin_try_recover(ExprMakeBlock, ExprMakeBlock) whenever jit was enabled. The rewrite renumbered every anonymous function after it, so one source hashed differently with and without -jit: the LLVM-AOT lane missed 400 functions over 52 files with error[50101], and 29 of those 52 carry try/recover against 36 of the 1696 files that link. The lowering moves to the LLVM emitter, which runs after simulate on an already-simulated program and therefore cannot move a hash. ExprTryCatch declines its own body through the new canVisitTryCatchBody visitor gate, emits each arm the way ExprMakeBlock emits a block, and calls jit_try_recover with the pair. builtin_try_recover is inlined into that runtime entry and deleted: the interpreter keeps SimNode_TryCatch and the C++ AOT keeps das_try_recover, so nothing else called it. A capture field holds a pointer, and a by-value argument the prologue did not promote exists only as an SSA parameter, so an arm that reads one spills it into the slot visitExprVar reads before it falls back to GetParam. The return-inside-try diagnostic leaves ast_lint for the emitter, where a Block's own return is already distinguishable from a return out of the function. It warns, on every rail. ast_lint asked exprReturns - does the arm ALWAYS return - so the conditional form (`return x if (cond)`) passed it and master's own rewrite miscompiled it just as quietly; the emitter sees every function-return in either arm, and the tree has them (the sequence game's dependency chain, reached through an exe build). Failing those builds would be a regression against a defect nobody introduced here, so this reports and emits what master emits. The fix is to lower the exit properly - a flag plus a branch after the call, the way daslib/aot_cpp.das does it with TryExitScanner - and that is its own change. The emitter also sees the recover arm, which the lint check missed, and skips a [no_jit] function for free. An arm becomes its own block function, so the collectors the block prologue runs - collect_finally and collect_labels (daslib/macro_boost.das) - must stop at an arm the way they already stop at a make-block body, and take skipTryCatch for it. Without that, an arm carrying a scope exit (a `var inscope` in it) has its finally basic block appended to the ENCLOSING function, where nothing fills or terminates it: `Basic Block ... does not have terminator!` on any function whose block holds such an arm, which is how the dasllama-server exe failed to verify. tests/jit_tests/try_recover_scope.das covers both arms. Two runtime seams the LLVM-AOT rail needs land beside it. SimNode answers rtti_node_jitFunction with the entry the jit node already holds, so a binder reads it without the node's definition - the aot branch beside it reads aotFunction off a SimNode_CallBase the same way. And jit_enabled() answers for the consumer of an offline object (jit_enabled && !jit_emit_object): the query folds, the emit folds it into the object, and an ordinary run binds that object - tests/jit_tests carries the shape in 64 files. The builtin is a compile policy read as a program fact, the defect is_in_aot already has, and it goes when its call sites do. The raw-quote diagnostics name the gate they have now - aot_macros - and the emitter pin follows.
Program::simulate skips fusion whenever jit_enabled is set - under the jit the interpreter's
nodes are the fallback, not the product, so fusing them is work nobody reads. Emitting an
offline AOT object is the exception: the semantic hash those objects are keyed by is taken
over that same node tree, and whoever binds them later is an ordinary run, which fuses. The
emitter keyed them by a tree nobody rebuilds, so every hash missed.
Nothing said so. The rail passed -jit to its own test binary and it worked, which read as
the jit being required to bind LLVM-emitted objects; what -jit actually did was turn fusion
off on the consumer too, and the jit then quietly recompiled whatever still missed - 992
processes paying full codegen against a 1800s watchdog, in a static binary with no DLL cache
to reuse any of it.
The emit fuses now, keeping the skip for the jit runs it was written for. run_tests_llvm_aot
loses -jit and reads like run_tests_aot: -use-aot on both sides, binding the objects of
dastest itself before `--` and of each test file after it. The harness needs objects of its
own for that - only the test files had them - so the object corpus gains dastest and the
daslib modules the C++ rail already pre-compiles, whose list moves out of that rail's guard
to serve both. json_boost and regex_boost stay out: their typeinfo(ast_typedecl) wants rtti,
which they inherit from the program requiring them, and the emitter compiles each file alone.
QuotePass loses its jit_enabled trigger in the same pass: a flag that selects a backend
should not decide what the front end emits, and aot_macros already says it.
The two C++ gates that still read the flag follow QuotePass out of the front end: ExprQuote's
noAot mark and the macro context's stack sizing read aot_macros only. The emit left a
quote-carrying function AOT-able while the consumer marked it noAot, and since linkCppAot hashes
no noAot function, every caller's folded hash moved with it - error[50101] on
tests/quote/test_quote_lowering.das and tests/json/test_sprint_json.das.
linkCppAot arms SimFunction::jitFunction from the node it binds, so das_invoke_function dispatches
an LLVM-emitted function straight into native code. Nothing else set that mirror once the rail
dropped -jit (das_instrument_jit did, as a side effect of the jit run), so invoke fell to
callOrFastcall and pushed an interpreter frame per call: tests/aot/t_invoke_void.das overflowed its
4096-byte stack at depth 150. test_sprint_json's pretty test is [no_jit] rather than skipping
itself when jit_enabled(), so it runs on both lanes instead of neither.
The lanes this commit touches also carry the sanitizer runtime options. .github/lsan_suppressions.txt
reached run_tests_jit and nothing else, so run_tests_aot, run_tests_aot_subset and
run_tests_llvm_aot ran their binary bare - and the asan cell's Slow Release Tests died on the 77
bytes uri_compose leaks, which that file already names. The env prefix is built once in
SETUP_SANITIZERS, beside the -fsanitize flags that make it necessary, and the lanes spell
${DAS_SANITIZER_ENV}; the jit lane keeps its own detect_leaks=0, a statement about a sweep that
exits through exit(). A cmake -E env prefix is the only build-time environment cmake has:
set(ENV{}) reaches configure alone, and a custom target has no environment property. On an asan
build the lane leaked 77 bytes before and runs 12461 tests with no leak report after; the tsan
cell's tray test passes for the same reason.
jit: the LLVM-AOT object corpus carries shader_lingua_franca, which darwin links
tests/spirv/test_conf_half4.das and test_workgroup_size.das failed the darwin backend sweep with
error[50101] on @shader_lingua_franca::texture, textureSize, textureLod and texelFetch over the
sampler types - and on the test's own lambdas, whose folded hash moves with them. The functions are
plain members of daslib/shader_lingua_franca.das, which no list put in the object corpus, so
nothing emitted them.
Linux never asked: nothing there marks them used. On darwin dasMetal's metal_builtins requires the
module and re-exports those names, so the program carries them and linkCppAot wants objects that
were never built. The file joins the object list beside the daslib set. Master's own nightly fails
the same two files.
LLVMParseCommandLineOptions (init_jit_clopts, llvm_jit_common.das) is the only route the C API exposes to LLVM's unroll cost-model thresholds, and that parse fills LLVM's cl::opt and ManagedStatic registries - ~100 KB that llvm_shutdown alone frees. Nothing called it: free_jit_context releases what one compile owns, the cached DLL handle or the ExecutionEngine and LLVMContext, and LLVM's process globals are a different allocation, so they were live at exit, the RelWithDebInfo allocation tracker reported them, and every successful -jit run left exit code 1. shutdown_llvm, a [finalize(late)] in llvm_macro.das, calls it. That module's macro context is where every LLVM call in the process happens; a run that only hits the DLL cache never brings the emitter in and has nothing to free. Two jit compiles in one process both succeed with the shutdown running between them. The last 40 bytes are not ours: z3's g_memory_mux, taken when ld.so runs libz3's static initializers and left alone by a delete z3 comments out on purpose. lib/LLVM.dll at sentinel 22.1.5-r4 directly needs libz3, libffi, libedit, libxml2, libz and libzstd - exactly the libraries llvm_release.yml builds with OFF - so the artifact being served is not that workflow's own output, and a z3-free prebuilt is what would remove the need for the guard dasbind's library load now holds across the load call. With both halves a -jit run reports nothing and exits 0.
…ane joins it A -jit child paid ~47 s per file to rebuild the emitter module before it timed anything, which is most of a 3 h night and none of what the lane measures. The stand now runs the compiled hosts instead: benchmarks join the LLVM-AOT corpus, so test_llvm_aot carries LLVM-emitted code for every benchmark and the jit lane runs that binary, the same way the new aot lane runs test_aot_subset. Both lanes share one shape in bench_runner - pick the host, pass -jit or -use-aot, fall back to the interpreter binary when the host is absent. sql and terminal stay out of the object corpus: they need a dynamically loaded module the emitting daslang must resolve, so they ride only a build that has it. The aot lane excludes the groups test_aot_subset does not compile, and the jit lane excludes two gc benchmarks - heap_collect cannot attribute a frame entered through a compiled frame, and force_escape_free frees a non-escaping allocation as a heap chunk under jit, which kills the child. Both are recorded with their reason, not silently dropped; the escape_free one is a real jit bug. The run clock reads get_time_nsec: get_time_usec is 32-bit and wrapped at 35.8 minutes, which is how a night reported itself as -343 s. The site gains the aot lane and a per-group chart whose arms are geometric means normalised to each arm's own first run, so a group reads as one line instead of one chart per benchmark.
run_all builds and tests in one ninja invocation, so a module relink lands while another lane is already running daslang against that module - and a relink restamps the module's descriptor manifest, which is keyed on dep size and hash, so every concurrent reader then finds the descriptor cold. tests/module_cache asserts it finds one warm and deferred, and failed. CI never sees this: build.yml builds in one step and tests in the next. ADD_DAS_SHARED_MODULE_LIB records each module on a global property, das_build_products collects them with daslang, and every lane leaf depends on it. The edge goes on the leaves, not on the groups: a group's dependency is a sibling of its members, and ninja may still run them together.
aleksisch
force-pushed
the
aleksisch/bench-jit-native
branch
from
September 21, 2026 16:38
d7c320b to
62e8e28
Compare
…cked against the tree das2rst writes skills/daslang/references/everything.md, which is tracked - skills/daslang ships to agents that have no daslang to generate it with - and nothing gated the write, so a doc lane run silently rewrote it, dropping the one-line descriptions of symbols whose modules that build did not carry. It also raced every reader: the MCP discover tool reads that file as its corpus, and tests/watchdog failed on a half-written one. The lane now generates to <build>/doc/everything.md and compares. Regenerating the tracked copy is regen_docs_digest, a target you ask for. check_doc_verify also walks doc/source, so it waits for the generators. doc: the shipped digest carries what das2rst generates today The tracked digest was last written by an older generator, and nothing compared the two until check_docs_fresh started to: the doc lane failed on a file no branch had touched. Regenerated through regen_docs_digest against the doc lane's own configuration. It gains executable_on_path and prepare_processed_catalog, which reached master while no gate watched, and loses the one-line text of constants carrying a TRAILING `//!` (INVALID_ENTITY_ID, jsonrpc's five error codes, LCG_RAND_MAX, _fold): the generator emits no description for those into the RST the digest reads, so the text survived only in the stale copy. Harvesting a trailing `//!` is its own change; this file states what the pipeline produces.
The trap covers a normal exit and an interrupt, and a run killed outright leaves lib/ hidden - but recovering another run's leftovers at startup is not this script's job.
doc-verify compiles a page's das blocks as one program, and the clone block took `clip` from
the block above it, where it is declared inside `if (c.cloning) { ... }` - out of scope by the
time the prompt block runs, so the page was red on `can't locate variable 'clip'`. The block
loads the clip itself now, which is also what a reader copying it needs.
…checks the digest das2rst defaults --digest to skills/daslang/references/everything.md, which is tracked because skills/daslang ships to agents with no daslang to generate it with. check_docs_generated passes the flag; pages.yml and preflight's docs gate ran the generator themselves, so each run rewrote the shipped file - and preflight's gate reads the run's output and nothing else, so the write bought it nothing. Both call the target now, which is the one place the flag lives. preflight also gains docs/digest. check_docs_fresh compares the generated digest against the tracked copy in CI's doc lane and nothing here mirrored it, so a stale digest reached CI as a red doc lane with no local gate to catch it first.
…prefix
MSYSTEM_PREFIX names the subsystem root, /clang64, and only the msys2 shell has it. The step set
DAS_CLANG_INCLUDE through a ${{ env.* }} expression, which GitHub evaluates before that shell
exists, so it resolved to the msys2 root and the binder read D:/a/_temp/msys64/include - no
clang-c/Index.h there, and the lane has been red on every nightly since. The export moves into the
shell, cygpath hands the native daslang.exe a Windows path, and the step fails on the header rather
than on the parse.
… what no run touched Runs sharing a .jitted_scripts directory corrupted each other two ways. A partition object is probed with stat, which turns true the moment a peer CREATES the file, so a second run linked a half-written object and its dll came back missing functions - error "reopened <dll> disagrees with what was just written - 86 function(s) missing". And two runs of the same program link the same dll path, so one truncated the file the other was loading - "linked but cannot be loaded back". Both reproduce in four rounds of three concurrent dastest runs; both are what the extended-checks llvm_vector_math gate died of. The emit and the link land on a private name beside the target and are renamed in. rename is atomic within a directory: a reader sees the whole artifact or none of it, and a peer that published first wins - its file is complete by construction, so ours is dropped. The GC also stopped deleting artifacts belonging to a run it cannot see: a name outside the keep mask is stale only if nothing touched it for ten minutes.
…n http client The two live_host tests failed the tsan lane on a ready wait that expired, and the host was not what took the time: its own log says "serve-ready 1635 ms after launch" while the parent first saw /status answer 47, 96 and 147 seconds later across runs. Every poll built an http client and an event loop to ask one question, and under tsan that construction is the whole cost - one poll measured 64 s, and the answer arrived when a client happened to come up in time, hence the spread. probe_local_port binds a throwaway socket instead, so the wait polls the port and pays for an http client once, to confirm the 200. The same probe replaces the /status scan that picks a free port - a bind answers "is anyone there" for a port nobody has claimed too. Ready lands at 3.6 s under tsan, the budget stays 30 s, and the plain build is unchanged at half a second.
run_backend_sweeps re-emitted every jitted library and re-launched every context on a tree nothing had touched: both sweeps were plain custom targets, which ninja has no way to skip. They join add_das_target, so the daslang runs inside the jit sweep report what they read and the stamp carries a depfile. A second run over an unchanged tree falls from 533 s to 1 s - what remains is the C++ sweep, whose binary writes no depfile and therefore still runs, in about a second.
ADD_EXAMPLE_RUN printed a bare MESSAGE per call site, so a configure that registers thirty examples printed thirty lines nobody reads. The list it builds is what matters, and run_examples names every file it drives.
The rebase moved the branch past sixty-four commits of stdlib change, so the digest no longer matched what das2rst generates and the Doc lane's freshness gate failed. Regenerated, dasHV's new bound_port included.
…changed stage_doc_assets ran gh release download --clobber, so every doc build refetched all 68 MP4s - 29 seconds of network for a set that has changed once since July. A staged file's mtime is when it was downloaded, so the release held its then-current bytes at that moment and only an asset uploaded since reads newer. One metadata call answers it for the whole set: an unchanged tree now costs 1.3 seconds and no download.
aleksisch
force-pushed
the
aleksisch/bench-jit-native
branch
from
September 21, 2026 17:12
573ad8e to
1fed436
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The jit lane cost three times the interp lane and almost none of it was measurement. Every child process rebuilt the JIT emitter -
llvm_jit.dasand the ~52 files of dasLLVM around it - before it could compile a benchmark, and the bench stand spawns one process per file.The DLL cache is keyed per script, so every file pays the emitter build again. Across 68 benchmarks that is ~53 minutes a night spent compiling the emitter rather than measuring anything.
A standalone dastest carries the emitter as compiled code. What it did not carry was the llvm/dasbind machinery a JIT compile needs, and
jit_enabled()is true inside any exe whether the machinery is there or not. Having the exe detect its own capability does not work:has_module("llvm")is false even under a working-jit, because the llvm module loads on the first JIT compile - nothing distinguishes the two before one is attempted. So the host declares it:--jit-register-all-modulesat build time,--jitat run time.jitandlintalready take that build flag for the same reason; dastest joins them.The guard is a pure function of the suite context,
tests_jit_enabled(ctx), not module state behind a setter.One benchmark file, same arms:
daslang -jitarray_rw/100000array_sum/100000Still jit numbers - the interpreter reads 6.2 and 4.8 on those arms, which is also what the exe reports when
--jitis absent, so the flag is a real switch rather than always-on.The lane takes that host when the tree has one and falls back to
daslang -jitwhen it does not, so a tree that never built it still runs. A benchmark requiring a dynamically loaded module - pugixml, sqlite, terminal - cannot run from an exe that links none of them and takes the same fallback; that is five files of seventy, found by the child's own error rather than by a list that would drift. The interp lane keeps the daslang host: its startup was ~2.5 s, so the exe buys nothing there and the fallback retries cost more than they save.An aot lane. daslang runs three ways and the stand measured two, leaving the tier a shipped game executes unplotted. The lane spawns
test_aot_subsetunder-use-aot. What is compiled into that host is what it can measure -tests/aot/CMakeLists.txtregistersbenchmarks/core/jit_aotand nothing else underbenchmarks/- so the other groups are excluded per group with their reason. A file outside the set is skipped in 0 s:--use-aoton a file with no stubs would either burn 76 s failing or quietly file interpreter numbers under an AOT label.The nightly clean kept deleting the module binaries. It names where build output lives -
build,bin,lib,.jitted_scripts- but a module's binary is written beside its sources asmodules/<name>/*.shared_module. The build rewrites them a minute later so nothing notices; underBENCH_STAND_BUILD=skip, the documented local dry run, nothing does and the tool cannot load a module at all.What the lane's own CI runs turned up
Standing the lane up against the full matrix surfaced defects in the tiers it measures and in the lanes around it. They are separate commits.
JIT correctness.
try/recoverwas lowered during inference, which mutated the AST the semantic hash is taken over, so an AOT object and its consumer disagreed -error[50101]: AOT link failed. Lowering moved to the backend, leaving infer's AST untouched. An emitted AOT object is now keyed by the tree its consumer rebuilds rather than by the emitting tree, and the emitter's finalizer shuts LLVM down so a-jitrun exits clean instead of at a destructor.The JIT DLL cache corrupted itself under concurrency. Objects were written in place and collected by age, so one run read a partially written artifact another run was still emitting, and the collector deleted work in flight. Artifacts now publish by
rename(2)and the collector only removes names the run itself published - nothing else is its to delete. Verified with twelve concurrent copies.dasHV fixtures fought over ports.
run_alldrives the interpreter, jit and aot lanes overtests/at once, so up to four copies of one dasHV test run together, and every fixture bound a fixed port. The losers reportedbind: Address already in useand then served from a server that never started - one run took SIGSEGV at 0x120 in the following tick. A server now asks for port 0 and reports the port it got, the way Go'snet.Listen(":0"), Rust'sTcpListener::bind("…:0")and Java'snew ServerSocket(0)do. libHV listens only above zero and itslistenfdis an input it adopts rather than an output, sostart()learns a free port from a socket of its own, releases it, and binds that number;bound_portreads it back. The fixture publishes the port over a one-capacity channel whose pop blocks until the value lands - one primitive where a wait group plus an atomic used to carry the wait and the number separately.The live-host wait was asking the wrong question. It built an HTTP client per poll; under load one poll took 64 s, so a host ready in 1.6 s was seen at 47 s, 96 s or not at all. It now asks the port whether anything is listening.
Lanes that were doing work twice
run_docinvoked das2rst twice, racing itself; the callers outside cmake run the target instead, and preflight checks the digest.stage_doc_assetsrefetched all 68 tutorial MP4s on every doc build, 29 s of network for a set that has changed once since July. A staged file's mtime is when it was downloaded, so only an asset uploaded since reads newer; one metadata call answers it for the whole set, and an unchanged tree now costs 1.3 s and no download..shared_moduleanother lane is relinking.lib/at startup.ADD_EXAMPLE_RUNprinted a line per call site at configure time; the list it builds is what matters.Testing
utils/benchctl/testsand the lanes were exercised directly: the jit lane overbenchmarks/microwent 618 s to 390 s with three of its eight files on the pugixml fallback, and per file the fast path ran 63.4 -> 4.5 s, 50.3 -> 3.7 s, 60.5 -> 6.7 s, 119.1 -> 25.5 s, 130.2 -> 30.2 s. The aot lane runscore/jit_aot/primitives.dasgreen with 37 arms and skips a non-registered file in 0.0 s with its reason.The dasHV suites pass singly and under twelve concurrent copies, with no fixture directory left behind. The DLL cache passes twelve concurrent copies with no panic.
check_docs_freshreports the generated documentation fresh, andcheck_lint_changedreports 0 issues over this branch's files.A full local
run_allhas not completed end to end - it was interrupted several times while this branch was being rebased and reworked. CI is the check that matters.#nightly