Skip to content

ci: cache test-elixir NIF correctly#5832

Merged
max-sixty merged 1 commit intomainfrom
fix/elixir-cache-priv-nif
Apr 26, 2026
Merged

ci: cache test-elixir NIF correctly#5832
max-sixty merged 1 commit intomainfrom
fix/elixir-cache-priv-nif

Conversation

@prql-bot
Copy link
Copy Markdown
Collaborator

Problem

#5826 corrected the actions/cache paths in test-elixir.yaml (which had been wrong since the bindings reorg in #3683 — the cache silently never restored). Once those paths were correct, the next nightly on main (run 24955057135) hit the saved cache and test-elixir failed:

** (UndefinedFunctionError) function PRQL.Native.compile/2 is undefined (module PRQL.Native is not available)
Failed to load NIF library: '.../prqlc/bindings/elixir/_build/test/lib/prql/priv/native/prql.so: cannot open shared object file: No such file or directory

That run-failure is what motivated the revert in #5831.

Root cause

Rustler builds the NIF via cargo into the workspace target/ and then copies it to prqlc/bindings/elixir/priv/native/prql.so (see job 73060377548 logging Copying /home/runner/work/prql/prql/target/release/libprql.so to priv/native/prql.so). _build/<env>/lib/prql/priv is just a symlink back to that priv/ — the actual .so lives outside _build. The saved cache from #5826 was only ~288KB (beam files only, no NIF).

The elixir module uses use Rustler, otp_app: :prql (a runtime-loaded NIF). When the cache restored beam files, mix saw the project as already compiled and skipped Rustler entirely, so the .so was never rebuilt and mix test failed to load the NIF.

Fix

  • Restore the path corrections from ci: correct test-elixir cache paths #5826 (prqlc/bindings/elixir/deps, prqlc/bindings/elixir/_build).
  • Add prqlc/bindings/elixir/priv to the build cache so the .so survives across runs.
  • Bump cache-name to cache-compiled-build-v2 to invalidate the stale 288KB entry that was saved during ci: correct test-elixir cache paths #5826.
  • Hash Cargo.lock and prqlc/**/*.rs (plus Cargo.toml) in the build cache key so cache invalidation tracks any change that affects the .so. Mix's beam staleness check doesn't see Rust source changes, so without this the cache would happily serve a stale .so whenever Rust code changed but mix.lock didn't.
  • Drop the restore-keys on the build cache. With a comprehensive key, any miss means something the .so depends on changed; a partial-prefix restore would deliver a stale .so. The deps cache keeps its cache-name-prefix restore-key — partial dep updates can reuse most deps.

Trade-offs vs. just reverting

  • The revert (ci: revert test-elixir cache path fix #5831) restored the prior state where the cache silently never worked — slow but correct. Test-elixir spent ~2 min per run on the Rust build.
  • This PR keeps caching working and includes the deps half (which was always correct on its own).
  • The cost is: any change under prqlc/ invalidates the build cache → ~2 min Rust rebuild on those PRs. But that's the correctness floor — anything cheaper would risk stale .so test passes.

Verification

Per prql-bot comment on #5831 and @max-sixty's request.

#5826 fixed the cache paths but #5831 reverted it because the cache
hit on main started failing test-elixir: the Rustler-built NIF lives at
prqlc/bindings/elixir/priv/native/prql.so, outside _build, so the cache
restored beam files without the .so, mix compile saw the project as
up-to-date and skipped Rustler, and tests failed to load the NIF.

Changes:
- Restore the prqlc/bindings/elixir/{deps,_build} paths from #5826
- Add prqlc/bindings/elixir/priv to the build cache so the .so is
  preserved across runs
- Bump cache-name to invalidate the stale 288KB entry from #5826
- Hash Cargo.lock and prqlc Rust sources in the build key so the .so is
  rebuilt whenever the Rust code that produces it changes
- Drop restore-keys on the build cache; a partial-prefix restore would
  deliver a stale .so on Rust changes
@max-sixty max-sixty merged commit 9c37130 into main Apr 26, 2026
41 checks passed
@max-sixty max-sixty deleted the fix/elixir-cache-priv-nif branch April 26, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants