Skip to content

test(pipeline): the inherited-CUDA guard checks the platform workers hold - #118

Merged
mahdi-shafiei merged 1 commit into
mainfrom
fix/inherited-cuda-control
Nov 18, 2026
Merged

mahdi-shafiei merged 1 commit into
mainfrom
fix/inherited-cuda-control

Conversation

@mahdi-shafiei

Copy link
Copy Markdown
Contributor

Summary

 _INHERITED_CUDA (the test's script)
-  jax.config.update("jax_platforms", "cpu")        # module level: every spawned worker re-runs it
-  read = pipe.host_stage.read_for_workers(pipe)    # after the run was served: no units left
-  bare = MapDataset.range(2).map(read) ... mp_prefetch(...)   -> AssertionError, any platform
+  if __name__ == "__main__":
+      jax.config.update("jax_platforms", "cpu")    # the parent only
+  bare = MapDataset.range(1).map(platforms_seen) ... mp_prefetch(...)   # jax.config.jax_platforms
 test_an_inherited_cuda_platform_list_never_reaches_a_worker
-  assert report["control"].startswith("raised")
+  assert report["platforms"] == ["cpu"]           # datarax's workers
+  assert report["control"] == ["cuda,cpu"]        # a bare worker
+test_the_control_holds_whatever_list_the_worker_inherits   # cpu inherited -> ["cpu"]

The guard checks that an inherited JAX_PLATFORMS=cuda,cpu (the GPU recipe's) never reaches a datarax host-stage worker. The test passed without the guard: with jax.config.update("jax_platforms", "cpu") removed from WorkerSetUp.__call__ (host_workers.py:190) it still passed. Two causes:

  • The script set the platform at module level. A spawned worker runs the parent's main script as __mp_main__ (CPython 3.12.14 multiprocessing/spawn.py:275-300), so every worker, datarax's and the bare one, got the CPU platform from the script itself.
  • The control took read_for_workers after raw_batches had served the run. That read covers the run from where serving stopped, here its end, so the bare worker failed on run_units.py:216 (assert batches is not None) before reaching any platform.

The test now compares the platform list each worker holds. A control that waits for the CUDA start to fail was tried and dropped: jax 0.11.1 skips a listed cuda platform when no /dev/nvidia* file exists (xla_bridge.py:823, hardware_utils.has_visible_nvidia_gpu), so on a hosted runner the bare worker starts the CPU and serves, and that control would pass only on GPU machines.

Evidence

  • Before: with WorkerSetUp's jax_platforms line removed, main's check passes (1 passed); its control raises AssertionError with no message.
  • After: the same removal fails the test on this machine (RTX 4090) and, with jax._src.hardware_utils.has_visible_nvidia_gpu returning False in every process (a sitecustomize.py on PYTHONPATH, checked first to turn cuda,cpu into [CpuDevice(id=0)]), fails both this test and test_cpu_platform_gpu_hidden_no_tensorflow_spawned on the platform list: ['cuda,cpu'] == ['cpu']. Unchanged code passes in both settings.
  • tests/pipeline/test_host_worker_setup.py and every contract test: 825 passed. Every hook passes.

Merge Danger

Door: two-way

Blast Radius: one test

A test file only. #117 removes the flag lines two lines above the changed control; whichever merges second rebases.

…hold

test_an_inherited_cuda_platform_list_never_reaches_a_worker passed with
jax.config.update("jax_platforms", "cpu") removed from WorkerSetUp. Its
script set that platform at module level, which every spawned worker
re-runs (multiprocessing's spawn runs the main script as __mp_main__),
so no worker saw the inherited JAX_PLATFORMS=cuda,cpu; its control took
the read after serving the run, so the bare Grain worker failed on a
run with no units (run_units.py:216) whatever its platform.

The setting moves under the script's __main__ guard, and the test
compares the platform list each worker holds: datarax's workers hold
cpu, a bare worker holds the inherited cuda,cpu, and holds cpu when
that is inherited. A control waiting for the CUDA start to fail would
pass only where a GPU exists: jax 0.11.1 skips a listed cuda platform
when no NVIDIA device file exists, as on a hosted runner. Without the
WorkerSetUp line the test fails on a GPU machine and, with JAX's GPU
check returning False in every process, on one without.
@mahdi-shafiei
mahdi-shafiei force-pushed the fix/inherited-cuda-control branch from df4a0ac to 0e39de2 Compare November 18, 2026 01:45
@mahdi-shafiei
mahdi-shafiei marked this pull request as ready for review November 18, 2026 02:15
@mahdi-shafiei
mahdi-shafiei merged commit 1a50bfd into main Nov 18, 2026
19 of 20 checks passed
@mahdi-shafiei
mahdi-shafiei deleted the fix/inherited-cuda-control branch November 18, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant