Repository navigation
test(pipeline): the inherited-CUDA guard checks the platform workers hold - #118
Merged
Merged
Conversation
…hold
test_an_inherited_cuda_platform_list_never_reaches_a_worker passed with
jax.config.update("jax_platforms", "cpu") removed from WorkerSetUp. Its
script set that platform at module level, which every spawned worker
re-runs (multiprocessing's spawn runs the main script as __mp_main__),
so no worker saw the inherited JAX_PLATFORMS=cuda,cpu; its control took
the read after serving the run, so the bare Grain worker failed on a
run with no units (run_units.py:216) whatever its platform.
The setting moves under the script's __main__ guard, and the test
compares the platform list each worker holds: datarax's workers hold
cpu, a bare worker holds the inherited cuda,cpu, and holds cpu when
that is inherited. A control waiting for the CUDA start to fail would
pass only where a GPU exists: jax 0.11.1 skips a listed cuda platform
when no NVIDIA device file exists, as on a hosted runner. Without the
WorkerSetUp line the test fails on a GPU machine and, with JAX's GPU
check returning False in every process, on one without.
mahdi-shafiei
force-pushed
the
fix/inherited-cuda-control
branch
from
November 18, 2026 01:45
df4a0ac to
0e39de2
Compare
mahdi-shafiei
marked this pull request as ready for review
November 18, 2026 02:15
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The guard checks that an inherited
JAX_PLATFORMS=cuda,cpu(the GPU recipe's) never reaches a datarax host-stage worker. The test passed without the guard: withjax.config.update("jax_platforms", "cpu")removed fromWorkerSetUp.__call__(host_workers.py:190) it still passed. Two causes:__mp_main__(CPython 3.12.14multiprocessing/spawn.py:275-300), so every worker, datarax's and the bare one, got the CPU platform from the script itself.read_for_workersafterraw_batcheshad served the run. That read covers the run from where serving stopped, here its end, so the bare worker failed onrun_units.py:216(assert batches is not None) before reaching any platform.The test now compares the platform list each worker holds. A control that waits for the CUDA start to fail was tried and dropped: jax 0.11.1 skips a listed
cudaplatform when no/dev/nvidia*file exists (xla_bridge.py:823,hardware_utils.has_visible_nvidia_gpu), so on a hosted runner the bare worker starts the CPU and serves, and that control would pass only on GPU machines.Evidence
WorkerSetUp'sjax_platformsline removed,main's check passes (1 passed); its control raisesAssertionErrorwith no message.jax._src.hardware_utils.has_visible_nvidia_gpureturning False in every process (asitecustomize.pyonPYTHONPATH, checked first to turncuda,cpuinto[CpuDevice(id=0)]), fails both this test andtest_cpu_platform_gpu_hidden_no_tensorflow_spawnedon the platform list:['cuda,cpu'] == ['cpu']. Unchanged code passes in both settings.tests/pipeline/test_host_worker_setup.pyand every contract test: 825 passed. Every hook passes.Merge Danger
Door: two-way
Blast Radius: one test
A test file only. #117 removes the flag lines two lines above the changed control; whichever merges second rebases.