Skip to content

fix(tests): hold the parent watcher to its time from its own start - #125

Draft
mahdi-shafiei wants to merge 1 commit into
mainfrom
fix/parent-watch-startup-window
Draft

mahdi-shafiei wants to merge 1 commit into
mainfrom
fix/parent-watch-startup-window

Conversation

@mahdi-shafiei

Copy link
Copy Markdown
Contributor

Summary

 test_a_training_process_dying_as_its_workers_start_leaves_none
-  every worker gone within 2 s of the training process's death
+  each worker gone within 2 s of max(death, its watcher's start)   # TimedSetUp records the start
+  every worker ended within 30 s                                   # as Grain's own watcher test

A worker still starting when its training process dies ends only once its set-up starts the parent watcher, after its spawn, imports and unpickling. That time grows with load, and the old test charged it to the watcher's 2 s. The 2 s bound is unchanged; only what it measures from changed. Test-only.

Evidence

  • Before: on the One naming of record fields #97 rebase run, [0] failed with {'alive': [3014490], 'gone_after': None} at a load average near 24. A probe of the test's own harness with a 30 s window, delay 0, on 24 CPUs: every worker ended; 1.44-1.60 s after the death at a load average near 12, and 1.71-3.33 s with 24 busy processes added (5 of 6 over 2 s).
  • After: TestTheParentWatcher 8 passed with 24 busy processes added (load average 28.7). Watchers started 0.70-1.69 s after the death, and each worker exited within 0.007 s of its watcher's start. test_host_worker_setup.py and test_materialize_crashes.py: 33 passed.
  • Positive control: a watcher planted 3 s late fails all four delays (3.00-3.01 s against 2.0). The existing unwatched control still finds both workers alive.

Merge Danger

Door: two-way

Blast Radius: tests only (tests/pipeline/test_host_worker_setup.py, tests/test_common/worker_reads.py, CHANGELOG).

test_a_training_process_dying_as_its_workers_start_leaves_none required
every worker gone 2 s after its training process died. A worker still
starting at that moment ends only once its set-up starts the watcher,
after its spawn, imports and unpickling, and that time grows with load.
On 24 CPUs it took 1.4-1.6 s at a load average near 12 and 1.7-3.3 s
with 24 busy processes added, so [0] failed under load (a worker alive
after 2 s) while every worker ended within 30 s.

A worker now records when its watcher starts (TimedSetUp, beside
UnwatchedSetUp), and each must be gone within 2 s of the later of the
death and that start, and end within 30 s, the wait Grain's test of its
own watcher uses (google/grain#1475). The 2 s bound is unchanged. A
watcher planted 3 s late fails all four cases; the parent-watcher tests
pass beside 24 busy processes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant