Skip to content

Commit 0cd7c8d

Browse files
committed
tests/conversions: fence the per-rank sub-world before it is destroyed
tiles_of_array_unit_blocking and tiles_of_arrays_non_unit_blocking build, when the world has more than one rank, a one-rank sub-world per rank and split the fused array into arrays living in it. Those arrays are only lazily destroyed: DistArray's lazy_deleter schedules a lazy_sync task on the array's own world, i.e. on the sub-world's task queue. The tests then returned and destroyed the sub-world with that task possibly still queued; it ran later, from an unrelated fence, against a freed World, and the WorldObject destructor aborted -- the asynchronous "Abort trap" in the two-rank macOS Release Pthreads CI job (master f3e3c58, PR #583), logged while the next test case was already running. lazy_deleter's own comment describes the same hazard for einsum's per-Hadamard sub-worlds. Both tests now fence the sub-world (and the global world) before returning, so the deferred deletions complete while the world is alive. The race did not reproduce locally in eight two-rank runs of the suite before or after; the CI job hits it in a fuller task queue.
1 parent 733266c commit 0cd7c8d

1 file changed

Lines changed: 20 additions & 0 deletions

File tree

‎tests/conversions.cpp‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -318,6 +318,16 @@ BOOST_AUTO_TEST_CASE(tiles_of_array_unit_blocking) {
318318
check_equal(b_sparse, b_sparse_fused);
319319
}
320320
}
321+
// The per-rank sub-world (`world_ptr`, size > 1) dies when this test
322+
// returns, but the arrays split into it are only LAZILY destroyed:
323+
// DistArray's lazy_deleter schedules a lazy_sync task on the array's own
324+
// world, i.e. on this sub-world's task queue. Fence the sub-world so those
325+
// deferred deletions complete while it is alive; otherwise the task runs
326+
// later, from some unrelated fence, against a freed World and the
327+
// WorldObject destructor aborts (seen as an asynchronous "Abort trap" in the
328+
// two-rank CI run, attributed to whichever test case runs next).
329+
this_world.gop.fence();
330+
(*GlobalFixture::world).gop.fence();
321331
}
322332

323333
#if 1
@@ -422,6 +432,16 @@ BOOST_AUTO_TEST_CASE(tiles_of_arrays_non_unit_blocking) {
422432
check_equal(b_sparse, b_sparse_fused);
423433
}
424434
}
435+
// The per-rank sub-world (`world_ptr`, size > 1) dies when this test
436+
// returns, but the arrays split into it are only LAZILY destroyed:
437+
// DistArray's lazy_deleter schedules a lazy_sync task on the array's own
438+
// world, i.e. on this sub-world's task queue. Fence the sub-world so those
439+
// deferred deletions complete while it is alive; otherwise the task runs
440+
// later, from some unrelated fence, against a freed World and the
441+
// WorldObject destructor aborts (seen as an asynchronous "Abort trap" in the
442+
// two-rank CI run, attributed to whichever test case runs next).
443+
this_world.gop.fence();
444+
(*GlobalFixture::world).gop.fence();
425445
}
426446
#endif
427447

0 commit comments

Comments
 (0)