fix: multi-gpu nodes hangs in run_openfold predict

[jandom]

Summary

Never seen this when we run single-GPU inference but observed 1st with a multi-GPU node on an HPC cluster. The inference job starts to cleanup but then hangs indefinitely, until it's killed by walltime.

Here is the Claude TLDR

On run_openfold predict with 8 ranks, every prediction completes and summary.txt is written, but the ranks never exit — each one blocks at interpreter shutdown in multiprocessing.util._exit_function, whose untimed Process.join() waits forever on a DataLoader worker, so all 8 processes sit at 0% GPU holding their memory until SLURM kills the job at the walltime. The cause is the forkserver start method hardcoded in safe_multiprocessing_context: workers are children of the forkserver rather than of the rank, so a rank can't waitpid() them itself and instead waits on a sentinel pipe for the forkserver to report the exit status — which never comes, because the forkserver sits idle in ep_poll while all ten workers pile up as unreaped zombies.

Changes

This PR for now just introduces a test that shows this failure mode

Related Issues

Testing

Other Notes

[jnwei]

I know this is still in draft, but when it's ready, could you provide a little more context about the run and the exact error message / stack trace?

For running with multiple GPUs - were you using a batch submission, i.e. a query json with multiple queries?