Pin namespaces at attach time to fix race with container teardown

[liayan]

We hit this running large multi-node inference deployments: a setup step
creates a container on each node, several worker processes then attach to
it. Every so often a single worker on an otherwise healthy run would die
during startup, and since these are gang-scheduled distributed workers, that
one failure tore down the whole job — on an 18-node/72-worker run, 71 healthy
workers already registering with the control plane got killed because of one.

Problem

user_init() checks whether a named container is already running and stores
its PID in shm->ns_pid; task_init() later opens /proc/<ns_pid>/ns/* to join
those namespaces. Nothing holds a reference to that PID in between, so if the
step owning the container exits during that window, task_init() fails:

pyxis: ignoring --container-mounts when attaching to a running container error: pyxis: unable to open user namespace file: No such file or directory error: pyxis: couldn't get container namespaces error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1

Easy to hit when several job steps attach to one named container
concurrently, e.g. a setup step creates it and worker steps then attach to
it. Only some of the attaching steps fail, since it comes down to timing.
Also a PID-reuse hazard: if ns_pid gets recycled before task_init() runs, the
open succeeds against the wrong namespaces.

5f3f3a6 originally got the namespace fds right in user_init(), as soon
as it knew the PID was alive, for both the create and reuse cases. 41af3a0
moved container creation into task_init() to get it under
the job's cgroup, and took the fd lookup for both paths along with it — the
create path needed that move, the reuse path didn't.

Fixes: 41af3a0 ("Move container create/start to slurm_spank_task_init")

Change

Open the namespace fds in user_init(), right after enroot_container_get()
confirms the container is running, instead of waiting until task_init(). An
open fd on /proc//ns/ pins the namespace even after the owning
process exits, and it's inherited across fork(2) into the task processes, so
task_init() just skips the lookup when reuse_ns is already set.

Only affects the attach path; container creation is unchanged since ns_pid
there is pyxis's own freshly forked process.

Testing

Reproduced against unpatched Pyxis on a live Slurm+Pyxis/Enroot cluster,
using ordinary job submissions only — no plugin swap or slurmd restart. One
step holds a named container and exits after a fixed sleep, while concurrent
steps attach to it with no --container-image:

owner sleep (s)	attach attempts	ENOENT failures	rate
0.5s	480	416	86.7%
1s	480	224	46.7%
1.5s	480	32	6.7%
2s–5s	480 each	0	0.0%

672 / 3360 total attach attempts failed with the exact error chain above.
Failure rate peaking at an intermediate sleep duration and dropping to zero
at both extremes matches a timing window, not scheduling noise.

A/B'd baseline vs. this fix in an isolated single-node Slurm+munge+enroot
environment (same image, same parameters, only the plugin binary differs):

	baseline (main)	this fix
ENOENT failures	48 / 1440 (peak 20% at 1.5s owner sleep)	0 / 2520

[liayan]

Hi @flx42, I’m wondering if you could help review this. We hit this race condition pretty often in multi-node inference runs, and currently have to add a ~3-minute sleep before each job as a workaround, which is quite annoying. Would really appreciate your help taking a look.