macOS: GC suspension hangs forever when an activation signal carries another process's si_pid

[biqas]

Description

On macOS a process can hang in GC suspension forever. The suspending thread
loops in ThreadSuspend::SuspendAllThreads, one thread keeps running managed
code in cooperative mode, and every other thread waits in
Thread::RareDisablePreemptiveGC. The running thread has
Thread::m_hasPendingActivation == true, and the runtime never sends it
another SIGUSR1: Thread::InjectActivation returns early while the flag is
set. A thread in that state is only stopped if it reaches a GC safe point by
itself. A busy-wait loop never does, so the GC waits forever. We hit this in
MSBuild's ParallelWorkSet.WaitForAllWorkAndComplete, whose worker tasks were
blocked by the pending GC.

The flag stays set when inject_activation_handler rejects the activation
because siginfo->si_pid is neither getpid() nor 0. XNU keeps si_pid per
process (struct proc), not per signal:

• psignal_internal (bsd/kern/kern_sig.c) stores the sender's pid in
  p->si_pid for every caught signal except SIGCHLD;
• proc_exit (bsd/kern/kern_exit.c) stores the exiting child's pid in the
  parent's p->si_pid before posting SIGCHLD;
• sendsig (bsd/dev/arm/unix_signal.c) copies p->si_pid into the
  siginfo of whichever caught signal it delivers next, then zeroes it.

So an activation delivered while a child's exit, or a signal from another
process, is pending carries that other pid. The existing si_pid == 0
exception covers two activations racing each other, not this case.

.NET 8 is not affected in practice: InjectActivation sends a new signal on
every hijack pass there. Since .NET 9 the Unix path skips the signal while
m_hasPendingActivation is set, so one dropped signal is final. The flag is
only cleared by the activation handler or when the thread waits for a GC
(RareDisablePreemptiveGC, "in case a signal is lost").

Reproduction Steps

// Minimal reproducer: GC suspension hangs on macOS when a thread-activation signal is dropped.
//
// The main thread busy-waits like MSBuild's ParallelWorkSet: its loop calls a method that returns
// at once and never allocates, blocks or calls out, so only an activation signal can stop it for a
// GC. Worker threads allocate, which triggers GCs while the main thread spins. Two threads start
// and reap /usr/bin/true in a loop; every child exit stores the child's pid in the process-wide
// si_pid that XNU reports in the siginfo of the next caught signal.
//
// Expected: a line "R <round> <ms>" every few milliseconds until --seconds elapse.
// Actual on macOS with .NET 9 and later: the output stops within seconds. The process then uses one
// core (the spinning thread), the GC thread loops in ThreadSuspend::SuspendAllThreads, and the
// spinning thread has Thread::m_hasPendingActivation set but never receives another SIGUSR1.
//
// Options: --seconds <n> (default: run until stopped), --no-churn (no child processes).
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Runtime.CompilerServices;

var seconds = double.MaxValue;
var churn = true;

for (int i = 0; i < args.Length; i++)
{
    if (args[i] == "--seconds")
    {
        seconds = double.Parse(args[++i], System.Globalization.CultureInfo.InvariantCulture);
    }
    else if (args[i] == "--no-churn")
    {
        churn = false;
    }
}

Console.WriteLine($"pid={Environment.ProcessId} runtime={System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription} churn={churn}");

if (churn)
{
    for (int i = 0; i < 2; i++)
    {
        new Thread(static () =>
        {
            var startInfo = new ProcessStartInfo("/usr/bin/true") { UseShellExecute = false };

            while (true)
            {
                using var process = Process.Start(startInfo)!;
                process.WaitForExit();
            }
        })
        { IsBackground = true }.Start();
    }
}

var workers = Environment.ProcessorCount - 1;
var work = new BlockingCollection<Counter>();

for (int i = 0; i < workers; i++)
{
    new Thread(() =>
    {
        foreach (var counter in work.GetConsumingEnumerable())
        {
            Allocate();
            Interlocked.Decrement(ref counter.Pending);
        }
    })
    { IsBackground = true }.Start();
}

var queue = new ConcurrentQueue<object>();
var stopwatch = Stopwatch.StartNew();

for (long round = 1; stopwatch.Elapsed.TotalSeconds < seconds; round++)
{
    var counter = new Counter { Pending = 2 * workers };

    for (int i = 0; i < 2 * workers; i++)
    {
        work.Add(counter);
    }

    Spin(counter, queue);
    Console.WriteLine($"R {round} {stopwatch.ElapsedMilliseconds}");
}

return 0;

[MethodImpl(MethodImplOptions.NoInlining)]
static void Spin(Counter counter, ConcurrentQueue<object> queue)
{
    while (Interlocked.Read(ref counter.Pending) > 0)
    {
        Poll(queue);
    }
}

[MethodImpl(MethodImplOptions.NoInlining)]
static void Poll(ConcurrentQueue<object> queue) => queue.TryDequeue(out _);

static void Allocate()
{
    var survivors = new List<object>();

    for (int i = 0; i < 4096; i++)
    {
        var chunk = new object[16];

        for (int j = 0; j < chunk.Length; j++)
        {
            chunk[j] = new string('x', 12);
        }

        if ((i & 63) == 0)
        {
            survivors.Add(chunk);
        }
    }

    GC.KeepAlive(survivors);
}

internal sealed class Counter
{
    public long Pending;
}

Project file: an SDK console project with
<TargetFrameworks>net11.0;net10.0;net9.0;net8.0</TargetFrameworks>,
<ImplicitUsings>enable</ImplicitUsings> and <Nullable>enable</Nullable>.

Run it on macOS (Apple silicon) with dotnet run -c Release -f net10.0.

Expected behavior

A line R <round> <ms> every few milliseconds, indefinitely.

Actual behavior

The output stops. With the program built as above and run 10 times for up to
30 s each on the configuration below, the last progress line came after:

Runtime	Stalled runs	Seconds until the output stopped
11.0.0-rc.1.26425.128	10 of 10	0.1–3.3
11.0.0-preview.7.26381.103	10 of 10	0.3–1.1
10.0.11	10 of 10	0.2–7.5
9.0.7	10 of 10	1.1–27.7
8.0.18	0 of 10	–
11.0.0-preview.7, --no-churn	0 of 10	–

A stalled process keeps one core busy (the spinning thread) until it is
killed.

In a dump of a stalled process: the spinning thread is in cooperative mode
with m_hasPendingActivation == true; g_TrapReturningThreads == 1;
g_pSuspensionThread is in ThreadSuspend::SuspendAllThreads; a 2 s
sample shows no pthread_kill on the suspending thread and no signal
handler on the spinning thread. Without the child processes, or with .NET 8,
the same program runs without stalling. A variant without child processes
that another process sends SIGCONT every 200 µs stalled in 1 of 10 runs.

Regression?

Yes, from .NET 8 (see above).

Known Workarounds

Avoid busy-wait loops that never reach a safe point on their own; for
MSBuild's static graph, construct ProjectGraph with degreeOfParallelism: 1.

Configuration

.NET 11.0.0-rc.1.26425.128, 11.0.0-preview.7.26381.103, 10.0.11, 9.0.7 and 8.0.18; macOS 27.0 (26A428,
xnu-13432.1.9), Apple M2 Ultra, arm64. XNU source references are from
xnu-12377.1.9.

Other information

Possible directions, for whoever owns this code: on macOS, do not rely on
si_pid to recognise the runtime's own activations (for example, accept a
SIGUSR1 when the receiving thread has m_hasPendingActivation set; standard
signals coalesce, so a foreign SIGUSR1 merged with a pending activation is
delivered only once anyway); or have SuspendAllThreads resend an activation
that stayed pending for longer than a few hijack rounds.

Related

#130144 / #122768 fixed the Windows counterpart, where m_hasPendingActivation was never reset after QueueUserAPC2 failed. On macOS the flag stays set when the activation handler rejects a SIGUSR1 whose si_pid belongs to another process; this still reproduces on 11.0.0-rc.1.26425.128.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @VSadov
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.