TCP registration repeatedly scans unexpired worker-removal records #16340

Problem

TcpStreamServer::associate_instance, associate_request_instance, and cancel_instance_streams call prune_tombstones under the shared registration lock. The helper calls HashMap::retain on each call.

Worker-removal records live for five seconds. During this interval, unrelated requests repeatedly scan the same unexpired records. This adds work to TCP response registration after worker churn. It also affects request-only callbacks used with QUIC responses.

Proposed change

Keep the earliest possible expiry in the existing registration state. Skip the scan before that time. When pruning is due, scan once and compute the next expiry. Refresh and removal must never delay any record past its existing five-second expiry.

The request path can then perform a constant-time deadline check while no record can expire. The due scan still depends on hash-table capacity. No timer, task, new lock, or wire change is needed.

Validation plan

Use paused-time native server tests for exact expiry, refresh, removal, and full endpoint identity. Compare the new state to the existing retain rule over mixed operations. Measure both the prune helper and the registration path, including empty and small-map controls. These are CPU component measurements, not full-model serving results.

Bounded open PR and issue searches found no matching TCP tombstone-pruning change. The inspected #15231 adds a separate Velo response transport. It does not optimize this TCP helper.

AI assistance was used for investigation.

Results

Native CPU association timing fell from 0.385 to 0.287 µs with 32 removal records (25.6% less time), and from 1.144 to 0.286 µs with 256 records (75.0% less time). These are component results after worker churn, not full serving throughput. Three separate processes per version used the real server state and a registered callback. The runtime crate used opt-level 3; dependencies used the test profile. The clock was paused and one registered callback key was reused. The zero-record difference was 2%; the one-record case was 0.5% slower.

Native network validation passed 301 tests, including 49 server tests. Three existing base tests remain separate limits: one cancellation assertion fails, and two older stream tests time out. An exact source-extracted state check matched the prior retain rule for 131,072 mixed operations per process across three processes.

#[tokio::test(start_paused = true)]
    #[ignore]
    async fn research_native_association_timing() {
        use std::hint::black_box;
        for count in [0, 1, 8, 32, 256, 1024] {
            let server = test_server().await;
            let stamp = Instant::now();
            for i in 0..count {
                server.cancel_instance_streams(&make_eid("ns", "comp", "generate", i)).await;
            }
            let live = make_eid("ns", "comp", "generate", 999999);
            let (subject, _provider, _recv, _recv_provider) = register_and_get_bidi_subjects(&server).await;
            let mut samples = Vec::new();
            for sample in 0..16 {
                let start = std::time::Instant::now();
                for _ in 0..20000 {
                    assert!(server.associate_request_instance(black_box(&subject), black_box(&live)).await);
                }
                if sample > 0 { samples.push(start.elapsed().as_nanos() as f64 / 20000.0); }
            }
            assert!(stamp.elapsed() < TOMBSTONE_TTL);
            samples.sort_by(f64::total_cmp);
            println!("NATIVE_ASSOC {{\"entries\":{count},\"median_ns\":{:.3}}}", samples[7]);
        }
    }

cargo --config 'profile.test.package.dynamo-runtime.opt-level=3' test --locked -p dynamo-runtime --lib research_native_association_timing -- --ignored --nocapture