test(e2e): verify replica scale-down against matching upstream runtime images

[Verolop]

Background

The current drain tests mock UpdateConsensusRule. A permissive mock can accept an incomplete request, which allowed the missing expected_outgoing_rule problem tracked in #576 to go undetected.

The nightly compatibility suite exercises newer upstream runtime images, but it does not currently exercise data-plane replica scale-down. The deletion test removes an entire cluster, while the scaling test covers stateless components.

We need a test that talks to a real multipooler and proves that the operator follows the current consensus-rule RPC contract.

Required changes

• Create an end-to-end scenario with enough poolers to remove one replica while retaining a valid cohort.
• Wait until the shard has a primary and a decided consensus rule.
• Scale the pool down by one replica.
• Verify that the operator:
  • reads the primary's current rule;
  • submits a CAS-fenced cohort-removal request;
  • removes the intended pooler from the cohort;
  • completes deletion without waiting for the five-minute emergency timeout.
• Confirm that the remaining shard is healthy and can still serve traffic.
• Add a stale-rule scenario if it can be made deterministic: advance the rule between status discovery and update, then verify that the operator retries safely.
• Run this scenario in the nightly upstream compatibility workflow.
• Run the targeted scenario on automated proto-sync PRs using the exact runtime images selected by the PR.

Acceptance criteria

• The test fails if ExpectedOutgoingRule is omitted.
• The test fails if an incompatible protobuf operation is used.
• Successful scale-down completes before the force-unregister timeout.
• The removed pooler is absent from the committed cohort.
• The remaining cluster is healthy and queryable.
• A transient stale-rule failure does not incorrectly advance the drain state.
• The test runs against immutable, matching Multigres and pgctld image revisions.

References

• Drain compatibility fix: fix(drain): use CAS-fenced RuleOperation requests when removing replicas #576
• Proto sync PR: chore(deps): sync multigres proto to 3533683 #575