Skip to content

feat(spanner): add dynamic channel pool support#14604

Open
rahul2393 wants to merge 6 commits into
mainfrom
support-dynamic-channel-pooling
Open

feat(spanner): add dynamic channel pool support#14604
rahul2393 wants to merge 6 commits into
mainfrom
support-dynamic-channel-pooling

Conversation

@rahul2393

@rahul2393 rahul2393 commented May 16, 2026

Copy link
Copy Markdown
Contributor

Add Go Spanner Dynamic Channel Pooling (DCP) with scale-up/down, drain safety, DirectPath fallback support,
observability.

  • Adds opt-in DynamicChannelPoolConfig for Go Spanner.
  • Implements DCP picker, scale-up, scale-down, draining, priming, and error penalty.
  • Preserves transaction/session stickiness via operationRefs.
  • Supports DirectPath fallback with paired wrapper slots.
  • Adds OTel-only DCP metrics for debugging.

Internal reference: go/go-dcp-design

@rahul2393 rahul2393 requested review from a team as code owners May 16, 2026 20:22
@product-auto-label product-auto-label Bot added the api: spanner Issues related to the Spanner API. label May 16, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a Dynamic Channel Pool (DCP) for the Spanner client, which automatically scales the number of gRPC channels based on RPC load. The implementation includes features like Power-of-Two-Least-Busy selection, channel priming, graceful draining, and OpenTelemetry metrics. Review feedback highlights the need to improve configuration defaulting for initial channels and to propagate request contexts through the session management layer to ensure consistent observability and trace propagation.

Comment on lines +103 to +105
DCPMaxRemoveChannels: 2,
DCPDrainIdleGrace: 5 * time.Minute,
DCPMaxDrainTimeout: 30 * time.Minute,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current defaulting logic for DCPInitialChannels can lead to validation failures if a user provides a DCPMinChannels value greater than the default initial channels (4). Consider defaulting DCPInitialChannels to at least DCPMinChannels to ensure consistency, while ensuring that this change does not break backward compatibility for existing stable code paths.

Suggested change
DCPMaxRemoveChannels: 2,
DCPDrainIdleGrace: 5 * time.Minute,
DCPMaxDrainTimeout: 30 * time.Minute,
if cfg.DCPInitialChannels == 0 {
cfg.DCPInitialChannels = def.DCPInitialChannels
if cfg.DCPInitialChannels < cfg.DCPMinChannels {
cfg.DCPInitialChannels = cfg.DCPMinChannels
}
}
References
  1. When introducing new components with different default behaviors, avoid altering the defaults of existing, stable code paths to prevent breaking changes for backward compatibility.

Comment thread spanner/session.go Outdated
Comment thread spanner/session.go Outdated
Comment thread spanner/session.go Outdated
rahul2393 added a commit that referenced this pull request May 26, 2026
Split of #14604

Internal reference: go/go-dcp-design
rahul2393 added a commit that referenced this pull request Jun 5, 2026
…4613)

Split of #14604

Internal reference: go/go-dcp-design

 ## Add OpenTelemetry metrics for the dynamic channel pool

  ### What
Adds OpenTelemetry metric support for the dynamic channel pool (DCP),
giving visibility into pool size, scaling activity, and RPC load.

  ### Metrics (prefix `spanner/dynamic_channel_pool/`)
  | Metric | Type | Description |
  | --- | --- | --- |
  | `num_channels` | gauge | Active channels in the pool |
  | `draining_channel_count` | gauge | Channels currently draining |
  | `max_allowed_channels` | gauge | Max channels allowed |
  | `active_rpc_count` | gauge | Active RPCs on the pool |
| `max_active_rpc_per_channel ` | gauge | Maximum number of RPCs
currently active on any channel in the dynamic channel pool.|
| `channel_pool_scaling` | counter | Channels added/removed, tagged
`direction=up\|down` |

All metrics carry the common attributes `client_id`, `database`,
`instance_id`, `library_version`.

  ### Enablement
Consistent with existing client metrics: emitted only when
`EnableOpenTelemetryMetrics()` is set. Uses
`ClientConfig.OpenTelemetryMeterProvider`, falling back to the global
OpenTelemetry MeterProvider when unset. The observable callback is
unregistered on `dynamicChannelPool.Close()`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: spanner Issues related to the Spanner API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant