What happened?
When reading from an unbounded Spanner Change Stream (or a Mutable Change Stream with a capped query end timestamp), QueryChangeStreamAction issues change stream queries with an artificial changeStreamQueryEndTimestamp (now + 2 minutes).
Previously, when the query completed and needed to resume (!stopAfterQuerySucceeds), QueryChangeStreamAction called tracker.tryClaim(changeStreamQueryEndTimestamp) before returning ProcessContinuation.resume():
https://github.com/apache/beam/blob/ac4acbb6282/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/QueryChangeStreamAction.java#L360-L365
In a very rare timing corner case where a change stream query finishes without returning the ChildPartitionsRecord at the end of a partition's range, claiming changeStreamQueryEndTimestamp can advance the restriction tracker past the partition's actual end timestamp. On the next continuation, the query resumes from changeStreamQueryEndTimestamp + 1ns, which falls outside the partition's valid timestamp range and returns an out-of-range start_timestamp error, causing QueryChangeStreamAction to mark the partition FINISHED before scheduling its child partitions.
Workaround & Scope
This is a client-side workaround in Apache Beam for this rare edge case while a fix is being implemented on the Spanner server side:
- Unbounded queries: When
!stopAfterQuerySucceeds, leaving the restriction tracker at the last claimed position (from the last processed data or heartbeat record) instead of advancing to changeStreamQueryEndTimestamp ensures the subsequent query resumes from lastClaimedTimestamp + 1ns and reads any remaining records (including ChildPartitionsRecord) before the partition ends. This workaround addresses the issue for unbounded queries.
- Bounded queries: For bounded queries where
changeStreamQueryEndTimestamp reaches endTimestamp (stopAfterQuerySucceeds == true), this client-side workaround does not apply and will be addressed by the Spanner server-side fix.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
When reading from an unbounded Spanner Change Stream (or a Mutable Change Stream with a capped query end timestamp),
QueryChangeStreamActionissues change stream queries with an artificialchangeStreamQueryEndTimestamp(now + 2 minutes).Previously, when the query completed and needed to resume (
!stopAfterQuerySucceeds),QueryChangeStreamActioncalledtracker.tryClaim(changeStreamQueryEndTimestamp)before returningProcessContinuation.resume():https://github.com/apache/beam/blob/ac4acbb6282/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/QueryChangeStreamAction.java#L360-L365
In a very rare timing corner case where a change stream query finishes without returning the
ChildPartitionsRecordat the end of a partition's range, claimingchangeStreamQueryEndTimestampcan advance the restriction tracker past the partition's actual end timestamp. On the next continuation, the query resumes fromchangeStreamQueryEndTimestamp + 1ns, which falls outside the partition's valid timestamp range and returns an out-of-rangestart_timestamperror, causingQueryChangeStreamActionto mark the partitionFINISHEDbefore scheduling its child partitions.Workaround & Scope
This is a client-side workaround in Apache Beam for this rare edge case while a fix is being implemented on the Spanner server side:
!stopAfterQuerySucceeds, leaving the restriction tracker at the last claimed position (from the last processed data or heartbeat record) instead of advancing tochangeStreamQueryEndTimestampensures the subsequent query resumes fromlastClaimedTimestamp + 1nsand reads any remaining records (includingChildPartitionsRecord) before the partition ends. This workaround addresses the issue for unbounded queries.changeStreamQueryEndTimestampreachesendTimestamp(stopAfterQuerySucceeds == true), this client-side workaround does not apply and will be addressed by the Spanner server-side fix.Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components