[Rust] Add Arrow mux API - #866
Draft
danilonajkov-db wants to merge 1 commit into
Draft
danilonajkov-db wants to merge 1 commit into
danilonajkov-db wants to merge 1 commit into
Conversation
This was referenced Sep 22, 2026
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-arrow-api
branch
from
September 23, 2026 11:08
fbcdcbe to
846642f
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-core
branch
2 times, most recently
from
September 23, 2026 12:52
cd29799 to
fe13be4
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-arrow-api
branch
2 times, most recently
from
September 23, 2026 14:54
fe46935 to
e208838
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-core
branch
from
September 23, 2026 14:54
fe13be4 to
9328853
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-arrow-api
branch
from
September 23, 2026 15:18
e208838 to
71e31cb
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-core
branch
from
September 23, 2026 15:18
9328853 to
fe13be4
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-arrow-api
branch
from
September 25, 2026 10:49
71e31cb to
5e2aa6d
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-core
branch
from
September 25, 2026 13:32
ebeb366 to
85a48bb
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-arrow-api
branch
from
September 25, 2026 13:32
5e2aa6d to
7876529
Compare
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-arrow-api
branch
from
September 25, 2026 15:58
7876529 to
28e40c3
Compare
danilonajkov-db
force-pushed
the
danilo-najkov_data/effort/mux-streams-core
branch
from
September 25, 2026 15:58
85a48bb to
f55b303
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Add Rust-only multiplexed Arrow Flight ingestion through
.arrow(schema).multiplexed(n).build_arrow(), returningMultiplexedArrowStream. Whole RecordBatches route round-robin,MessageIdidentifies the lane and local batch offset, and the mux-widemax_inflight_batchesbudget is divided evenly among lanes. No global ordering is promised.Private Arrow reservation and admission hooks connect to the shared core from #865, inheriting its poisoning and cancellation handling. Close preserves an already-recorded mux failure; otherwise it returns the first observed lane-close error and logs additional errors. Arrow retains its recovery and close supervisor. All lanes share the stats exporter; events retain lane-local offsets. Arrow acknowledgment callbacks remain unsupported.
For standalone Arrow streams,
ingest_batchcalls blocked onmax_inflight_batchesnow fail as soon asclose()starts instead of waiting for close to finish. Other closed-admission checks retain the prompt rejection behavior already present in 2.10. Includes focused tests for both cases, builder validation, routing/IPC/message-ID/telemetry tests, API docs, a runnable example, and changelog entries. No FFI or wrapper API changes.Stack base: #865. Follow-up: #867 adds broader lifecycle and failure tests.
How is this tested?
make testandmake lintmatrices passed on the combined implementation, plus the standalone Avro mux regression.