Skip to content

[Rust] Add Arrow mux API - #866

Draft
danilonajkov-db wants to merge 1 commit into
danilo-najkov_data/effort/mux-streams-corefrom
danilo-najkov_data/effort/mux-streams-arrow-api
Draft

danilonajkov-db wants to merge 1 commit into
danilo-najkov_data/effort/mux-streams-corefrom
danilo-najkov_data/effort/mux-streams-arrow-api

Conversation

@danilonajkov-db

@danilonajkov-db danilonajkov-db commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

What changes are proposed in this pull request?

Add Rust-only multiplexed Arrow Flight ingestion through .arrow(schema).multiplexed(n).build_arrow(), returning MultiplexedArrowStream. Whole RecordBatches route round-robin, MessageId identifies the lane and local batch offset, and the mux-wide max_inflight_batches budget is divided evenly among lanes. No global ordering is promised.

Private Arrow reservation and admission hooks connect to the shared core from #865, inheriting its poisoning and cancellation handling. Close preserves an already-recorded mux failure; otherwise it returns the first observed lane-close error and logs additional errors. Arrow retains its recovery and close supervisor. All lanes share the stats exporter; events retain lane-local offsets. Arrow acknowledgment callbacks remain unsupported.

For standalone Arrow streams, ingest_batch calls blocked on max_inflight_batches now fail as soon as close() starts instead of waiting for close to finish. Other closed-admission checks retain the prompt rejection behavior already present in 2.10. Includes focused tests for both cases, builder validation, routing/IPC/message-ID/telemetry tests, API docs, a runnable example, and changelog entries. No FFI or wrapper API changes.

Stack base: #865. Follow-up: #867 adds broader lifecycle and failure tests.

How is this tested?

  • Arrow integration suite at this layer: 104 passed.
  • SDK unit tests with all features: 388 passed, 1 ignored.
  • Full Rust make test and make lint matrices passed on the combined implementation, plus the standalone Avro mux regression.
  • Rust formatting and diff whitespace checks passed.

@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from fbcdcbe to 846642f Compare September 23, 2026 11:08
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch 2 times, most recently from cd29799 to fe13be4 Compare September 23, 2026 12:52
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch 2 times, most recently from fe46935 to e208838 Compare September 23, 2026 14:54
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch from fe13be4 to 9328853 Compare September 23, 2026 14:54
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from e208838 to 71e31cb Compare September 23, 2026 15:18
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch from 9328853 to fe13be4 Compare September 23, 2026 15:18
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from 71e31cb to 5e2aa6d Compare September 25, 2026 10:49
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch from ebeb366 to 85a48bb Compare September 25, 2026 13:32
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from 5e2aa6d to 7876529 Compare September 25, 2026 13:32
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from 7876529 to 28e40c3 Compare September 25, 2026 15:58
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch from 85a48bb to f55b303 Compare September 25, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant