A high-performance, multimodal-native engine for AI workloads
Vane unifies multimodal data, intelligence, and continuous learning with Python and SQL interfaces, seamlessly scaling from local environments to Ray clusters.
Note
Project status
- Vane Data — Supports most of the capabilities described below and is under active development. Its interfaces and internals may continue to evolve as the codebase is reviewed and hardened.
- Vane RL and Vane Agent — In the early stages of design and implementation. Their source code will be released in future updates.
- Vibe Coding and Agentic Engineering — Some parts of our system were initially built through Vibe Coding. We are now continuously analyzing, understanding, and improving the codebase, applying an Agentic Engineering approach to drive iterative optimization and enhance the quality, maintainability, and efficiency of the system.
Vane Data is a high-performance, multimodal-native data engine for AI workloads. Built on a fork of DuckDB, it extends the core execution engine with native multimodal processing and a unified framework for local and distributed execution.
- Multimodal-native processing — Process images, video, audio, text, documents, events, sensor data, and tables through a unified type system. Dynamic batching and backpressure control handle variations in data size and computational cost.
- Python and SQL interfaces — Build data and AI pipelines with DuckDB SQL or the Python Relation API.
- Built-in AI operations — Invoke LLMs, generate embeddings, and run batch inference through OpenAI and Anthropic APIs or native vLLM integration. Prefix-aware bucketing improves vLLM prefix-cache hit rates and inference throughput.
- Heterogeneous execution — Overlap CPU, GPU, I/O, and model inference workloads through asynchronous scheduling.
- Local-to-cloud execution — Run the same pipeline locally or across distributed Ray clusters, with a foundation for future edge-cloud coordination.
- Designed for production AI workloads — Build multimodal training-data preprocessing pipelines and enterprise-scale batch inference workflows.
Vane supports Python 3.10 through 3.14. Python 3.12 is recommended and is the primary development version.
Install the vane-ai package from PyPI:
pip install vane-aiVane owns only the vane Python namespace. It does not install duckdb,
_duckdb, or adbc_driver_duckdb, so the official duckdb distribution can
be installed in the same environment and both engines can be imported in the
same process. Vane code must use import vane; import duckdb always refers
to the separately installed official package. Vane does not provide a legacy
duckdb alias or fall back to an official DuckDB native module.
import duckdb
import vane
assert vane.connect().execute("SELECT 42").fetchone() == (42,)
assert duckdb.connect().execute("SELECT 43").fetchone() == (43,)Vane's ADBC driver is exposed as vane.adbc; the official driver's
adbc_driver_duckdb namespace remains owned by the official distribution.
Install adbc-driver-manager (also included by vane-ai[all]) to use either
ADBC facade.
Optional features are provided as extras:
pip install 'vane-ai[openai]' # OpenAI provider (anthropic / google / transformers likewise)
pip install 'vane-ai[vllm]' # Native vLLM inference on Linux x86-64
pip install 'vane-ai[sglang]' # Native SGLang 0.5.17 inference on Linux x86-64
pip install 'vane-ai[image]' # ndarray image inputs for AI providers (Pillow)
pip install 'vane-ai[video]' # video data source (Pillow, psutil, decord)
pip install 'vane-ai[milvus]' # distributed full-row Milvus upserts
pip install 'vane-ai[qdrant]' # distributed full-point Qdrant upsertsThe SGLang extra follows SGLang 0.5.17's default CUDA 13 dependency set. Python package metadata cannot select a CUDA-specific wheel index for the host. For a CUDA 12.9 environment, install the extra and then apply SGLang's cu129 wheel overrides before running Vane:
uv pip install 'vane-ai[sglang]'
uv pip install --force-reinstall torch==2.11.0 torchaudio==2.11.0 torchvision \
--index-url https://download.pytorch.org/whl/cu129
uv pip install --force-reinstall sglang-kernel==0.4.5 \
--index-url https://docs.sglang.ai/whl/cu129/
uv pip install --force-reinstall sgl-deep-gemm==0.1.5.post1 \
--index-url https://docs.sglang.ai/whl/cu129/ --no-depsThis CUDA 12.9 installation path was smoke-tested on NVIDIA Ada (compute capability 8.9) with driver 570.207,
PyTorch 2.11.0+cu129, sglang-kernel 0.4.5+cu129, and Ray 2.58.0. This is a validated reference configuration, not an
exhaustive hardware compatibility list. Do not run a later unconstrained dependency sync after the overrides, because
it can replace the cu129 packages with SGLang's default CUDA 13 variants.
The video extra installs decord on Linux x86-64, Vane's currently supported native platform. decord itself publishes no wheels for modern Python on macOS or for any ARM platform; if Vane adds Windows support later, decord's existing win_amd64 wheel can be enabled explicitly.
For more details, see the Installation Guide.
Follow the Quickstart guide to build and run your first Vane pipeline.
MilvusSink writes distributed relation batches with ordinary full-row
override upserts. The collection must disable AutoID and dynamic fields, must
not define collection functions, and must have one caller-assigned INT64 or
VARCHAR primary key. Every required collection field must be present; use
field_mapping when relation and collection names differ. Partial updates,
merge modes, and array append/remove operations are not exposed.
Supported relation fields are Arrow booleans, signed 8/16/32/64-bit integers,
32/64-bit floats, strings, and lists of 32-bit floats for FLOAT_VECTOR
fields. max_batch_rows limits rows per worker call, while max_batch_bytes
limits the Arrow buffer size before conversion to Milvus records.
uri accepts a base HTTP(S) endpoint without embedded credentials or a
database path; select a non-default database with database=.... Credential
values must come from an EnvironmentSecret resolved on each worker.
from vane import EnvironmentSecret, MilvusSink
sink = MilvusSink(
"documents",
uri="https://milvus.example:19530",
primary_key="id",
token=EnvironmentSecret("MILVUS_TOKEN"),
)
summary = relation.write_datasink(sink)The default max_retries=0 disables Vane's full-operation replay after an
unknown outcome; PyMilvus can still recover transport failures within an SDK
call's configured timeout. If Vane retries are enabled, it replays the complete
input with the same operation ID; the adapter replaces the same explicit
primary keys, but concurrent external writers can still race. Successful
batches are acknowledged and applied independently, so a failed operation can
be partially applied. Vane does not provide an atomic transaction, rollback,
or exactly-once delivery, and read visibility follows the consistency level
used by Milvus readers.
QdrantSink writes distributed relation batches as full-point upserts. Each
row must provide an explicit Arrow uint64 or UUID-string point ID. Use a
single source column for a collection with one unnamed dense vector, or map
source columns to every named dense vector in the collection. Payload fields
are also explicitly mapped; project away any relation columns that should not
be written. Dense vector columns must be Arrow lists of float32, and their
dimensions must match the collection. Payload values may be nulls, booleans,
integers whose values fit Qdrant's signed 64-bit payload range, finite
float32/float64 values, strings, or nested Arrow lists and structs composed
from those types. Convert binary, decimal, temporal, and other values to an
explicit supported representation before writing.
UUID text is normalized before Vane's global key check, so invalid, null, and
semantically duplicate point IDs are rejected before workers open.
max_batch_rows limits rows per worker call, while max_batch_bytes limits
the Arrow buffer size before point conversion.
from vane import EnvironmentSecret, QdrantSink
sink = QdrantSink(
"documents",
url="https://qdrant.example:6333",
point_id="id",
vector_mapping="embedding",
payload_mapping={"title": "title", "source": "source"},
api_key=EnvironmentSecret("QDRANT_API_KEY"),
)
summary = relation.write_datasink(sink)The endpoint itself may instead be supplied as an EnvironmentSecret when it
must not be serialized with the plan. API-key values are always resolved from
the worker environment. Qdrant receives wait=True, and Vane reports a batch
as applied only when Qdrant returns completed. Each upsert replaces all
vectors and payload for its point ID; partial updates are not exposed.
The default max_retries=0 disables Vane's full-operation replay after an
unknown outcome. Enabling retries replays the complete input with the same
normalized point IDs. Successful worker batches remain independently visible
if another batch or the overall job later fails. Vane does not provide an
atomic transaction, rollback, deletion, or exactly-once delivery.
Vane uses the Ray runner by default. If no runner is configured, executing a lazy relation through consumers such as display, result fetching, or file writes selects Ray and may lazily initialize it. An experimental local runner can be selected explicitly before creating connections:
import vane
vane.configure(runner="local")Vane follows Ray's trusted-cluster model: the driver, workers, submitted code, and east-west network belong to one trusted computing boundary. Same-process local-disk shuffle reads directly from the process-local registry, and object-storage shuffle reads committed manifests. Only cross-worker local-disk shuffle uses Arrow Flight.
A worker lazily starts one process-owned plaintext grpc:// Flight service when a local-disk exchange sink first needs it. The service provides no TLS, client authentication, query-level authorization, or tenant isolation. Keep its port reachable only inside the controlled Ray cluster network; workloads that do not trust one another require separate isolated Ray clusters.
Workers advertise their Ray private address by default. VANE_FLIGHT_BIND_HOST may select a different local bind address, including 0.0.0.0 in a container with appropriate network policy, while VANE_FLIGHT_ADVERTISE_HOST must always be a routable non-wildcard address. The advertised-host override is worker-local: set it in each worker node's environment rather than on the driver or in a Ray Job/actor runtime environment. DUCKDB_FLIGHT_PORT selects a fixed worker-local port; the default 0 lets the operating system allocate one. See SECURITY.md for the complete trust boundary.
Each cross-worker partition read has a five-minute deadline covering its complete Flight DoGet stream. Override it with VANE_FLIGHT_CALL_TIMEOUT_S, or set it to 0 to disable the deadline. The deadline is not reset when a schema or record batch arrives. Query interruption independently cancels an in-flight Flight call, so interrupted consumers release the producer-side stream and its shuffle-file read lease.
Relations can create catalog-backed tables with storage-format-neutral table properties and partition expressions:
import vane
relation = vane.sql("SELECT id, event_date FROM source_table")
relation.create(
"catalog.schema.table",
properties={
"format-version": "2",
"write.data.path": "s3://bucket/table/data",
},
partition_by=["bucket(16, id)", vane.ColumnExpression("event_date")],
)The target catalog defines which properties and partition expressions it supports. With the Ray runner, CTAS is
submitted as a distributed write and any planning or execution error is returned directly; it is never retried in
local DuckDB. Set VANE_RUNNER=local-fast to explicitly select the native DuckDB backend. An unset or empty
VANE_RUNNER selects Ray.
Hardware configuration: 1 node, 36 CPU cores, 64 GB memory, and 1× NVIDIA GeForce RTX 2080 Ti (22 GB VRAM).
We use the Ray Data benchmark suite to compare Vane with Ray Data and Daft. The benchmark source code is included in this repository.
The Ray runner targets distributed workloads. The current results are single-node only; validation on the multi-node environments used in the Ray Data benchmarks is still pending.
See the benchmarking page for detailed results.
Contributions and collaborations are welcome. Contribution guidelines and community channels will be published as the project opens further.
Vane is distributed under the Apache License 2.0. See LICENSE and NOTICE for details and third-party attributions.
Vane Data is built on top of DuckDB and inspired by infrastructure systems such as Ray Data, Daft, and Trino.
- DuckDB: The core modular architecture and inspiration. A high-performance analytical database system. It is designed to be fast, reliable, portable, and easy to use.
- DuckDB-Python: The core modular architecture and inspiration. The DuckDB Python package.
- Ray Data: A scalable data processing library for AI workloads built on Ray
- Daft: High-Performance Data Engine for AI and Multimodal Workloads
- Trino: A fast distributed SQL query engine for big data analytics.
Special thanks to these projects.
Give Vane a ⭐️ if it helps you!