Skip to content

CLAUDE.md

Latest commit

 

History

History
98 lines (72 loc) · 4.97 KB

CLAUDE.md

File metadata and controls

98 lines (72 loc) · 4.97 KB
Raw

Python SDK

Python wrapper around the Rust core via PyO3 and maturin.

Client code patterns

When writing or reviewing client code that uses this SDK, follow the cross-SDK performance flow from the root CLAUDE.md:

  • Idiomatic flow: ingest in a loop with ingest_record_offset(), or prefer ingest_records_offset() for bulk data, then call flush() once to confirm durability. ingest_record_offset() returns as soon as the record is queued; the SDK sends it and tracks its acknowledgment in the background. Do not feature ingest_record_nowait() in examples: it spawns a detached task and is not safely synchronized with flush().
  • In async code, an AckCallback is a good way to track durability without blocking.
  • wait_for_offset() blocks until a specific offset is acknowledged — use it to confirm a specific record before continuing. Acks are ordered, so waiting on the LAST offset confirms all prior records. Avoid calling it after every record in a tight loop, since that limits throughput to one record per round-trip.
  • ingest_record() is deprecated — prefer the offset-based APIs.

Structure

python/
├── rust/src/lib.rs   # PyO3 bindings (the Rust ↔ Python bridge)
├── zerobus/          # Python package (pure-Python layer over _zerobus_core)
├── tests/            # pytest test suite
└── pyproject.toml    # Build config (maturin)

The native extension module is _zerobus_core, exposed as three submodules:

  • zerobus._zerobus_core.sync — Synchronous API
  • zerobus._zerobus_core.aio — Async API (Python async/await)
  • zerobus._zerobus_core.arrow — Arrow Flight support

Build commands

Run from python/:

  • make dev — Create venv, install deps, build Rust extension
  • make build-rust — Release build of the Rust extension
  • make develop-rust — Debug build (faster iteration)
  • make test — pytest with coverage
  • make lint — pycodestyle + autoflake
  • make fmt — black + autoflake + isort

Features

Avro record format support is always compiled in — the avro feature is enabled on the databricks-zerobus-ingest-sdk dependency in python/rust/Cargo.toml. The avro pip extra installs fastavro, which is required only for encoding dict records; pre-encoded Avro bytes need no extra.

FFI boundary: PyO3

PyO3 handles memory management automatically via Python reference counting. Key considerations:

  • GIL release: Async operations release the GIL, allowing concurrent Python threads. The tokio runtime is initialized at module import via pyo3_async_runtimes::tokio::init().
  • PyO3 API: The bindings use PyO3 0.29 and the Bound<'py, T> API. Use Python::attach (not with_gil), Python::detach (not allow_threads), and cast (not downcast). Py<T> is not unconditionally Clone — use clone_ref(py) under an attached interpreter.
  • Object lifetime: PyO3 classes are ref-counted by Python's GC. No explicit free needed, but close() should still be called to flush pending records before GC cleanup.
  • Error mapping: Rust ZerobusError maps to ZerobusException / NonRetriableException Python exception classes.
  • Serialization: Proto descriptors and record payloads cross as Python bytes. JSON records cross as Python str. No extra serialization layer — PyO3 handles the conversion.

Breaking change rules

The public Python API is defined by the classes and functions in zerobus/. Changes here follow semver:

  • Removing a class, method, or parameter is breaking.
  • Deprecate with warnings.warn(..., DeprecationWarning) first.
  • The PyO3 binding layer (python/rust/src/lib.rs) is internal; refactoring it is safe as long as the Python-facing API doesn't change.

Performance notes

  • PyO3 has minimal overhead for bytes/str payloads (no copy for immutable buffers).
  • Async API avoids GIL contention — prefer it for high-throughput ingestion.
  • Batch ingestion (ingest_records) amortizes the Python→Rust call overhead.

Changelog and documentation

  • Every PR must update python/NEXT_CHANGELOG.md under the appropriate section if it changes user-facing behavior.
  • Update python/README.md if the change affects usage, setup, or API surface.
  • Add or update examples in python/examples/ for new or modified APIs.
  • Add docstrings for all new public classes/methods. Update type stubs (_zerobus_core.pyi) if the native module interface changes.

Release

  • Version is derived from python/rust/Cargo.toml via maturin (no separate Python version file).
  • Tag: python/v<semver> → triggers release-python.yml → builds wheels for all platforms/Python versions → publishes to PyPI.
  • On version bump PR: update the version in python/rust/Cargo.toml, move NEXT_CHANGELOG.md contents to CHANGELOG.md, reset NEXT_CHANGELOG.md.

Config

  • Python >= 3.9 required; tested through 3.14 (free-threaded builds are not supported)
  • Rust >= 1.88 required for the wrapper crate (Tonic 0.14.6 MSRV)
  • Line length: 120 (black)
  • Formatter: black, isort