SDP META Release v0.1.0#328
Open
ravi-databricks wants to merge 110 commits into
Open
Conversation
Merging Main branch release docs changes
- Added `bronze_cluster_by_auto` and `silver_cluster_by_auto` parameters to support automatic clustering in onboarding templates. - Updated relevant documentation to include descriptions for new parameters. - Enhanced dataflow specifications to handle `clusterByAuto` property for both bronze and silver layers. - Modified integration tests to validate the new clustering functionality. - Adjusted existing tests to ensure compatibility with the new parameters and their expected behavior.
Implement automatic liquid clustering for bronze and silver tables
Update dataflow_pipeline.py to use quarantine_table instead of quaran…
…ython module name and databricks-labs-sdp-meta as the PyPI package name for consistency with the sdp-meta CLI name. Updates all source, tests, docs, demos, compat layer, and integration tests.
Renaming dlt-meta to sdp-meta
…data generation - Multi-section YAML support for enhanced dlt-meta functionality - Synthetic data generation using dbldatagen with proper API - Lakeflow Connect integration for database ingestion - Complete examples with variables, transformations, and dataflows - Enhanced CLI commands for single-file configuration Co-authored-by: Cursor <cursoragent@cursor.com>
Brings issue_266's Lakeflow Connect demo, SQL Server demo, and TechSummit demo updates onto the v0.0.10 → v0.0.11 backward-compatibility tip (PR #289). Single conflict in run_integration_tests.py reconciled additively: PR #304's bronze_silver combined-layer wiring is preserved and an optional extra_config parameter (used by the LFC demo to pass dtix_snapshot_method) is merged after the per-layer keys so callers can override defaults.
- Replace Hugo (config.toml, content/, layouts/, theme submodules)
with a Docusaurus 3.8.1 site under docs/ — 34 pages across 6
sections (Getting Started, Concepts, Reference, Guides, Operations,
Contributing) plus a landing page and Databricks-branded CSS.
- Pin webpack@5.97.1 via package.json `overrides` to work around the
webpackbar 6.0.1 / ProgressPlugin schema incompatibility on
webpack 5.95+.
- Update .github/workflows/gh_pages.yml to build with `npm ci` +
`npm run build`; drop the Hugo install step and submodule fetch.
- Rename docs/static/images/ → docs/static/img/ (Docusaurus
convention); update every README image reference accordingly.
- Refresh dlt-meta → sdp-meta repo URLs across root, compat/, demo/,
integration_tests/, and lakehouse_app/ READMEs.
- Replace dead `docs-new/*` ignore rules with `docs/{build,.docusaurus,
node_modules}` and drop the obsolete Hugo lock/public entries.
Changed release version from v0.0.11 to v0.1.0
Migrate docs site from Hugo to Docusaurus
Switch gh_pages workflow from branch-based publishing to GitHub Actions Pages artifact/deploy flow for issue #320, with scoped permissions and docs-only triggers.
Update gh_pages to use GitHub Actions Pages environment
Fixed readme with logo Fixed all versions
PyPI-ready sdp-meta, release workflow hardening
- Fix all broken README badges (workflow→actions, pypi pkg name, coverage repo)
- Replace examples/ links with correct demo/conf/ paths throughout docs
- Replace landing page feature cards with inline SVG architecture diagram
- Remove "Built with Docusaurus" from footer copyright
- Fix overwrite casing ("true"→"True") in silver-fanout guide
- Fix silver_transformation_json missing _prod suffix across guides
- Add 16 new FAQ entries: migration CLI rename, DAB deploy workflow, App usage
- Remove duplicate YAML question from FAQ
Updated feature table Updated docs
Fixed main readme with links and descriptions
Added new sdp-meta cli commands gif into readme and docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR delivers the complete v0.1.0 release of SDP-META (formerly DLT-META), merging
feature/sdp-metaintov0.1.0ahead of the repo rename todatabrickslabs/sdp-meta.Breaking Changes
databricks labs install sdp-metaimport dlt→from pyspark import pipelines as dp— native Lakeflow SDP APIsrc.*→databricks.labs.sdp_meta.*New Features
bundle-init,bundle-add-flow,bundle-validate,bundle-prepare-wheel; git-tracked, CI/CD-ready pipelines withdev/prodtargetsbronze_cdc_apply_changes_flows/silver_cdc_apply_changes_flowsto fan N CDC sources into one target tablebronze_row_filter/silver_row_filterand quarantine variantscluster_by_autooncreate_streaming_tabledatabricks_app/) replacinglakehouse_app/Bug Fixes
dataflow_pipeline.pyruntime bugs (Lakeflow Connect)Backward Compatibility
compat/shipsdlt-meta==0.1.0— an empty wrapper that installsdatabricks-labs-sdp-metaas a dependency. Existing users onpip install dlt-metaget v0.1.0 with zero notebook changes. Thesrc.*import shim fires aDeprecationWarningand will be removed in v0.2.0.A two-phase backward compat test (
integration_tests/run_backward_compat_tests.py) validates that v0.0.x pipelines, checkpoints, and DataflowSpec tables all work unchanged after upgrading the wheel.Files Changed (summary)
src/databricks/labs/sdp_meta/compat/dlt-metacompatibility wrapper packagedatabricks_app/lakehouse_app/)docs/demo/integration_tests/README.md