Skip to content

Add out-of-core OpenStreetMap example (examples/osm)#216

Merged
adhami3310 merged 3 commits into
mainfrom
osm-example
Jul 22, 2026
Merged

Add out-of-core OpenStreetMap example (examples/osm)#216
adhami3310 merged 3 commits into
mainfrom
osm-example

Conversation

@adhami3310

Copy link
Copy Markdown
Member

Packages the planet-scale OSM demo as a proper example under examples/osm/, instead of loose files in scripts/ and a sibling ../osmium-rs checkout.

What's here

  • osmium-rs/ — the native Rust crate, moved in-tree: osm-nodes (PBF → f64 columns, ~250× pyosmium; multithreaded + zlib-ng) and osm-sort (builds the Tier-3 spatial index). Build artifacts (target/) gitignored.
  • ingest.py — parse a planet .pbf into f64 columns and render the out-of-core density scatter (with a memory report + zoom timings).
  • viewer.py — interactive browser viewer serving xy's real WebGL client over plain HTTP; pan/zoom re-aggregate every viewport from disk.
  • _pbf_split.py — PBF blob-framing helper.
  • README.md — the full parse → sort → view pipeline, prerequisites, and timings.

Scope notes

  • Zone-map disk cache (which the user also asked about) already shipped in Out-of-core canonical columns (disk-backed mmap store) #155 — this PR just documents and demonstrates it. Verified live: figure build over the 10.7B-node planet drops from ~51 s → 0.12 s on reopen, because the first figure() persists per-chunk statistics to a .xyzones sidecar that later runs validate (size/mtime) and reuse.
  • Adjusts paths for the new location (repo root two levels up; native binaries under osmium-rs/target/release), removes hardcoded personal paths, and errors clearly if the render client bundle (built, not committed — Build the JS render client at packaging time instead of committing it #214) is missing.
  • No engine/core code changes; example-only. Renders 10.7B nodes with 0 bytes RAM-resident canonical data, 172 GB on disk.

Verified

  • Crate builds in the new location; osm-nodes/osm-sort binaries present.
  • Ruff check/format clean repo-wide; codespell clean on new files.
  • End-to-end figure build against the real 172 GB columns: 10,742,674,832 nodes, canonical_bytes = 0, 0.12 s via the zone-map cache.

Package the planet-scale OSM demo as a proper example instead of loose
scripts. Renders all ~10.7B OpenStreetMap nodes as an out-of-core density
scatter: 172 GB of f64 columns on disk, screen-bounded resident memory.

- examples/osm/osmium-rs/: native Rust crate moved in-tree — osm-nodes
  (PBF -> f64 columns, ~250x pyosmium) and osm-sort (Tier-3 spatial index).
- examples/osm/ingest.py: parse a planet .pbf and render the scatter.
- examples/osm/viewer.py: interactive browser viewer over xy's WebGL client.
- examples/osm/_pbf_split.py: PBF blob-framing helper.
- README documents the parse -> sort -> view pipeline, timings, and how the
  .xyzones zone-map cache turns the 51s figure-build domain scan into a
  one-time cost (verified: 0.12s on reopen).

Moved from scripts/ (osm_viewer.py, osm_planet_bench.py, _pbf_split.py) and
from the sibling ../osmium-rs; build artifacts and local data gitignored.
@codspeed-hq

codspeed-hq Bot commented Jul 22, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 102 untouched benchmarks
⏩ 1 skipped benchmark1


Comparing osm-example (fc29d4c) with main (68617fe)

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports.

@greptile-apps

greptile-apps Bot commented Jul 22, 2026

Copy link
Copy Markdown

Greptile Summary

This PR adds an out-of-core OpenStreetMap example under examples/osm/. The main changes are:

  • An in-tree Rust PBF decoder and spatial sorter.
  • Python ingestion and interactive viewer scripts.
  • Documentation for the parse, sort, and view pipeline.
  • Validation for truncated PBF files and invalid sorter arguments.
  • A request-body size limit for the viewer.

Confidence Score: 5/5

This looks safe to merge.

The parser fixes cover sparse nodes and truncated input. The sorter fixes reject the reported invalid argument cases.

No additional blocking issue qualifies.

T-Rex T-Rex Logs

What T-Rex did

  • Observed that the pre-feature revision had no examples/osm/viewer.py and the browser request returned HTTP 404.
  • Validated that after the feature, the real page renders a density chart from the generated fixture, starting with an unzoomed view.
  • Demonstrated user interaction by zooming to 265% with the wheel and dragging, causing the viewport to update and the page to respond with HTTP 200.
  • Confirmed browser validation finished with exit code 0 and Chromium emitted only non-fatal WebGL performance warnings.
  • Collected and organized artifacts, including before/after videos and images, plus logs and a Python harness, to support verification.

View all artifacts

T-Rex Ran code and verified through T-Rex

Important Files Changed

Filename Overview
examples/osm/osmium-rs/src/lib.rs Decodes dense and sparse OSM nodes and reports incomplete PBF blobs as errors.
examples/osm/osmium-rs/src/sort.rs Builds the spatial index and validates grid and partition settings before processing.
examples/osm/osmium-rs/src/sort_main.rs Parses sorter options and rejects incomplete extent arguments through the normal error path.
examples/osm/viewer.py Serves the interactive viewer and limits declared request-body sizes.
examples/osm/ingest.py Runs native PBF ingestion and demonstrates out-of-core rendering and memory reporting.

Reviews (2): Last reviewed commit: "osm-sort: parse CLI options without pani..." | Re-trigger Greptile

Comment thread examples/osm/osmium-rs/src/lib.rs Outdated
Comment thread examples/osm/osmium-rs/src/lib.rs Outdated
Comment thread examples/osm/osmium-rs/src/sort.rs
Comment thread examples/osm/osmium-rs/src/sort.rs
Comment thread examples/osm/osmium-rs/src/sort_main.rs Outdated
Comment thread examples/osm/viewer.py
Khaleel Al-Adhami added 2 commits July 22, 2026 15:37
…viewer)

- osm-nodes: a truncated PBF (partial download) was silently accepted as a
  complete decode; now EOF is clean only at a blob boundary and any mid-blob
  EOF is a hard error (never publish incomplete data as complete, §28).
- osm-nodes: sparse (non-dense) Node messages were counted and skipped, so
  'every node' was false for extracts using them; now decoded (lat=8/lon=9
  zigzag sint64, same scaling as DenseNodes). Planet files are unaffected.
- osm-sort: --grid 0 (empty-histogram index) and --partitions 0 (divide by
  zero) panicked; now validated up front and returned as InvalidInput through
  the CLI's normal error path, plus a g*g overflow / u32-fit guard.
- viewer: cap the /msg request body (413 over 1 MiB) so the demo can't be made
  to allocate unboundedly.
- Rust unit tests cover the truncation primitive, sparse-node decode, and the
  sort validation.
A flag at the end of argv (`--grid` with no value) or an incomplete
`--extent` indexed args out of bounds, and non-numeric values hit
`.unwrap()` — both panicked. Every option now reads through
`args.get(..).parse()` and reports a clean usage error instead.
@adhami3310
adhami3310 merged commit ad29127 into main Jul 22, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant