Add out-of-core OpenStreetMap example (examples/osm)#216
Merged
Conversation
Package the planet-scale OSM demo as a proper example instead of loose scripts. Renders all ~10.7B OpenStreetMap nodes as an out-of-core density scatter: 172 GB of f64 columns on disk, screen-bounded resident memory. - examples/osm/osmium-rs/: native Rust crate moved in-tree — osm-nodes (PBF -> f64 columns, ~250x pyosmium) and osm-sort (Tier-3 spatial index). - examples/osm/ingest.py: parse a planet .pbf and render the scatter. - examples/osm/viewer.py: interactive browser viewer over xy's WebGL client. - examples/osm/_pbf_split.py: PBF blob-framing helper. - README documents the parse -> sort -> view pipeline, timings, and how the .xyzones zone-map cache turns the 51s figure-build domain scan into a one-time cost (verified: 0.12s on reopen). Moved from scripts/ (osm_viewer.py, osm_planet_bench.py, _pbf_split.py) and from the sibling ../osmium-rs; build artifacts and local data gitignored.
Merging this PR will not alter performance
Comparing Footnotes
|
Greptile SummaryThis PR adds an out-of-core OpenStreetMap example under
Confidence Score: 5/5This looks safe to merge. The parser fixes cover sparse nodes and truncated input. The sorter fixes reject the reported invalid argument cases. No additional blocking issue qualifies.
What T-Rex did
Important Files Changed
Reviews (2): Last reviewed commit: "osm-sort: parse CLI options without pani..." | Re-trigger Greptile |
added 2 commits
July 22, 2026 15:37
…viewer) - osm-nodes: a truncated PBF (partial download) was silently accepted as a complete decode; now EOF is clean only at a blob boundary and any mid-blob EOF is a hard error (never publish incomplete data as complete, §28). - osm-nodes: sparse (non-dense) Node messages were counted and skipped, so 'every node' was false for extracts using them; now decoded (lat=8/lon=9 zigzag sint64, same scaling as DenseNodes). Planet files are unaffected. - osm-sort: --grid 0 (empty-histogram index) and --partitions 0 (divide by zero) panicked; now validated up front and returned as InvalidInput through the CLI's normal error path, plus a g*g overflow / u32-fit guard. - viewer: cap the /msg request body (413 over 1 MiB) so the demo can't be made to allocate unboundedly. - Rust unit tests cover the truncation primitive, sparse-node decode, and the sort validation.
A flag at the end of argv (`--grid` with no value) or an incomplete `--extent` indexed args out of bounds, and non-numeric values hit `.unwrap()` — both panicked. Every option now reads through `args.get(..).parse()` and reports a clean usage error instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Packages the planet-scale OSM demo as a proper example under
examples/osm/, instead of loose files inscripts/and a sibling../osmium-rscheckout.What's here
osmium-rs/— the native Rust crate, moved in-tree:osm-nodes(PBF → f64 columns, ~250× pyosmium; multithreaded + zlib-ng) andosm-sort(builds the Tier-3 spatial index). Build artifacts (target/) gitignored.ingest.py— parse a planet.pbfinto f64 columns and render the out-of-core density scatter (with a memory report + zoom timings).viewer.py— interactive browser viewer serving xy's real WebGL client over plain HTTP; pan/zoom re-aggregate every viewport from disk._pbf_split.py— PBF blob-framing helper.README.md— the full parse → sort → view pipeline, prerequisites, and timings.Scope notes
figure()persists per-chunk statistics to a.xyzonessidecar that later runs validate (size/mtime) and reuse.osmium-rs/target/release), removes hardcoded personal paths, and errors clearly if the render client bundle (built, not committed — Build the JS render client at packaging time instead of committing it #214) is missing.Verified
osm-nodes/osm-sortbinaries present.canonical_bytes = 0, 0.12 s via the zone-map cache.