Point-window cache (T13) + resolvability gate (#225)#227
Merged
Conversation
The Tier-2 density map colormapped the binned COUNT, ignoring the points' own colors — a colored scatter's aggregate view matched neither its marks nor its legend, and every density<->points zoom transition visibly recolored the chart. The surface now wears the data's colors (LOD doc §2): each cell carries the alpha-weighted mean of its binned points' resolved colors — one law for continuous, categorical, and direct-RGBA channels — averaged in linear light through an integer-only pipeline (checked-in sRGB<->linear u16 tables, u64 sums; bitwise deterministic across thread counts and platforms). The log-tone-mapped count drives only the alpha channel: more points, deeper color; fewer, lighter. Constant-color traces keep the compact count-only wire and a client tint (the mean of a constant is the constant). - Rust core: bin_2d_mean_color kernel, with a fused cell accumulator that fans out under a points-per-cell gate (<=4 workers, exact integer merge — output identical to the serial scan); pyramid mean-color planes (build_color/compose_color; count grids bit-identical to the count-only compose; colored pyramids refuse appends and are invalidated + rebuilt lazily). C ABI v40. - Python: channels.resolve_bin_colors maps every channel mode onto the kernel color source; initial emit + density_view (exact and pyramid paths) ship the RGBA plane as density.rgba with color_agg: "mean"; color is no longer listed in dropped_channels; the windowed-exact spatial tier is gated to colorless traces (its index is position-only, §27 — the upsampled colored-pyramid grid stays instead); SVG/PNG exporters follow the same color law; continuous-channel density scatters render their colorbar again. - Client: DENSITY_FS mean-color branch over a premultiplied RGBA8 texture (count tone curve baked at upload so bilinear filtering weights color by coverage); exposure easing re-encodes color grids; the standalone to_html re-bin worker aggregates the sample's colors under the same law; the density-tier gradient legend swatch is removed (count is alpha, so a gradient would claim color == density; a named density trace falls through to the plain marker swatch, matching the static exporters). - Handoff (§5): intensity-only. Hue is continuous by construction, so drilled points arrive in native colors, enter at their cell's count-alpha, and ease to native opacity; density_colormap left the points wire, and density_val (still u8 on the live wire, #221) now weights alpha instead of indexing a colormap LUT. The aggregate backdrop retires once a drill settles inside its window (T10 amended: a mean-color wash under exact marks reads as data) and eases back fast when the view leaves the window, a refinement goes pending, or the drill dies — zoom-outs never blank and interleaved replies never flash. - Spec: LOD doc §2 rules rewritten around the shipped mean-color law, pyramid §4 color planes and refused appends, T3/T10 amended, Phase 1/3 progress; dossier §5 Tier-2, F5 verdict, and constants table; wire protocol records density.rgba/color_agg, the u8 unit-scalar encodings, and the slimmer points message; chart-kind contract covers the aggregate kernel pairing and the legend rule; rust-engine inventory row. - Tests: kernel-vs-NumPy oracle suite (tests/test_density_mean_color, including a serial parallel-merge oracle and exporter pixel checks), wire and colorbar assertions, render-smoke probes for the mean-color surface (meancolor) and backdrop retirement (dretire). The pyramid color test asserts the direction the #153 area-weighted compose keeps invariant (nonzero count implies lit, unlit implies zero count) rather than exact mask equality, which boundary-bin count slivers can legitimately break.
The demo predated the kernel pyramid and carried two extra density implementations to stay interactive at 100M points: a 6144^2 integral-image overview server-side and ~350 lines of page JS (browser local re-bins, request parking/single-flight/timeout-retry, per-client staleness maps, pending-status heuristics). Both were count-only, so once density surfaces started wearing the mean point color, mid-zoom windows flipped back to a count-colormapped green wash, flashing old<->new across zoom levels. The engine has since absorbed every job they existed to do: Figure.density_view serves wide windows from the mean-color pyramid in O(visible cells) with drill bookkeeping and recorded binning, and the render client debounces requests, seq-drops stale replies, and keeps the best cached texture drawn while a reply is in flight. The module is now the pattern it demonstrates: build one figure (pyramid warmed at startup), serve payload + bundled client as HTML, and wire ChartView to a comm object that POSTs messages to one endpoint dispatching density_view/pick under a lock. Page JS drops from ~400 lines to ~60 (fetch + status badge), and the HTML route reuses the shared figure instead of regenerating 100M rows per GET. Round-trips ship as XYBF binary frames (wire-protocol §7): the reply message is the frame's compact JSON metadata and each numeric buffer rides raw and 8-byte aligned, so the browser decodes one xy.decodeFrame and hands the kernel zero-copy views with no base64 step on either side (~33% smaller replies on the grids and point buffers that dominate a drill). First paint still embeds its blob as base64 — inline HTML has no binary channel (§6) — and malformed requests still return a JSON error the client rejects on res.ok before decoding. tests/test_drilldown_pan_alignment.py is removed with its subject: it guarded the browser-local re-bin's window-clamp bug, and engine-served grids always report exactly the window they binned, so that failure mode is unreachable. The example test now decodes the XYBF frame and asserts on its message and raw buffers.
Serve the fastapi example's live drilldown scatter — same seed-11 chunked columns, same mark config, same XY_LIVE_POINTS override — from the reflex-xy adapter as one inline() token with zero transport code. The fastapi app wires this chart through a custom HTTP transport (a Starlette endpoint plus a comm bridge); the adapter's websocket namespace and the kernel's density tiers replace all of it, so behavior differences between the two hosts isolate what that custom code adds versus what is fundamental to the engine (density tiers, pyramid binning, drill-in). - examples/reflex: §6 section + drilldown_chart()/drilldown_view(), README run notes (import-time build, memory, XY_LIVE_POINTS override) - tests/test_example_apps.py: source markers for §6; the import test pins XY_LIVE_POINTS=50000 (same override the fastapi test uses) and asserts the inline token mints - scripts/reflex_ws_smoke.py: count the seventh live source, wait for the drilldown payload, and pixel-check its density surface - spec: reflex-integration §7 inventory + production-readiness example focus bullet name the deliberate cross-host overlap
Once a points reply has shipped its window exactly (reduction "none" — the subset IS every point in the window), any view contained in that window is already answered by the marks on the GPU: the smaller window's points are a subset of the shipped ones. Drilling deeper therefore no longer round-trips the kernel — _scheduleViewRequest skips the trace's pending markers and its density_view message (lodDrillServesView, re-checked at debounced send time), while still bumping seq so an in-flight reply for an older, wider view dies stale instead of yanking exact marks out from under a view it cannot improve. Three conditions re-arm the request: leaving the window, a drill that is dying or not exact (only reduction "none" arms the elision, recorded as drill.exact), and depth — the shipped geometry is f32, offset-encoded on the window midpoint (dossier §16), so once the view span drops below 1/256 of the window span on either axis one request goes out purely to re-center the encoding. Data changes cannot serve stale marks through the elision: streaming append and full payload updates rebuild the GPU trace, which drops the drill. Recorded as invariant T12 in spec/design/lod-architecture.md §5, in the wire protocol's density_view entry, and in dossier §16/§28; covered by a headless-Chromium probe (tests/test_drill_zoomin_elides_request.py) asserting the elision and each re-arm condition.
…dows Fixes #225: zooming out from a drilled window brought fixed-rate "sampled points" back over the density surface — individual marks at a zoom where real points are sub-pixel, misrepresenting the dataset. Now: - Interactive density_view replies ship no point sample at all. The mean-color density surface (LOD doc §2) stands alone above the drill budget; real points still arrive the moment a window fits it. The retained first-payload sample (the standalone re-bin worker's CPU source) draws only below a resolvable-count gate: estimated in-view count <= LOD_DIRECT_POINT_BUDGET (lodOverlayResolvable, T9). Datasets under the budget keep the hybrid look everywhere. Point caching (LOD doc T13) so pan/zoom stops re-shipping the same points: - Kernel: a points-tier reply ships the largest ALIGNED window around the view whose exact count fits the budget (lod.aligned_window: bounds snap outward to a power-of-two grid over the trace's extent, per dimension; DRILL_PAD_TARGETS ladder, pyramid-count-gated, exact-verified, DRILL_PAD_SPAN_CAP guards the §16 re-encode ladder). Consecutive pans resolve to the SAME window; lod_blend stays keyed on the view's own count. Nonlinear axes keep the exact view window. - Client: a reply for a new window retires the previous exact drill into a bounded per-trace cache (lodRetireDrill/LOD_POINT_CACHE_WINDOWS) instead of overwriting its buffers — so does a drill dying outside its window — and any later view covered by a cached window promotes it back alpha-continuously with zero round-trips (lodPromoteCachedDrill). Outgrown windows free geometry-only (T11 rule, §27). - Kernel keeps a bounded drill-subset history (Trace.drill_history, DRILL_HISTORY_KEEP) so picks against a promoted retired window still translate exactly; expired seqs drop the pick, data changes clear it. - Client suppresses identical density_view requests (same window, screen, unchanged data) and accepts the suppressed duplicate's in-flight reply per-trace instead of killing it with the bumped seq — ending the re-request loop at gesture end. Spec: lod-architecture T9 rewritten + new T13, wire-protocol density_view/points/pick contracts, dossier constants table, CHANGELOG, README representation labels, docs ladder table. Tests: new test_drill_window_padding (alignment, budget, span cap, nonlinear skip, history clear), test_density_sample_resolvability_gate and test_drill_point_window_cache (headless-Chromium client probes), elision test extended for duplicate suppression; drill tests updated to the padded-window contract.
masenf
marked this pull request as draft
July 23, 2026 12:50
Greptile SummaryThis PR updates the LOD path for density and drill-down rendering. The main changes are:
Confidence Score: 5/5This PR appears safe to merge. The highest-risk paths are covered by targeted browser probes and server-side tests for cache promotion, request economy, padding, and overlay gating. No new verified blocking issues were found in this pass. No files require special attention.
 T-Rex LogsWhat T-Rex did
Important Files Changed
Reviews (8): Last reviewed commit: "Merge origin/main into claude/point-cach..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…yered draw Field HAR from the 100M live drilldown (PR #227 review): every pan/zoom step at 200-450% zoom re-requested a ~2.7 MB full-screen density grid — including back-to-back windows differing by sub-pixel amounts — and the frame dropped to the blurriest cached texture whenever a pan left the freshest window. Three fixes, all recorded under LOD doc T13: - Kernel: pyramid-served grids are clamped to the source cells the finest level resolves under the window (interaction._pyramid_source_shape, ceil(base*frac)+1 per axis) — composing to full screen resolution just upsamples blocky cells the client's texture filtering reproduces from the source-resolution grid at a fraction of the bytes. The attainable per-axis cell size rides the reply as density.min_cell; exact and spatial grids (true full-detail bins) omit it. The HAR's 2.7 MB replies drop to ~0.5-1.4 MB in its zoom range and ~200 KB deeper. - Client request elision (lodDensityCacheServes): a cached texture that contains the view and is as detailed as anything the kernel could return — one texel per screen pixel, or already at min_cell (zooming into a source-limited texture cannot sharpen it) — answers with no round-trip. Guarded so the kernel stays in charge where it can do better: estimated in-view count must exceed budget x LOD_DENSITY_ELIDE_EST_FACTOR, and the cached window may be at most LOD_DENSITY_ELIDE_MAX_AREA_RATIO x the view area (uniform-density estimates overshoot in sparse corners). Entries without recorded counts never elide; lodRememberDensity now supersedes an unpinned same-window twin so elision always reads current facts. - Sub-texel request dedup: the identical-request memo compares windows numerically within half an output texel per edge (gesture-end and settle emit sub-pixel twins; a grid shifted below half a texel is the same picture) instead of exact string equality. - Finer-detail layering (lodDensityDetailForView): when no fine cached window contains the view, the smallest cached texture overlapping >= 5% of it draws on top of the broad backdrop after every crossfade layer, so the already-fetched region stays sharp while the request for the rest is in flight — a resolution seam beats uniform blur. Spec: lod-architecture T13 extended, wire-protocol density_view/density contracts (min_cell, clamped grids, elision), dossier constants table, CHANGELOG. Tests: test_density_wire_economy (source-shape math, min_cell presence/ absence, 20M-point clamp integration), test_density_request_economy (headless-Chromium probe: source-resolution elision + guards, sub-texel dedup, fine-over-broad layering).
Field feedback on PR #227 (100M live drilldown), superseding the previous commit's adequacy-elision approach with a simpler rule: the density texture already covering the view STANDS — however blurry — until the view could plausibly resolve into real points. Intermediate-zoom blur is an accepted, recorded cost; the kernel's sharper answers (exact re-bins, drill-in) stay reachable because the points band still requests. - Client gate (lodAggregateStands): a density_view goes out only when the estimated in-view count sits within LOD_POINTS_REQUEST_BAND (4x) of the direct budget. Two estimators, LOWER wins so an over-estimate can never strand a view in blur: the smallest cached density window containing the view (recorded count, area-scaled; seeded by the home texture's first-payload count) and the retained deterministic sample counted in-view and re-weighted (lodSampleViewCount) — the sample follows the data's actual distribution, so sparse tails reach their points where uniform-density area-scaling over-estimates by orders of magnitude. Traces with no recorded counts always request. - Kernel-attached clients never draw the retained sample at ANY zoom (field capture: 8k arbitrary sample rows at full alpha over a view the kernel serves real points for): resolvable views get exact points, and the sample's kernel-mode job is now purely CPU-side estimation. The standalone client keeps drawing it below the resolvability gate — its only point representation. - Reverted the fine-over-broad detail layer from the previous commit (field capture: a stale darker rectangle) — density textures alpha-composite, so the overlap double-counts opacity, and per-window normalization makes the seam a brightness step. Recorded in T13; the tier draws one texture per frame. Dropped the density.min_cell reply field with it (no reader remains); the source-resolution grid clamp and the sub-texel request dedup stay. Spec: lod-architecture T9/T13 revised (never-refines rule, sample-based estimator, recorded reversal), wire-protocol, dossier constants (LOD_POINTS_REQUEST_BAND replaces the elision guards), config comment, CHANGELOG, docs ladder table. Tests: request-economy probe rewritten to the stands-rule (band requests, sub-texel dedup, single-texture frames), resolvability-gate probe gains kernel-mode assertions, wire-economy tests drop min_cell.
Field feedback on PR #227: the count->alpha tone curve made the surface's lightness swing against the very points it aggregates — visibly at the texture<->points boundary, and between windows (each reply normalized against its own max). The §2 philosophy already defines a cell as the physically downsampled image of its points; this applies that to ALPHA: - A channel-bearing cell's displayed alpha is now the physical compositing of its own points, 1 - (1 - a_pt)^count, with a_pt = channel alpha x trace style opacity folded INSIDE the exponent — a one-point cell reads exactly like one point, dense cells saturate past the style opacity exactly like overplotted marks. No window normalization touches these textures (the upload is max-independent), so exposure easing (T4) no longer applies to them and the draw path passes only transition fades in u_opacity (count-only grids keep the log ramp + style opacity uniform — count is their only structure). - Mean-color drills swap at native opacity with no intensity handoff: the texture composites like the marks replacing it, so density_val / lod_blend are ignored on such traces (still shipped; count-only drills keep the count-alpha ramp). The recorded cost: per-cell density beyond a few points no longer reads in the surface — exactly as saturated as the overplotted truth it downsamples; drill-in is where per-point structure returns. Same law in the client texture upload (lodWriteGridTexture), the SVG and native-raster exporters (shared _svg._physical_density_alpha), and the standalone re-bin worker (shared upload path). Spec: LOD doc §2 rules 1/2/5, T3/T4 scoping, tier table, Phase-1 notes, dossier renderer notes, config/marks warning text, CHANGELOG. Tests: new test_density_physical_alpha (headless-Chromium texture-byte probe: law values, no norm anim, no handoff) and an exporter-law unit test in test_density_mean_color; full suite green.
Field capture on PR #227 (screen recording): zooming around the points band read as "jumping between zoom levels of the density map". Frame analysis showed the drawn aggregate flipping between two texture characters — the smooth standing surface (coarse, linear-upscaled) and fresh transition-band exact grids (~16 points/cell), whose one-point cells now draw at the points' own alpha (§2 physical compositing) and so read as hard speckle. Every band probe repainted smooth -> speckle -> smooth, one jump per crossing. Completing the stands-rule (T13) on the display side: with a kernel attached, an aggregate reply never REPLACES a texture that already covers the view. The standing surface holds until real POINTS land; the mid-band reply contributes a facts-only cache entry (window + exact count, lodRememberDensityFacts) that recalibrates the points-band gate, and the drill lifecycle proceeds as before (dying marks fade into the standing surface). The picture changes only aggregate->points and back. A reply for an UNCOVERED view (pan past every cached window) still applies — silence never blanks a frame (T1) — and standalone clients keep applying everything (their re-binned grids are their only refinement). Facts entries are tex-less cache citizens: the draw path skips them, the gate reads them, eviction guards their missing texture. Spec: T13 display-side paragraph; CHANGELOG. Tests: the request-economy probe now asserts the standing texture is byte-identical after a covered mid-band reply, that its facts landed, and that an uncovered reply still applies.
Field feedback on PR #227: the aggregate-to-points boundary now feels smooth, but the standing texture reads soft at times — the home texture can stretch ~10x by the time points land. Requested: one or two more density texture steps before real points, stepwise, NOT per-window (per-view textures were the recorded zoom/pan-jumping failure mode). - While the aggregate stands (outside the points band), the one density request allowed is the next LADDER STEP window (lodAggregateStepWindow): the view snapped outward to a power-of-LOD_AGG_STEP_FACTOR (4) block grid over the data extent, per axis, at most LOD_AGG_STEP_MAX (2) steps below home, and only when every covering texture is coarser than the step by more than LOD_AGG_STEP_SLACK. Quantized windows are pan-stable and dedupable: every view in a region resolves to the SAME window, pans inside a step repaint nothing, revisits hit the cache, and a zoom sees at most two smooth-to-smooth swaps before points — worst-case softness bounded at ~4x stretch per axis instead of unbounded home-stretch. - A step reply (matched against the marked request window, g._stepReqWin) is the ONE density reply that may repaint a covered view. Points-band probes keep requesting the RAW view — the kernel decides the tier with real counts — and their density replies stay facts-only, so the band's speckled exact grids still never repaint the smooth surface. Step windows are always >= the view, so their counts sit above the band and the kernel serves them from the pyramid: smooth by construction. A step window sparse enough to fit the budget comes back as points — applied as a (padded) drill, points early. - Request routing unified in _densityRequestPlan: drill/point-cache service first, then band probe (raw view) or ladder step; the sub-texel dedup memo now keys on whichever window is actually sent. Spec: T13 display-side rewritten around the ladder (+ recorded trade-off: in-band the probe owns the single request slot), dossier constants, CHANGELOG. Tests: the request-economy probe asserts the step request is the aligned block window (never the raw view), that its reply repaints while probe replies stay facts-only, that a covered step elides further requests, and that band probes request raw views.
test_adaptive_drilldown_cycle pinned the drill reply's visible count to the raw view window's population — a pre-T13 contract. Drills now ship the widest ALIGNED window around the view that still fits the budget (LOD doc T13), so the fixture derives the oracle from the warm-up reply's shipped window instead: alignment is a pure function of the extent and the view's span bucket, so every benchmark iteration resolves to the same window and the same exact count. The cycle now also asserts that window's determinism per iteration.
Merging this PR will degrade performance by 18.81%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
masenf
marked this pull request as ready for review
July 24, 2026 01:58
Most conflicts were squash artifacts: main's #226 is the same mean-color density work this branch already carries as commits (main's conflicted spec/client/kernel/test files are byte-identical to this branch's pre-work state), so those resolve to this branch's extended versions. Real incoming changes kept from main: - src/tiles.rs: separable area-weighted compose (#228) — taken wholesale (this branch made no tiles.rs changes beyond the shared #226 content); kernels.rs/lib.rs wasm panic-abort hardening (#200) auto-merged. - README.md: the #230 adoption refresh replaced the inline benchmark table (this branch's only README edit) with links to the launch report, so main's version stands; the 0.1.0 launch-baseline report keeps its historical "density + sample" labels — it records what that release did. Verified on the merged tree: cargo tests, ABI smoke, render smoke, and the full pytest+benchmarks suite — the only failures are 21 doc-contract tests (test_docs_examples/test_verify_local/pyplot perf guardrail) that fail identically on pristine origin/main in this environment: the #230 README rewrite removed maintainer-shortcut text its guardrail tests still assert. Pre-existing on main, not introduced here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements two interconnected LOD improvements: a per-trace LRU cache of retired exact point windows (T13) that eliminates re-requests on pan/zoom sequences, and a resolvability gate (#225) that prevents sample overlays from drawing above the direct point budget where they misrepresent the dataset.
Key Changes
Point-Window Cache (T13)
LOD_POINT_CACHE_WINDOWS = 3constant to bound retired window memorylodRetireDrill(),lodPromoteCachedDrill(), and supporting functions to manage the per-trace LRU cache of exact point windowslodWindowServesView()as a shared containment check for both live drill and cached windowsResolvability Gate (#225)
lodOverlayResolvable()function that gates sample overlay drawing on estimated in-view countoverlay.sample.visible * (viewArea / windowArea) <= LOD_DIRECT_POINT_BUDGETdensity_viewreplies no longer ship point samples at all (only the first-payload sample is retained for the standalone re-bin worker)lodSampleForView()to apply the gate before considering overlay candidatesPadded Aligned Windows (T13 support)
aligned_window()and_padded_drill_window()to compute the widest aligned window that fits the budgetDRILL_PAD_SPAN_CAP) to keep deep zooms inside the §16 re-encode ladderDrill Subset History
drill_historydict toTraceto remember recent shipped subsets (bounded byDRILL_HISTORY_KEEP)drill_history()returns the shipped subset for a givendrill_seq, or None when expired (drops the pick)clear_drill_history()forgets subsets after data changesIdentical-Request Suppression (T13)
_densityRequestDup()to detect when the same window at the same screen size is already in flight or answeredClient-Side Changes
lodClearDrillState()consolidates drill lifecycle reset (used by drop, retire, promote)lodFreeDrillBuffers()frees all GPU buffers for a drill objectlodSameWindow()compares windows with floating-point toleranceNotable Implementation Details
(span * 1e-9 + 1e-300)to handle floating-point slopvisiblecount on the sample metadata; overlays without recorded counts (hand-built/legacy specs) keep drawinghttps://claude.ai/code/session_01XQT3v7NdL4dmC6S3g8HjJt