Skip to content

Latest commit

 

History

History
25 lines (23 loc) · 8.21 KB

NOTES.md

File metadata and controls

25 lines (23 loc) · 8.21 KB

NOTES

Debugging log. One line per surprise (wrong sign, API mismatch, tensor layout, etc).

  • 2026-08-20: Verified typegpu latest published version is 0.12.0 on npm (registry.npmjs.org). Description: "A thin layer between JS and WebGPU/WGSL that improves development experience and allows for faster iteration."
  • 2026-08-20: Verified HF model onnx-community/depth-anything-v2-small exists, Apache-2.0. onnx/ subfolder has: model.onnx (fp32), model_fp16.onnx, model_int8.onnx, model_uint8.onnx, model_quantized.onnx, model_bnb4.onnx, model_q4.onnx, model_q4f16.onnx. Using dtype: 'fp16' with device: 'webgpu' per Transformers.js WebGPU guidance (fp32 is the WebGPU default, fp16 is the recommended smaller/faster option; q8/q4 are WASM-oriented quant paths, not guaranteed to run well on the WebGPU backend).
  • 2026-08-20: Docs site at docs.swmansion.com/TypeGPU is JS-rendered (Starlight/Astro) and several guessed sub-paths 404'd through WebFetch's raw-HTML fetch. Ground truth for the pipeline API is being taken from the installed package's own .d.ts files and README under node_modules/typegpu instead of scraped docs pages.
  • 2026-08-20: TypeGPU 0.12 API shape confirmed from installed .d.ts: tgpu.init() -> root; root.configureContext({canvas, format}); shell-less root.createRenderPipeline({vertex, fragment, targets, primitive}); root['~unstable'].createCommandEncoder() -> TgpuCommandEncoder; pipeline.with(bindGroup).with(encoder).withColorAttachment({view: context}).draw(3). pipeline.with(...) is overloaded for bind groups, passes, encoders and vertex layouts.
  • 2026-08-20: Gotcha — bindGroupLayout.$.<key> is a TGSL-only accessor. It throws "Direct access to texture views values is possible only as part of a compute dispatch or draw call" when evaluated eagerly, so it CANNOT be interpolated into a WGSL template string. Tagged-template interpolation of the underlying handles also fails: they get toString()ed into the source instead of resolving, and no @group/@binding declarations are emitted. What works for raw-WGSL shaders: pass a body-only WGSL string to the shell (the shell generates the @fragment fn name(...) -> ... header itself — supplying your own header duplicates it) and attach bindings with .$uses({ name: handle }), referencing them by that name. The handles live on the layout under Symbol(typegpu:0.12.0:$internal) as an array in key-declaration order; see bindingHandles() in src/renderer.ts.
  • 2026-08-20: TypeGPU rewrites identifiers inside body-only WGSL: in.vertexIndex becomes the flattened param name vertexIndex, and a var out: Out declaration resolves to the generated _Output struct name. Verified by dumping tgpu.resolve([vert, frag]) in Node before ever opening a browser — that offline dump is the fastest way to check shader codegen.
  • 2026-08-20: Observed tensor output on a live webcam: shape [448, 448], dtype float32, raw value range drifting roughly 0.0 … 13.3 across frames. Confirms disparity (not depth), positive, non-metric, and drifting — exactly as the plan predicted. Near objects = larger values = brighter in the debug view.
  • 2026-08-20: Gotcha — min/max normalization is dominated by outliers. With a desk edge close to the lens the raw max hit 13.25 and crushed the person-vs-wall separation into a few dark levels. Replaced with a 512-bin histogram percentile window (2%/98%) plus the EMA. Costs ~0.4ms extra on the CPU convert step.
  • 2026-08-20: Gotcha — the model runs on a centre-cropped SQUARE (cover-fit into 448x448) while the camera fills the whole canvas, so sampling the depth texture with raw screen UVs misregisters depth against the image whenever the canvas and video aspect ratios differ. Fixed with separate camScale/depthScale UV transforms in the uniform; see render() in src/renderer.ts.
  • 2026-08-20: Measured on the target machine (RTX 5080, Chrome): render 0.15-0.20ms, CPU tensor convert 0.7-1.1ms, depth upload 0.03ms, display 180-280fps. Inference is 35-41ms, NOT the sub-8ms the plan targets. Transformers.js/onnxruntime-web's WebGPU backend also logs "Some nodes were not assigned to the preferred execution providers" — part of the graph falls back to CPU. This gap is precisely the reason the original author hand-wrote the kernels (Phase 6); it is not something slider tuning fixes.
  • 2026-08-20: The ⚠️ [external-omitted] ... 'in.uv' warning at resolution time is benign — the resolver tries to treat in.uv as an external, fails, and correctly leaves the member access in the emitted WGSL.
  • 2026-08-20: User forwarded a LinkedIn post describing the original author's result: 448x448 depth model reimplemented as ~250 hand-written TypeGPU compute dispatches, ~8ms on an M4 Pro, with inference + lighting + draw all recorded into a single command encoder (no GPU->CPU->GPU round trip). This matches Phase 6 in the build plan exactly and confirms it's achievable, but it's still being treated as a separate, later effort — Phases 1-5 (Transformers.js-based inference + TypeGPU lighting pass) come first per the working agreement.
  • 2026-08-20: Feedback — the first lighting pass read as a flat sprite, not a light: it neither illuminated the scene nor made a dark room more visible. Root cause was the model itself, not tuning. 1 - smoothstep(0, radius, dist) is a hard-edged disc with zero contribution past radius, and compositing it as cam.rgb + light pastes a blob over the image rather than lighting the surface. Replaced with: pseudo-3D distance (screen offset + weighted depth offset), inverse-square attenuation 1/(1+k^2) which has an infinite tail, a facing term that dims surfaces nearer than the light, and MULTIPLICATIVE composition cam.rgb * (ambient + lightColor * irradiance) so the light reveals surface albedo. Added ambient so the scene can be darkened and the virtual light does the work.
  • 2026-08-20: Added sourceGain — the depth is also sampled at the light's own screen position, so covering the bulb with a hand dims the whole room (capped at 85% so it never fully blacks out).
  • 2026-08-20: Root cause of "I have to get too close to occlude it": with a keyboard/desk close to the lens the normalization window ran 0.33..10.68, so the seated subject normalized to roughly 0.02-0.10 — the entire useful range sat in the bottom 5% of the lightDepth slider, and a softness of 0.05 spanned all of it. Fixed by dropping HI_PCT from 0.98 to 0.90 (clip the near outliers) plus a depthGamma contrast curve (default 0.45) applied in the shader. Subject now lands nearer 0.3-0.5.
  • 2026-08-20: Gotcha (self-inflicted) — autoTuneDepth samples the raw r8unorm texel on the CPU while the shader compares against the gamma-curved disparity. The two live in different spaces, so the curve has to be applied in both places or auto-tune sets a value that means nothing to the shader.
  • 2026-08-20: Auto-tune biases the light +0.07 in front of the sampled surface. Parking it exactly on the surface leaves the subject sitting on the occlusion threshold and only half-lit.
  • 2026-08-20: Feedback with screenshots — partially covering the bulb killed the light entirely. Root cause: sourceBlocked sampled the depth at a SINGLE texel (the bulb centre), so coverage was effectively binary; occluding any part of the centre read as fully occluded. Fixed by measuring coverage across the bulb's whole disc: 37 area-weighted concentric-ring taps (r = sqrt(k/rings) so samples spread evenly instead of bunching at the centre), each run through the same smoothstep the shader uses, averaged into a 0..1 fraction. Verified with a horizontal light sweep: values now land at 15/20/21/28/70/97% instead of snapping between 0 and 100.
  • 2026-08-20: The coverage average is computed on the CPU from lastDepth, not in the shader. It is one number per frame, so doing 37 taps per pixel would waste ~2M x 37 samples on a value that is constant across the whole image. Passed in as the sourceCoverage uniform.
  • 2026-08-20: screenToDepthUv is affine, so the screen->depth scale for the tap offsets is recovered by differencing two mapped points rather than exposing the scale separately.
  • 2026-08-20: Coverage is EMA-smoothed (0.35). Depth only refreshes at the inference cadence (~27fps), so raw per-frame coverage steps visibly as a hand moves.