Debugging log. One line per surprise (wrong sign, API mismatch, tensor layout, etc).
- 2026-08-20: Verified
typegpulatest published version is0.12.0on npm (registry.npmjs.org). Description: "A thin layer between JS and WebGPU/WGSL that improves development experience and allows for faster iteration." - 2026-08-20: Verified HF model
onnx-community/depth-anything-v2-smallexists, Apache-2.0.onnx/subfolder has: model.onnx (fp32), model_fp16.onnx, model_int8.onnx, model_uint8.onnx, model_quantized.onnx, model_bnb4.onnx, model_q4.onnx, model_q4f16.onnx. Usingdtype: 'fp16'withdevice: 'webgpu'per Transformers.js WebGPU guidance (fp32 is the WebGPU default, fp16 is the recommended smaller/faster option; q8/q4 are WASM-oriented quant paths, not guaranteed to run well on the WebGPU backend). - 2026-08-20: Docs site at docs.swmansion.com/TypeGPU is JS-rendered (Starlight/Astro) and several guessed sub-paths 404'd through WebFetch's raw-HTML fetch. Ground truth for the pipeline API is being taken from the installed package's own
.d.tsfiles and README undernode_modules/typegpuinstead of scraped docs pages. - 2026-08-20: TypeGPU 0.12 API shape confirmed from installed
.d.ts:tgpu.init()-> root;root.configureContext({canvas, format}); shell-lessroot.createRenderPipeline({vertex, fragment, targets, primitive});root['~unstable'].createCommandEncoder()->TgpuCommandEncoder;pipeline.with(bindGroup).with(encoder).withColorAttachment({view: context}).draw(3).pipeline.with(...)is overloaded for bind groups, passes, encoders and vertex layouts. - 2026-08-20: Gotcha —
bindGroupLayout.$.<key>is a TGSL-only accessor. It throws "Direct access to texture views values is possible only as part of a compute dispatch or draw call" when evaluated eagerly, so it CANNOT be interpolated into a WGSL template string. Tagged-template interpolation of the underlying handles also fails: they gettoString()ed into the source instead of resolving, and no@group/@bindingdeclarations are emitted. What works for raw-WGSL shaders: pass a body-only WGSL string to the shell (the shell generates the@fragment fn name(...) -> ...header itself — supplying your own header duplicates it) and attach bindings with.$uses({ name: handle }), referencing them by that name. The handles live on the layout underSymbol(typegpu:0.12.0:$internal)as an array in key-declaration order; seebindingHandles()insrc/renderer.ts. - 2026-08-20: TypeGPU rewrites identifiers inside body-only WGSL:
in.vertexIndexbecomes the flattened param namevertexIndex, and avar out: Outdeclaration resolves to the generated_Outputstruct name. Verified by dumpingtgpu.resolve([vert, frag])in Node before ever opening a browser — that offline dump is the fastest way to check shader codegen. - 2026-08-20: Observed tensor output on a live webcam: shape
[448, 448], dtypefloat32, raw value range drifting roughly0.0 … 13.3across frames. Confirms disparity (not depth), positive, non-metric, and drifting — exactly as the plan predicted. Near objects = larger values = brighter in the debug view. - 2026-08-20: Gotcha — min/max normalization is dominated by outliers. With a desk edge close to the lens the raw max hit 13.25 and crushed the person-vs-wall separation into a few dark levels. Replaced with a 512-bin histogram percentile window (2%/98%) plus the EMA. Costs ~0.4ms extra on the CPU convert step.
- 2026-08-20: Gotcha — the model runs on a centre-cropped SQUARE (cover-fit into 448x448) while the camera fills the whole canvas, so sampling the depth texture with raw screen UVs misregisters depth against the image whenever the canvas and video aspect ratios differ. Fixed with separate
camScale/depthScaleUV transforms in the uniform; seerender()insrc/renderer.ts. - 2026-08-20: Measured on the target machine (RTX 5080, Chrome): render 0.15-0.20ms, CPU tensor convert 0.7-1.1ms, depth upload 0.03ms, display 180-280fps. Inference is 35-41ms, NOT the sub-8ms the plan targets. Transformers.js/onnxruntime-web's WebGPU backend also logs "Some nodes were not assigned to the preferred execution providers" — part of the graph falls back to CPU. This gap is precisely the reason the original author hand-wrote the kernels (Phase 6); it is not something slider tuning fixes.
- 2026-08-20: The
⚠️ [external-omitted] ... 'in.uv'warning at resolution time is benign — the resolver tries to treatin.uvas an external, fails, and correctly leaves the member access in the emitted WGSL. - 2026-08-20: User forwarded a LinkedIn post describing the original author's result: 448x448 depth model reimplemented as ~250 hand-written TypeGPU compute dispatches, ~8ms on an M4 Pro, with inference + lighting + draw all recorded into a single command encoder (no GPU->CPU->GPU round trip). This matches Phase 6 in the build plan exactly and confirms it's achievable, but it's still being treated as a separate, later effort — Phases 1-5 (Transformers.js-based inference + TypeGPU lighting pass) come first per the working agreement.
- 2026-08-20: Feedback — the first lighting pass read as a flat sprite, not a light: it neither illuminated the scene nor made a dark room more visible. Root cause was the model itself, not tuning.
1 - smoothstep(0, radius, dist)is a hard-edged disc with zero contribution pastradius, and compositing it ascam.rgb + lightpastes a blob over the image rather than lighting the surface. Replaced with: pseudo-3D distance (screen offset + weighted depth offset), inverse-square attenuation1/(1+k^2)which has an infinite tail, afacingterm that dims surfaces nearer than the light, and MULTIPLICATIVE compositioncam.rgb * (ambient + lightColor * irradiance)so the light reveals surface albedo. Addedambientso the scene can be darkened and the virtual light does the work. - 2026-08-20: Added
sourceGain— the depth is also sampled at the light's own screen position, so covering the bulb with a hand dims the whole room (capped at 85% so it never fully blacks out). - 2026-08-20: Root cause of "I have to get too close to occlude it": with a keyboard/desk close to the lens the normalization window ran 0.33..10.68, so the seated subject normalized to roughly 0.02-0.10 — the entire useful range sat in the bottom 5% of the
lightDepthslider, and a softness of 0.05 spanned all of it. Fixed by dropping HI_PCT from 0.98 to 0.90 (clip the near outliers) plus adepthGammacontrast curve (default 0.45) applied in the shader. Subject now lands nearer 0.3-0.5. - 2026-08-20: Gotcha (self-inflicted) —
autoTuneDepthsamples the rawr8unormtexel on the CPU while the shader compares against the gamma-curved disparity. The two live in different spaces, so the curve has to be applied in both places or auto-tune sets a value that means nothing to the shader. - 2026-08-20: Auto-tune biases the light +0.07 in front of the sampled surface. Parking it exactly on the surface leaves the subject sitting on the occlusion threshold and only half-lit.
- 2026-08-20: Feedback with screenshots — partially covering the bulb killed the light entirely. Root cause:
sourceBlockedsampled the depth at a SINGLE texel (the bulb centre), so coverage was effectively binary; occluding any part of the centre read as fully occluded. Fixed by measuring coverage across the bulb's whole disc: 37 area-weighted concentric-ring taps (r = sqrt(k/rings) so samples spread evenly instead of bunching at the centre), each run through the same smoothstep the shader uses, averaged into a 0..1 fraction. Verified with a horizontal light sweep: values now land at 15/20/21/28/70/97% instead of snapping between 0 and 100. - 2026-08-20: The coverage average is computed on the CPU from
lastDepth, not in the shader. It is one number per frame, so doing 37 taps per pixel would waste ~2M x 37 samples on a value that is constant across the whole image. Passed in as thesourceCoverageuniform. - 2026-08-20:
screenToDepthUvis affine, so the screen->depth scale for the tap offsets is recovered by differencing two mapped points rather than exposing the scale separately. - 2026-08-20: Coverage is EMA-smoothed (0.35). Depth only refreshes at the inference cadence (~27fps), so raw per-frame coverage steps visibly as a hand moves.