Skip to content

Precompute the vision resize tap weights - #1008

Open
namespaceMarcello wants to merge 1 commit into
antirez:mainfrom
namespaceMarcello:vision-resize-tap-tables
Open

Precompute the vision resize tap weights#1008
namespaceMarcello wants to merge 1 commit into
antirez:mainfrom
namespaceMarcello:vision-resize-tap-tables

Conversation

@namespaceMarcello

@namespaceMarcello namespaceMarcello commented Sep 9, 2026

Copy link
Copy Markdown

ds4_resize_rgb_bicubic called ds4_cubic() once per contributing source pixel. A tap weight depends only on the output coordinate and on the source sample it lands on, so a 12 MP photo scaled to the DeepSeek canvas evaluated the kernel about 200 million times to obtain a few thousand distinct values.

This builds one weight row per output sample and accumulates exactly as before: same products, same sums, same order. Tabulating the weights also lets the compiler vectorize the inner loop, which the call used to block, and that is where most of the gain comes from.

Preprocessing is synchronous, right before the GPU vision encode, so this is latency removed from every image on every backend.

Ryzen 9 7940HX, default CFLAGS (-O3 -ffast-math -march=native), mean of a timed batch, ms:

image             DeepSeek V4         GLM 5.3
4032x3024      283.00 -> 140.50   456.00 -> 261.00
2560x1440      112.80 ->  63.12    24.19 ->  25.35
1280x720        35.71 ->  18.74     5.62 ->   5.23
640x480         13.50 ->   7.35     1.70 ->   1.66
3200x400        44.33 ->  24.33     7.83 ->   8.49

GLM only pays the resize when the image does not already match its padded canvas; its flat rows are the exact-size copy path, unchanged and within run to run noise.

On output: a differential harness against the previous implementation over 457 geometries reports zero differing floats at -O0, -O2, -O3, -O3 -ffast-math, -O3 -ffast-math -mavx2 -mfma, -O3 -ffast-math -march=x86-64-v4 and -O3 -ffast-math -fno-tree-vectorize. The corpus is built to make rounding decide the result: two-level checkerboards and stripes at exact and near-exact integer ratios, where three quarters of the values land on an exact .5 tie, plus 1x1 sources and targets, decoder-limit dimensions and padded strides.

One combination differs: -ffast-math with -march=native, which resolves to -march=znver4 here. One geometry out of 457 then moves 48 of its 3108 values by one level of 255, a 1036x1 destination from a 16384-wide checkerboard. Neither AVX2 nor AVX-512 alone reproduces it, so it is a tuning-specific reassociation rather than a property of the change.

For scale, on that same geometry the current code already differs from itself between build flags: 1143 of 3108 floats between an -O2 and a -march=native build, 1224 between -O2 and -mavx2 -mfma. This resampler has no bit-exact output across builds to preserve, and this change stays well inside the spread it already has.

Also: resize now reports allocation failure instead of leaving an unfiltered canvas to be normalized. Failing each of the 16 allocation sites in turn returns a clean error with no leak. The added test asserts a uniform image stays uniform across the content area; it passes on the previous implementation too.

Not tested: no Metal, CUDA or ROCm machine and no GGUF here, so make test, the logprob vectors, the vision quality runs and speed-bench are untested. The file is backend independent and contains no GPU code. I can add the differential harness as an in-tree test if you want it.

#1013 stacks a separable two-pass version on top of this one: three times faster again, but it moves tie values by one level under any flags.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VuqnYdebtpz6NpooFpVY1p

Both image preprocessors resample through the same bicubic filter, and the
filter called ds4_cubic() once per contributing source pixel. A tap weight
depends only on the output coordinate and on which source sample it lands
on, so the same handful of values was recomputed for every output row and
column: a 12 MP photo scaled to the DeepSeek canvas evaluated the kernel
about 200 million times to obtain a few thousand distinct weights.

Build one weight row per output sample instead, then accumulate exactly as
before: same products, same sums, same order. Tabulating the weights also
lets the compiler vectorize the inner loop, which the call to ds4_cubic()
used to block, and that is where most of the speedup comes from.

Preprocessing time on a Ryzen 9 7940HX with the default CFLAGS, -O3
-ffast-math -march=native, mean of a timed batch, in ms:

  image             DeepSeek V4         GLM 5.3
  4032x3024      283.00 -> 140.50   456.00 -> 261.00
  2560x1440      112.80 ->  63.12    24.19 ->  25.35
  1280x720        35.71 ->  18.74     5.62 ->   5.23
  640x480         13.50 ->   7.35     1.70 ->   1.66
  3200x400        44.33 ->  24.33     7.83 ->   8.49

The work is synchronous and single threaded, right before the GPU vision
encode, so this is latency removed from every image on every backend.
DeepSeek V4 always compresses to a small canvas and gains everywhere. GLM
5.3 only pays the resize when the image does not already match its padded
canvas, so its flat rows are the exact-size copy path, unchanged and within
run to run noise.

On output: a differential harness against the previous implementation over
457 geometries reports zero differing floats at -O0, -O2, -O3, -O3
-ffast-math, -O3 -ffast-math -mavx2 -mfma, -O3 -ffast-math
-march=x86-64-v4 and -O3 -ffast-math -fno-tree-vectorize. The corpus is
built to make rounding decide the result: two-level checkerboards and
stripes resampled at exact and near-exact integer ratios, where three
quarters of the output values land on an exact .5 tie, plus 1x1 sources and
targets, decoder-limit dimensions and padded destination strides.

One combination differs: -ffast-math with -march=native, which resolves to
-march=znver4 on this machine. One geometry out of 457 then moves 48 of its
3108 values by one level of 255, a 1036x1 destination from a 16384-wide
checkerboard. Neither AVX2 nor AVX-512 alone reproduces it, so it is a
tuning-specific reassociation rather than a property of the change. For
scale, on that same geometry the current code already differs from itself
between build flags: 1143 of 3108 floats between an -O2 and a -march=native
build, 1224 between -O2 and -mavx2 -mfma. This resampler has no bit-exact
output across builds to preserve, and the change stays well inside the
spread it already has.

The tap tables are the only new allocation, well under 1 MB for a 12 MP
photo, and resize now reports allocation failure so both callers surface it
instead of normalizing an unfiltered canvas. Failing each of the 16
allocation sites in turn returns a clean error with no leak.

Add a flat-image regression to tests/test_deepseek4_vision_image: a uniform
source must stay uniform across the content area whatever the scale, which
pins the tap normalization and the content placement inside the padding.
The check passes on the previous implementation too.

Tested on CPU builds only: this file is backend independent and has no GPU
code, but I have no Metal, CUDA or ROCm machine and no GGUF here, so make
test, the logprob vectors and the vision quality runs are untested.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VuqnYdebtpz6NpooFpVY1p
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant