Filter vision images in two separable passes - #1013
Open
namespaceMarcello wants to merge 2 commits into
Open
Conversation
Both image preprocessors resample through the same bicubic filter, and the filter called ds4_cubic() once per contributing source pixel. A tap weight depends only on the output coordinate and on which source sample it lands on, so the same handful of values was recomputed for every output row and column: a 12 MP photo scaled to the DeepSeek canvas evaluated the kernel about 200 million times to obtain a few thousand distinct weights. Build one weight row per output sample instead, then accumulate exactly as before: same products, same sums, same order. Tabulating the weights also lets the compiler vectorize the inner loop, which the call to ds4_cubic() used to block, and that is where most of the speedup comes from. Preprocessing time on a Ryzen 9 7940HX with the default CFLAGS, -O3 -ffast-math -march=native, mean of a timed batch, in ms: image DeepSeek V4 GLM 5.3 4032x3024 283.00 -> 140.50 456.00 -> 261.00 2560x1440 112.80 -> 63.12 24.19 -> 25.35 1280x720 35.71 -> 18.74 5.62 -> 5.23 640x480 13.50 -> 7.35 1.70 -> 1.66 3200x400 44.33 -> 24.33 7.83 -> 8.49 The work is synchronous and single threaded, right before the GPU vision encode, so this is latency removed from every image on every backend. DeepSeek V4 always compresses to a small canvas and gains everywhere. GLM 5.3 only pays the resize when the image does not already match its padded canvas, so its flat rows are the exact-size copy path, unchanged and within run to run noise. On output: a differential harness against the previous implementation over 457 geometries reports zero differing floats at -O0, -O2, -O3, -O3 -ffast-math, -O3 -ffast-math -mavx2 -mfma, -O3 -ffast-math -march=x86-64-v4 and -O3 -ffast-math -fno-tree-vectorize. The corpus is built to make rounding decide the result: two-level checkerboards and stripes resampled at exact and near-exact integer ratios, where three quarters of the output values land on an exact .5 tie, plus 1x1 sources and targets, decoder-limit dimensions and padded destination strides. One combination differs: -ffast-math with -march=native, which resolves to -march=znver4 on this machine. One geometry out of 457 then moves 48 of its 3108 values by one level of 255, a 1036x1 destination from a 16384-wide checkerboard. Neither AVX2 nor AVX-512 alone reproduces it, so it is a tuning-specific reassociation rather than a property of the change. For scale, on that same geometry the current code already differs from itself between build flags: 1143 of 3108 floats between an -O2 and a -march=native build, 1224 between -O2 and -mavx2 -mfma. This resampler has no bit-exact output across builds to preserve, and the change stays well inside the spread it already has. The tap tables are the only new allocation, well under 1 MB for a 12 MP photo, and resize now reports allocation failure so both callers surface it instead of normalizing an unfiltered canvas. Failing each of the 16 allocation sites in turn returns a clean error with no leak. Add a flat-image regression to tests/test_deepseek4_vision_image: a uniform source must stay uniform across the content area whatever the scale, which pins the tap normalization and the content placement inside the padding. The check passes on the previous implementation too. Tested on CPU builds only: this file is backend independent and has no GPU code, but I have no Metal, CUDA or ROCm machine and no GGUF here, so make test, the logprob vectors and the vision quality runs are untested. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VuqnYdebtpz6NpooFpVY1p
Follow-up to the tap tables: the bicubic kernel is separable, and the clamped support box is the product of two independent 1D ranges, so filtering rows first and columns after computes the same normalized average with kx+ky taps per output pixel instead of kx*ky. Row-filtered lines live in a ring sized for the vertical window, whose start grows with the output row, so every source row is filtered exactly once. Preprocessing time on a Ryzen 9 7940HX with the default CFLAGS, -O3 -ffast-math -march=native, mean of a timed batch, in ms: image original tap tables separable 4032x3024 DS4 283.00 140.50 38.69 4032x3024 GLM 456.00 261.00 112.20 2560x1440 DS4 112.80 63.12 19.42 1280x720 DS4 35.71 18.74 8.23 3200x400 DS4 44.33 24.33 9.43 This one is not output-identical, which is why it is a separate commit. The two-pass sum groups the terms differently, so a value that lands exactly on a .5 rounding tie can fall to the other side. Over the same 457 geometry differential corpus, 13 cases differ from the current implementation, always by one level of 255, and only on two-level content: 512x512 -> 170x170 checkerboard 930/86700 1.07% 512x512 -> 341x341 checkerboard 954/348843 0.27% 900x600 -> 300x200 checkerboard 2082/180000 1.16% 512x512 -> 511x511 checkerboard 1362/783363 0.17% 16384x4 -> 1036x1 checkerboard 1392/3108 44.79% 4x16384 -> 1x574 checkerboard 879/1722 51.05% Nothing differs on photographic, noisy or flat content, at any optimization level; the exact counts shift by a few values between build flags, since the previous implementation is itself flag dependent on tie inputs. Ties are common precisely because flat synthetic content averages to a half-integer, so this is a real trade rather than a rounding curiosity: one level of 255 on hard-edged images, for three times the speed of the tap tables. Take whichever half of this branch you prefer. The ring is bounded by the vertical window, under 1 MB for a 12 MP photo, and failing each of the 24 allocation sites in turn returns a clean error with no leak. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VuqnYdebtpz6NpooFpVY1p
namespaceMarcello
force-pushed
the
vision-separable-resize
branch
from
September 9, 2026 20:02
e6cc8b4 to
bab5b5a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1008: the first commit here is that PR, the second is this one. Take either half.
The bicubic kernel is separable, and the clamped support box is the product of two independent 1D ranges, so filtering rows first and columns after computes the same normalized average with kx+ky taps per output pixel instead of kx*ky. Row-filtered lines live in a ring sized for the vertical window, whose start grows with the output row, so every source row is filtered exactly once.
Ryzen 9 7940HX, default CFLAGS (
-O3 -ffast-math -march=native), mean of a timed batch, ms:This one is not output-identical, which is why it is a separate commit. The two-pass sum groups the terms differently, so a value landing exactly on a .5 rounding tie can fall to the other side. Over the same 457 geometry differential corpus, 13 cases differ from the current implementation, always by one level of 255, and only on two-level content:
Nothing differs on photographic, noisy or flat content, at any optimization level. The exact counts shift by a few values between build flags, since the current implementation is itself flag dependent on tie inputs: on the 16384x4 geometry it already differs from itself by 1143 of 3108 floats between an
-O2and a-march=nativebuild.Ties are common precisely because flat synthetic content averages to a half-integer, so this is a real trade rather than a rounding curiosity: one level of 255 on hard-edged images, for three times the speed of the tap tables.
The ring is bounded by the vertical window, under 1 MB for a 12 MP photo. Failing each of the 24 allocation sites in turn returns a clean error with no leak.
Same testing limits as #1008: no Metal, CUDA or ROCm machine and no GGUF here, so
make test, the logprob vectors, the vision quality runs and speed-bench are untested. The file is backend independent and contains no GPU code.🤖 Generated with Claude Code
https://claude.ai/code/session_01VuqnYdebtpz6NpooFpVY1p