LTX-2.3 I2V: two-stage dev pipelines lose the conditioning image after frame 0 (stage-1 half-res anchor); single-stage dev works

[obrhlabs]

Summary

On LTX-2.3, image-to-video with the two-stage dev pipelines (dev-two-stage, dev-two-stage-hq) does not hold the conditioning image: the output starts on the exact uploaded photo for frame 0, then cuts or morphs to a completely different scene. The same image/seed/prompt on distilled and on single-stage dev stays anchored for the full clip.

The misleading part is that frame 0 always matches the photo perfectly — stage 2 re-pins the conditioning latent at full resolution, so the clip looks conditioned before diverging.

Environment

• mlx-video main (installed 2026-06, v0.0.1), Apple Silicon (Mac Studio, macOS 15)
• Weights: prince-canuma/LTX-2.3-dev + prince-canuma/LTX-2.3-distilled
• Text encoder: mlx-community/gemma-3-12b-it-bf16
• PR fix(ltx2): cap encoder channels to match pretrained VAE weights #27 applied (required for the I2V VAE encoder path on LTX-2.3 weights)

Controlled comparison (same start image, same seed where noted)

Start image: photo of two barn swallows perched on a woven birdhouse hanging from a porch ceiling. Prompt: "Two birds in a nest are looking around and singing." All runs 129 frames (97 where noted), --image-strength 1.0. Verified by extracting frames 0 / mid / last with ffmpeg.

Pipeline	Resolution	CFG / APG	Result
distilled	384×576	–	✅ anchored, animates the photo
dev	768×512, 97f	3.0 / off	✅ anchored
dev	256×384	2.0 / on	✅ anchored
dev-two-stage-hq	256×384, seed 1650840191	2.0 / on	❌ hard scene cut after frame 0
dev-two-stage	256×384	2.0 / on	❌ hard scene cut
dev-two-stage-hq	768×512	2.0 / on	❌ no hard cut, but gradually morphs into a lookalike scene (bird species, background materials change)

Attempted rescues on dev-two-stage-hq @ 256×384, seed 1650840191 (the failing baseline) — none keep the actual image:

Variation	Result
--cfg-scale 1.0 --stg-scale 0 (guidance fully off)	❌ still cuts → not a CFG issue
negative prompt "scene cut, jump cut, transition, …"	❌ still cuts
--image-strength 0.9	❌ cuts, blurrier
long detailed prompt describing the photo	⚠️ produces a text-reconstructed similar scene (different birds/birdhouse), still not anchored to the image

Analysis

The conditioning plumbing looks correct in all pipelines — apply_conditioning pins the frame-0 latent, the per-token timestep is sigma × denoise_mask (0 for the conditioned frame), and apply_denoise_mask re-pins each step, in denoise_distilled, denoise_dev, denoise_dev_av, and denoise_res2s_av alike, for both stages.

The failure tracks the two-stage architecture: stage 1 denoises at half the requested resolution, where the single conditioned latent frame is a very small token grid (4×6 at a 256×384 request). That anchor appears too weak for the sampler to propagate scene identity, so stage 1 invents its own video; stage 2 then re-pins frame 0 at full res, producing the "first frame is my photo, rest is another video" output. Consistent with that, the failure softens from a hard cut to a gradual identity morph when the request size doubles (stage 1 at 384×256) — but it never becomes anchored. The HQ stage 1 also runs with the LoRA merged at 0.25 and ~15 steps, which may further weaken adherence.

The guiding-token mechanism in PR #29 (attend to the reference as appended clean tokens instead of relying on the in-place pinned frame) seems like a plausible fix direction for the two-stage pipelines' first-frame conditioning as well, since it would survive the stage-1 downscale better than a 4×6 pinned frame.

Repro

python -m mlx_video.models.ltx_2.generate \
  --prompt "Two birds in a nest are looking around and singing." \
  --model-repo prince-canuma/LTX-2.3-dev --pipeline dev-two-stage-hq \
  --width 256 --height 384 --num-frames 129 --fps 26 --seed 1650840191 \
  --text-encoder-repo mlx-community/gemma-3-12b-it-bf16 \
  --steps 30 --cfg-scale 2.0 --apg \
  --image <any-photo>.png --image-strength 1.0 \
  --output-path out.mp4

Compare against --pipeline dev (single-stage) with identical args: dev holds the image, dev-two-stage-hq cuts away. Happy to run further tests — the weights are local here and the turnaround at 256×384 is ~4 minutes.