Albert Martinez

@albertdbio

The object won't move — diagnosing motion collapse in a one-step video model

A one-step causal video-diffusion student renders objects perfectly and refuses to move them. Localizing the failure to the distillation objective, and fixing it for single-digit dollars of compute.

The car is rendered perfectly. It just won't drive across the frame.

I'm building quality-qualified real-time video-to-video on a single GPU. My one-step model paints objects in crisp detail — and then refuses to move them. This is the investigation that found out why, and where the fix actually lives.

1.3B Wan params · 1-step NFE per block · 30.8 fps warm end-to-end · 2.5/10 blind motion

01 — The system

Fast, faithful, and static

CF++1 is my real-time student: a one-step Causal-Forcing++ distillation of Wan-2.1-1.3B. It generates video causally, one latent block at a time — 21 blocks become 81 frames (≈5s at 16 fps) — and streams them through a rolling TAEHV decoder. On one H100 it runs warm at ~27–31 fps. (Four denoising steps on the first block, one plus a clean-cache repopulation on each later block → 45 forwards for 21 blocks.)

The goal is deliberately narrow and honest: quality-qualified real-time on a single GPU — sustained ≥29 fps and ≥7/10 motion quality, non-inferior to a 4-step teacher, under blind multi-judge review. Throughput I have. Quality is the whole game — and quality here means one thing: can the objects move the way you asked?

02 — The symptom

Watch what these clips don't do

Each prompt commands a clear traverse across the frame. The model was given a fixed seed and the exact trajectory in words. Here is what it produced, decoded through the deployed rolling TAEHV, looping:

03 — The re-diagnosis

It was never "morph." It's motion collapse.

For weeks this was logged as "objects morph instead of displacing." But look again at those clips: nothing melts. The car is a car in every frame; the ball stays perfectly round; the walker's jacket never smears. Identity is preserved beautifully. What's missing is the commanded displacement — the object regresses to small, generic, decaying motion near where it started.

The failure isn't appearance-morphing. It's motion-mode collapse: the one-step student produces safe, near-static motion with excellent image fidelity. Identity ✓ · displacement ✗

That distinction changes everything downstream. "Morph" points you at decoders, VAEs, and history smoothing. Motion collapse points you at the distillation objective — the exact signature of a reverse-KL loss that is mode-seeking and settles into the lowest-energy, lowest-motion mode. So I stopped guessing and ran a root-cause audit.

04 — The root-cause audit

Where does the displacement disappear?

The audit ("P0") localizes the failure by holding the latents fixed and swapping only one stage at a time, then blind-scoring motion on the same four prompts. Three cells, one H100, ~$5 of compute:

Cell A — history. Recent-clean KV ring OFF → ON. Motion 3.25 → 2.50: the ring evicts older context, buying +12% fps for −0.75 motion. → dropped

Cell B — decoder state. Rolling TAEHV → reset every block. Motion 5 → 3: resetting adds flicker, so rolling state is helping, not hurting. → not it

Cell C — decoder. Rolling TAEHV → full offline Wan VAE. Motion 5 → 5: the ground-truth decoder renders the same failed trajectory. → the latents

Cell C is the hinge. Decode the identical latents through the deployed TAEHV or through the full ground-truth Wan VAE and you get the same 5/10 motion. If the decoder were smearing displacement into stillness, the reference VAE would recover it. It doesn't:

The failure is not in the decoder and not in history selection. It is baked into the latents the DiT generates. The whole history-selection and decoder-swap family is off the table as a motion cure.

05 — Why the generator collapses

Mode-seeking distillation finds the stillest mode

A one-step causal student is distilled with an asymmetric Distribution-Matching objective — an approximate reverse-KL. Reverse-KL is mode-seeking: it happily concentrates on one plausible mode and abandons the rest. For video, the lowest-cost mode to commit to is the one with the least motion — a stable, well-rendered, barely-moving object minimizes the loss without ever risking a wrong trajectory. That is exactly the behavior in the clips above.

The literature agrees this is a generator/objective problem, and it also warns off the obvious fixes:

• One-Forcing measures that swapping consistency-distillation init for ODE-trajectory init moves VBench Dynamic Degree 23.6 → 52.8 under a matched objective — initialization strongly shapes one-step motion.
• The tempting "just add a term that pulls toward the clean frame" is a trap: One-Forcing's endpoint-MSE ablation drives Dynamic Degree to 1.30 — near-total motion death. A deterministic target annihilates dynamics.
• The adversarial fix isn't free either. AAD-1's clean isolation shows a real-data GAN stage reduces motion (DD 50.3 → 41.5) while buying realism — so the celebrated "add a critic" move trades displacement for consistency.

Net: the fix lives at the generator — in initialization and the distillation objective — and the naive objective patches make motion worse, not better. Which fork? That's the open question P0 couldn't close: is the collapse baked in at consistency-distillation init (Stage 2), or introduced by the DMD self-rollout (Stage 3)?

06 — How the frontier actually does it

The same recipe, minus a scale budget

I reverse-engineered Oasis, Mirage and Lucy from released code, papers and first-party posts. The reassuring finding: it's the same recipe I'm on — a causal framewise DiT emitting one latent at a time, trained with Diffusion Forcing, distilled to few/one step, with bounded self-generated history. Corrected against the primary sources, a few myths fall away:

• "Dynamic noising" is a non-lever. Oasis's much-cited noisy-history trick was deleted from the codebase six minutes after it was added; HEAD ships a constant t=14 tag on clean history — a drift stabilizer, not a motion prior.
• The special silicon is economics, not the breakthrough. Oasis ran real-time on stock H100s; the ASIC is a cost multiplier. The touted low-precision speedups (MXFP8/NVFP4) are Blackwell-only and don't port to H100 — only FP8 does.
• None of the systems tricks teach displacement. Mega-kernels, custom WebRTC, KV policy — all buy latency budget. Motion still lives in the objective and the init.

The takeaway is liberating: real-time-with-motion is achievable algorithmically on the hardware I already have. I don't need their fab. I need their objective.

07 — The fork, resolved

The distillation stage is the killer

The question P0 left open: is the collapse baked in at Stage-2 initialization, or introduced by Stage-3 DMD? The two Stage-2 reference checkpoints — and the official final CF++ student — turned out to be publicly downloadable. So instead of funding a retrain, I generated the same four displacement prompts from every stage of the reference pipeline on one H100 (~$7, zero training) and looked:

Three conclusions fall out at once:

1. Stage-3 DMD introduces the collapse. Every pre-DMD checkpoint moves (messily); every post-DMD final is a beautiful near-still. The mode-seeking objective keeps the high-fidelity static/camera-follow mode and discards object transport.
2. My implementation is exonerated. The official released checkpoint on the reference code reproduces my failure on the same prompts — this is the method, not my port.
3. An init swap alone won't fix it. ODE-init moves more than CD-init pre-DMD (consistent with One-Forcing's numbers), but both movers pass through the same Stage-3 — and that's where the motion dies. The fix must change the Stage-3 objective.

This also reconciles the literature's apparent contradiction: benchmark motion scores (VBench Dynamic Degree on generic prompts) reward any dynamics, including camera motion — the mode DMD keeps. My prompts ban camera motion and command object transport — the mode DMD drops. Both numbers are true; mine measures the one users feel.

Caveats: 4 prompts, seed 0, single run, eyeball-scored montages pending metric batch-scoring; the One-Forcing final checkpoint was incompatible with the reference inference path and produced noise — excluded as invalid, not evidence.

08 — The path forward

Change the objective, not just the init

• Init-vs-DMD fork: done. Resolved above — Stage-3 DMD is where displacement dies. ~$7 of decode instead of a multi-GPU retrain.
• A real displacement metric: shipped. CoTracker3 tracks + camera-compensated flow + DINOv2 identity, red-first tested and calibrated against the blind scores (Spearman ρ≈0.61–0.70). It independently scores every P0 clip as displacement-collapsed.
• Motion-preserving Stage-3: piloted. SGMD (score-gradient matching) replacing DMD, same init, 100 updates on 2×H100 (~$12).
• Cheap inference-only levers: properly measured. KV attention-bias and context-noise sweeps — worth running now that the metric harness can catch a real effect, but they operate downstream of a generator that no longer knows how to move.

The first car that drives — and the trade it exposes

I swapped the Stage-3 objective for score-gradient matching — whose gradient flows through the fake-score network's input Jacobian, the exact path DMD's stop-gradient trick bypasses — and trained 100 updates from the same ODE init. At update 25, the displacement metric reads 2.03 on the vehicle prompt: matching the 4-step init's motion (2.13), 7× the DMD floor (0.29), in a 1-step student.

And the honest other half — the same checkpoint on the ball prompt:

So the finding is a mechanism, not a model: the distillation objective is the dial that trades motion against stability. DMD sits at one extreme (perfect stills), this SGMD config overshoots toward the other (motion, then chaos), and the measured dose-response — 2.03 at update 25 decaying to 0.23 by update 100 — says the target is an operating point in between.

Epilogue — the dial found its setting. Two follow-up runs the same day closed the loop. Normalizing the Fisher weight's ~10¹⁰ dynamic range (the reviewer-flagged suspect) cured the dissolution outright — the ball stays a ball for all 81 frames — and a λ sweep then mapped the trade with a two-axis metric (motion × coherence, machine-scored, every spot-check byte-identical). DMD sits at (0.3 motion, 10 coherence); the winning recipe — normalized Fisher, λ=0.2, updates 20–40 — reaches (2.65, 7.0) and holds (1.7–2.0, 7.3–7.4) across the window. From a parked car to real transport at watchable coherence: no architecture change, no data change — just the objective, its normalization, and one hyperparameter.

Pilot caveats: one seed, one λ, four prompts; several degraded clips were honestly refused by the metric's coherence guardrails.

09 — The rule I work by

A green build is the floor, not the finish line

Every number here is separated warm from cold, reported end-to-end (decode included), and paired with a quality score — never a throughput headline alone. Motion is blind-scored on contact sheets. Findings get pinned as red-first regression tests so a future refactor can't silently undo them. The whole point of P0 was to look at the pixels instead of trusting a metric — which is exactly how "morph" got corrected to "motion collapse," and how an expensive retrain got demoted behind a four-hour decode.

Investigation in progress · results as of 2026-07-21 · one H100 · single-seed cells B/C are directional (n=1), cell A is n=4. CF++1 is a from-scratch student on the Wan-1.3B / Causal-Forcing++ lineage. Clips are the model's own generated output; no real people or footage. Numbers are development evidence pending the exact-stack quality gate — not a published benchmark.