<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://pages.vision.ischool.illinois.edu/blog/feed.xml" rel="self" type="application/atom+xml" /><link href="https://pages.vision.ischool.illinois.edu/blog/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-10-06T18:37:55+00:00</updated><id>https://pages.vision.ischool.illinois.edu/blog/feed.xml</id><title type="html">CVML Group Blog</title><subtitle>Notes on computer vision and machine learning</subtitle><author><name>CVML Group</name></author><entry><title type="html">4Director: Directing Video World Models with Rigid 3D Geometry</title><link href="https://pages.vision.ischool.illinois.edu/blog/4director/" rel="alternate" type="text/html" title="4Director: Directing Video World Models with Rigid 3D Geometry" /><published>2026-10-02T00:00:00+00:00</published><updated>2026-10-02T00:00:00+00:00</updated><id>https://pages.vision.ischool.illinois.edu/blog/4director</id><content type="html" xml:base="https://pages.vision.ischool.illinois.edu/blog/4director/"><![CDATA[<p>Joint work with <a href="https://stability.ai/">Stability AI</a>.</p>

<p>A film director stages a shot by deciding where the camera goes and where every actor goes, and leaves <em>how</em> an actor walks or how the light falls to the cast and crew. Video generation models are getting good at the second half: given a text prompt and an image, they produce footage with convincing motion, shading and detail. What they still lack is the first half. If you want the camel to turn around 180° while the camera arcs to the left, there is no reliable way to say so.</p>

<p>We are excited to share <strong>4Director</strong>, a collaboration between our group and Stability AI. 4Director is a video world model that takes exactly the director’s inputs: a single image, a 3D trajectory for the camera, and a rigid 3D trajectory for each object, and generates a video that follows them. The key idea is to make the control signal an explicit <strong>4D scene</strong>: every object is reconstructed once as a complete canonical mesh and then moved by one rigid transformation per frame, sharing a coordinate frame with the camera and a background point cloud. We render that scene as a depth video and train a <strong>Motion Adapter</strong> that turns the geometric scaffold into video, supplying the appearance, illumination and non-rigid dynamics the scaffold leaves out. Because the mesh has a surface on every side, geometry that a turn or a camera move reveals is rendered rather than hallucinated anew in each frame.</p>

<div class="grid grid-3">
  <figure><img src="/blog/assets/4director/hiker_input.jpg" alt="Input image of a hiker on a rocky mountain trail" /><figcaption>Input image</figcaption></figure>
  <figure><video src="/blog/assets/4director/hiker_depth.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Rigid 3D scene rendered to depth (the control)</figcaption></figure>
  <figure><video src="/blog/assets/4director/hiker_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>4Director output</figcaption></figure>
</div>

<h2 id="why-existing-controls-fall-short">Why existing controls fall short</h2>

<p>Controls for object motion in video generation fall into two families, and each misses something a director needs.</p>

<p><strong>Image-plane cues</strong>, such as dragged paths, bounding boxes, masks and point tracks, state where an object should appear in each frame. But a single projection is consistent with many 3D motions: a box does not determine the object’s orientation, a shrinking box cannot distinguish an object moving away from one getting smaller, and a 2D path conflates camera parallax with object motion. <strong>3D-aware controls</strong> fix position with rigid trajectories, 6-DoF poses or 3D boxes, or lift a proxy of the object from the input image: spheres on tracked parts (Perception-as-Control), 3D point tracks over a point cloud (SymphoMotion), or one 3D Gaussian blob per object (VerseCrafter). These remove the depth ambiguity, but their geometry is incomplete. A trajectory or box carries no surface at all, and a lifted proxy only captures the visible shell, so there is nothing to render when a turn brings the far side of the object into view. Point-based proxies are also driven by per-point trajectories, which are tedious to author.</p>

<figure class="full-width">
  <img src="/blog/assets/4director/fig2_controls.jpg" alt="A camel is asked to turn 180° in place. Under 2D box control it barely turns; under 3D point tracks and a 3D Gaussian blob it turns partway or deforms; under rigid 3D geometry control the depth video shows the full silhouette at every step and the generated camel completes the turn." />
  <figcaption>The same request, a 180° turn in place, under four control signals. A 2D bounding box, 3D point trajectories and a 3D Gaussian blob all lead the generator to keep roughly the original heading or distort the camel. Our rigid 3D geometry renders the complete mesh at every orientation, and the generated camel turns all the way round.</figcaption>
</figure>

<p>The table below summarizes what each representation can express. Only the rigid 3D geometry fixes camera trajectory, 3D position, orientation and complete geometry at once, and it is the only one that can insert a brand-new object into the scene.</p>

<table>
  <thead>
    <tr>
      <th>Method</th>
      <th>Camera trajectory</th>
      <th>3D object position</th>
      <th>Object orientation</th>
      <th>Complete geometry</th>
      <th>New object insertion</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>MotionCtrl</td>
      <td>✓</td>
      <td>partial</td>
      <td>✗</td>
      <td>✗</td>
      <td>–</td>
    </tr>
    <tr>
      <td>Perception-as-Control</td>
      <td>✓</td>
      <td>✓</td>
      <td>✗</td>
      <td>✗</td>
      <td>–</td>
    </tr>
    <tr>
      <td>SymphoMotion</td>
      <td>✓</td>
      <td>✓</td>
      <td>partial</td>
      <td>✗</td>
      <td>–</td>
    </tr>
    <tr>
      <td>VerseCrafter</td>
      <td>✓</td>
      <td>✓</td>
      <td>partial</td>
      <td>✗</td>
      <td>–</td>
    </tr>
    <tr>
      <td><strong>4Director (ours)</strong></td>
      <td>✓</td>
      <td>✓</td>
      <td>✓</td>
      <td>✓</td>
      <td>✓</td>
    </tr>
  </tbody>
</table>

<h2 id="rigid-3d-geometry-as-the-control">Rigid 3D geometry as the control</h2>

<figure class="full-width">
  <img src="/blog/assets/4director/fig3_overview.jpg" alt="Overview of 4Director. Left: the input image is lifted into complete canonical meshes for each object and a background point cloud; user-defined object and camera trajectories move them; the scene is rendered to a depth video. Right: the depth video, input image and prompt are encoded and injected by the Motion Adapter's context blocks into a frozen Wan-DiT, which generates the video." />
  <figcaption>Overview. Left: each object is reconstructed once as a complete canonical mesh and moved by one rigid transformation per frame; meshes, background point cloud and camera share one coordinate frame, and the scene is rendered to a depth video. Right: the Motion Adapter, a trainable branch of context blocks, injects this rendering into a frozen pretrained video generator, which supplies appearance, illumination and non-rigid dynamics.</figcaption>
</figure>

<p><strong>Lifting the image.</strong> Given one image, MoGe-2 estimates depth and camera intrinsics, and SAM 2 turns a few clicks on each object into a mask. The camera of the input image defines the reference frame. The background, everything outside the masks, is back-projected into a static point cloud. Each marked object is cut out and reconstructed by Pixal3D into a complete, textured canonical mesh, including the sides the photo does not show, and aligned to its mask and depth. Background, camera and objects now share one coordinate frame.</p>

<p><strong>Directing the scene.</strong> In a 3D viewer, the user sets keyframes for the camera trajectory and for each object’s rigid trajectory, a rotation and translation per frame. Because a turn is literally a rotation and motion in depth is literally a displacement, there is no ambiguity for the generator to resolve. A new object can be inserted by reconstructing a mesh from a separate reference image, placing it in the scene and giving it its own trajectory; its textured mesh is rendered into the input image, which becomes the first frame. Altogether the rigid 3D scene is the background cloud, the set of meshes, their per-frame transformations, the intrinsics and the camera trajectory, of which only the last two vary in time.</p>

<p><strong>Rendering the control.</strong> For each frame, the background and the placed meshes are projected into a depth map in which every pixel takes the depth of the nearest surface, so occlusion ordering is decided before generation. The resulting depth video is the entire control signal. It carries the viewpoint, the rigid motion of every object and their mutual occlusion, and deliberately nothing else: no appearance, no illumination, no limb motion, no background beyond what the image shows. Supplying all of that is the Motion Adapter’s job.</p>

<h2 id="the-motion-adapter">The Motion Adapter</h2>

<p>We build on Wan2.1-VACE-14B, keeping its VAE, umT5 text encoder and DiT frozen as a generic video prior. The Motion Adapter follows VACE’s context-branch design: the VAE encodes the depth video and the input image into a context stream, which eight context blocks, one for every fifth DiT block, propagate with cross-attention to the prompt and inject into the DiT as linearly projected hints. The adapter has 3.0B parameters and is initialized from the released VACE branch.</p>

<p>The crucial point is what the adapter is trained to do. Every training pair comes from a real clip: the control is the depth video rendered from the clip’s recovered rigid 3D scene, and the target is the clip itself, with its first frame as the input image and its caption as the prompt. The control contains only the rigid part of each object’s motion. If a surface point of an object is observed at \(X_i^t = R_o^t Y_i + t_o^t + r_i^t\), the first two terms, the rigid motion of the mesh, are in the control, and the residual \(r_i^t\), a swinging limb or a turning head, is not. To reproduce the clip, the adapter must follow the rigid geometry exactly and let the generator fill in the rest. It is trained with the standard flow-matching objective, for three epochs on 24 GPUs at 832×480 and 81 frames. Because the adapter only ever sees the scene as a depth video, a hand-authored scene is treated exactly like a recovered one, so new trajectories and inserted objects need no extra training.</p>

<h2 id="realcod-rigid-20774-clips-with-rigid-3d-scenes">RealCOD-Rigid: 20,774 clips with rigid 3D scenes</h2>

<p>No dataset pairs videos with complete object meshes and per-frame rigid transformations, and nobody can annotate that by hand at scale. So we built an automatic pipeline and ran it on RealCOD-25K, a collection of 81-frame stock and web clips that come with a text prompt and SAM3 masks for one or two objects. Four stages recover the rigid 3D scene of each clip: MegaSaM (with MoGe-2 as depth prior and UniDepthV2 as metric branch) estimates per-frame depth, intrinsics and the camera trajectory; Pixal3D reconstructs each object’s canonical mesh from the first frame and a robust similarity fit places it in the scene; TAPIP3D tracks 3D points on the object through the clip, and a robust alternating fit finds the object’s <em>stable core</em>, the tracks that move together rigidly, so that a swinging arm does not drag the whole-body pose with it; finally the placed meshes and background are rendered to the depth video. Clips in which any object fails a geometric or silhouette consistency gate are dropped, which leaves 20,774 training clips. Genuine occlusion is kept, so an object is allowed to disappear for some frames.</p>

<h2 id="measuring-object-control-identity-gated-iou">Measuring object control: Identity-Gated IoU</h2>

<p>Camera control has a standard metric, the rotation and translation error between the requested trajectory and the one re-estimated from the generated video. Object control did not. Mask IoU against the source clip’s masks tells you whether <em>something</em> is where the trajectory says, but it happily rewards an object that is in the right place and has turned into a different object. We introduce <strong>Identity-Gated IoU (IG-IoU)</strong>: on 16 frames spread uniformly over the clip, a vision–language judge (Qwen3-VL) compares the generated object crop to the reference crop and decides whether it is still intact and the same object. A frame that passes contributes its mask IoU; a frame that fails contributes zero but stays in the denominator. The fraction of frames that pass is reported separately as the recognition rate. Because the reference mask is the object’s silhouette at its true pose, an orientation error that changes the silhouette is penalized too.</p>

<h2 id="results">Results</h2>

<p>We compare against four public methods for joint camera and object control, MotionCtrl, Perception-as-Control (PaC), SymphoMotion and VerseCrafter, on 100 held-out clips of RealCOD-25K. To be fair, every method is given the same motion to follow: the camera and object trajectories recovered from each clip are converted into whatever control each baseline expects, from 2D trajectories to 3D Gaussians.</p>

<table>
  <thead>
    <tr>
      <th>Method</th>
      <th>FID ↓</th>
      <th>FVD ↓</th>
      <th>CLIP-SIM ↑</th>
      <th>RotErr (°) ↓</th>
      <th>TransErr ↓</th>
      <th>Recognition ↑</th>
      <th>IG-IoU ↑</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>MotionCtrl</td>
      <td>115.6</td>
      <td>1653.3</td>
      <td>27.5</td>
      <td>4.74</td>
      <td>0.231</td>
      <td>32.6</td>
      <td>18.8</td>
    </tr>
    <tr>
      <td>Perception-as-Control</td>
      <td>55.7</td>
      <td>1454.6</td>
      <td>30.2</td>
      <td>7.00</td>
      <td>0.228</td>
      <td>80.5</td>
      <td>40.8</td>
    </tr>
    <tr>
      <td>VerseCrafter</td>
      <td>55.0</td>
      <td>484.7</td>
      <td>31.7</td>
      <td>6.16</td>
      <td>0.174</td>
      <td>94.8</td>
      <td>54.8</td>
    </tr>
    <tr>
      <td>SymphoMotion</td>
      <td>45.8</td>
      <td>405.4</td>
      <td>30.9</td>
      <td>3.89</td>
      <td>0.132</td>
      <td>90.2</td>
      <td>52.9</td>
    </tr>
    <tr>
      <td><strong>4Director</strong></td>
      <td><strong>44.1</strong></td>
      <td><strong>370.4</strong></td>
      <td><strong>31.8</strong></td>
      <td><strong>3.65</strong></td>
      <td><strong>0.122</strong></td>
      <td><strong>94.9</strong></td>
      <td><strong>60.4</strong></td>
    </tr>
  </tbody>
</table>

<p class="table-note">Joint camera and object control on 100 held-out RealCOD-25K clips. Camera errors follow the CameraCtrl protocol; recognition rate and IG-IoU are in percent. See the paper for the eight VBench-I2V dimensions, on which 4Director also leads.</p>

<p>4Director is best on every metric, and the margin is widest where it should be: IG-IoU rises from 54.8 for VerseCrafter to 60.4 at the <em>same</em> recognition rate (94.9 versus 94.8). In other words, 4Director does not keep the object recognizable more often than VerseCrafter; it places it more accurately, which is what a complete mesh with a fixed orientation buys you over a Gaussian blob. MotionCtrl, driven by 2D trajectories, keeps the object recognizable in only a third of frames.</p>

<p>Automatic metrics compress a lot. In a user study, 20 participants rated the five methods on nine cases for visual quality, control accuracy and consistency on a 1–5 scale. 4Director scored 4.62, 4.82 and 4.69; the next best method on each criterion scored 3.69, 2.39 and 3.88. The gap on control accuracy, 4.82 against 2.39, is the whole story of the paper in one number.</p>

<p><strong>Object rotation.</strong> The hardest case for the baselines is a turn. Below, a black swan and a car are each asked to rotate 180° in place with the camera held still. The baselines keep the original heading, turn partway, or lose the subject; 4Director completes the turn and shows the side that comes into view.</p>

<div class="grid grid-3">
  <figure><img src="/blog/assets/4director/swan_input.jpg" alt="Input image of a black swan on a pond" /><figcaption>Input</figcaption></figure>
  <figure><video src="/blog/assets/4director/swan_motionctrl.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>MotionCtrl</figcaption></figure>
  <figure><video src="/blog/assets/4director/swan_pac.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Perception-as-Control</figcaption></figure>
  <figure><video src="/blog/assets/4director/swan_symphomotion.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>SymphoMotion</figcaption></figure>
  <figure><video src="/blog/assets/4director/swan_versecrafter.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>VerseCrafter</figcaption></figure>
  <figure><video src="/blog/assets/4director/swan_ours.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>4Director (ours)</figcaption></figure>
</div>

<div class="grid grid-3">
  <figure><img src="/blog/assets/4director/car_input.jpg" alt="Input image of a car at a roundabout" /><figcaption>Input</figcaption></figure>
  <figure><video src="/blog/assets/4director/car_motionctrl.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>MotionCtrl</figcaption></figure>
  <figure><video src="/blog/assets/4director/car_pac.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Perception-as-Control</figcaption></figure>
  <figure><video src="/blog/assets/4director/car_symphomotion.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>SymphoMotion</figcaption></figure>
  <figure><video src="/blog/assets/4director/car_versecrafter.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>VerseCrafter</figcaption></figure>
  <figure><video src="/blog/assets/4director/car_ours.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>4Director (ours)</figcaption></figure>
</div>

<p><strong>Leaving and re-entering the view.</strong> A representation with a trajectory defined at every frame handles a case that trips up proxies lifted from pixels. Here the camel is directed to walk out of the frame and come back while the camera stays fixed. In 4Director the mesh is simply not rendered while it is outside the view and rendered again from the same mesh when it returns, so the model shows an empty yard, fills in the ground the camel stood on, and brings the same camel back. MotionCtrl, SymphoMotion and VerseCrafter keep the camel in frame throughout; Perception-as-Control lets it leave but cannot preserve its shape when it returns.</p>

<div class="grid grid-3">
  <figure><img src="/blog/assets/4director/camel_exit_input.jpg" alt="Input image of a camel in an enclosure" /><figcaption>Input</figcaption></figure>
  <figure><video src="/blog/assets/4director/camel_exit_motionctrl.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>MotionCtrl</figcaption></figure>
  <figure><video src="/blog/assets/4director/camel_exit_pac.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Perception-as-Control</figcaption></figure>
  <figure><video src="/blog/assets/4director/camel_exit_symphomotion.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>SymphoMotion</figcaption></figure>
  <figure><video src="/blog/assets/4director/camel_exit_versecrafter.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>VerseCrafter</figcaption></figure>
  <figure><video src="/blog/assets/4director/camel_exit_ours.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>4Director (ours)</figcaption></figure>
</div>

<p><strong>What does the control need to encode?</strong> An ablation swaps the rigid 3D geometry for three simpler controls trained with the same recipe. A 2D box encodes neither depth nor orientation and scores lowest (IG-IoU 48.5). A 3D box keeps depth and orientation but has no surface (50.3). A 3D mesh that translates but does not rotate keeps the complete surface but loses orientation (53.0); in that setting the motorcycle in the test clip ends up facing the camera instead of turning sideways. Only the full rigid 3D geometry, with depth, rotation and a complete surface, reaches 60.4, and the four controls rank in the same order on FID and FVD. Notably, recognition rate stays between 91.9 and 95.0 for all four, so what the simpler controls lose is placement, not identity.</p>

<h2 id="composing-shots">Composing shots</h2>

<p>Since every object is a mesh in a shared 3D space, directing becomes compositional. The hiker sequence below starts from one photo. We first move the hiker along a path; then we insert a camel reconstructed from a separate reference image and give it its own trajectory; then we add a camera move on top. Each row shows the depth control the adapter receives and the video it generates. The adapter never saw an insertion pair during training; it only sees depth, so a composed scene looks to it like any other.</p>

<div class="grid grid-2">
  <figure><img src="/blog/assets/4director/hiker_input.jpg" alt="Input image of a hiker" /><figcaption>Input image</figcaption></figure>
  <figure><img src="/blog/assets/4director/camel_ref.jpg" alt="Reference image of a camel used for object insertion" /><figcaption>Reference image for the inserted camel</figcaption></figure>
  <figure><video src="/blog/assets/4director/hiker_camel_depth.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Hiker + inserted camel: depth control</figcaption></figure>
  <figure><video src="/blog/assets/4director/hiker_camel_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Hiker + inserted camel: output</figcaption></figure>
  <figure><video src="/blog/assets/4director/hiker_camel_camA_depth.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>+ camera trajectory A: depth control</figcaption></figure>
  <figure><video src="/blog/assets/4director/hiker_camel_camA_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>+ camera trajectory A: output</figcaption></figure>
  <figure><video src="/blog/assets/4director/hiker_camel_camB_depth.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>+ camera trajectory B: depth control</figcaption></figure>
  <figure><video src="/blog/assets/4director/hiker_camel_camB_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>+ camera trajectory B: output</figcaption></figure>
</div>

<p>The same recipe works with less friendly co-stars. Below, a bear is first turned in place, then joined by a T-Rex inserted from a reference image, then filmed with a moving camera.</p>

<div class="grid grid-2">
  <figure><img src="/blog/assets/4director/bear_input.jpg" alt="Input image of a brown bear among rocks" /><figcaption>Input image</figcaption></figure>
  <figure><img src="/blog/assets/4director/trex_ref.jpg" alt="Reference image of a T-Rex model used for object insertion" /><figcaption>Reference image for the inserted T-Rex</figcaption></figure>
  <figure><video src="/blog/assets/4director/bear_depth.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Bear rotation: depth control</figcaption></figure>
  <figure><video src="/blog/assets/4director/bear_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Bear rotation: output</figcaption></figure>
  <figure><video src="/blog/assets/4director/bear_trex_depth.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>+ inserted T-Rex: depth control</figcaption></figure>
  <figure><video src="/blog/assets/4director/bear_trex_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>+ inserted T-Rex: output</figcaption></figure>
  <figure><video src="/blog/assets/4director/bear_trex_cam_depth.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>+ camera trajectory: depth control</figcaption></figure>
  <figure><video src="/blog/assets/4director/bear_trex_cam_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>+ camera trajectory: output</figcaption></figure>
</div>

<p>Beyond the controlled comparisons, the model carries over to subjects far from the training distribution, including game characters and fast vehicles, with the generator supplying the running, skiing and jumping that the rigid control leaves unspecified.</p>

<div class="grid grid-3">
  <figure><video src="/blog/assets/4director/skier_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Skier</figcaption></figure>
  <figure><video src="/blog/assets/4director/sportscar_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Sports car</figcaption></figure>
  <figure><video src="/blog/assets/4director/rooftop_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Rooftop jump (game character)</figcaption></figure>
</div>

<h2 id="limitations">Limitations</h2>

<p>The representation controls motion at the level of a rigid body, which is both its strength and its boundary. An articulated or deforming object moves as one whole, so finer motion such as running, jumping or the swing of a limb cannot be prescribed; it is left to the generator. Usually the generator does the right thing, but in a failure case from the paper a breakdancer in a handstand is translated sideways as a rigid body, and the generated dancer simply holds the handstand throughout, faithfully following the rigid silhouette instead of dancing. Extending the rigid 3D scene with articulated parts, so that such motion can be prescribed too, is the natural next step. A lighter variant of the adapter, trained with rank-16 LoRA (15.3M parameters), still beats all four baselines on FID, FVD and IG-IoU, though the full 3.0B adapter leads it on every metric.</p>

<h2 id="read-more">Read more</h2>

<p>The <a href="https://downloads.cvml.group/preprints/cao2026fordirector.pdf">paper</a> is on arXiv (2610.02160). The <a href="https://stability-ai.github.io/4director/">project page</a> has interactive point clouds you can rotate for every case, a qualitative comparison carousel against all four baselines, and the full set of authored trajectories and insertions. The <a href="https://github.com/VVeiCao/4Director">code</a> is on GitHub. 4Director builds on the same intuition as our earlier <a href="/blog/freeorbit4d/">FreeOrbit4D</a>: give the video generator complete geometry for what it cannot see, and it will stop making it up.</p>

<div class="language-bibtex highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">@article</span><span class="p">{</span><span class="nl">cao20264director</span><span class="p">,</span>
  <span class="na">title</span>   <span class="p">=</span> <span class="s">{{4Director}: Controlling Video World Models with Rigid {3D} Geometry}</span><span class="p">,</span>
  <span class="na">author</span>  <span class="p">=</span> <span class="s">{Wei Cao and Hao Zhang and Vikram Voleti and Yuqun Wu and Mallikarjun B R and Shimon Vainer and Mark Boss and Yaoyao Liu}</span><span class="p">,</span>
  <span class="na">journal</span> <span class="p">=</span> <span class="s">{arXiv preprint arXiv:2610.02160}</span><span class="p">,</span>
  <span class="na">year</span>    <span class="p">=</span> <span class="s">{2026}</span>
<span class="p">}</span>
</code></pre></div></div>]]></content><author><name>https://vveicao.github.io/</name></author><category term="video-generation" /><category term="world-models" /><category term="3d-control" /><category term="camera-control" /><summary type="html"><![CDATA[A video world model that lets you stage a shot from a single image: every object becomes a complete 3D mesh moved by one rigid transformation per frame, the camera lives in the same coordinate frame, and a Motion Adapter turns the rendered depth video into footage with plausible motion, appearance and lighting.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://pages.vision.ischool.illinois.edu/blog/assets/4director/fig1_teaser.jpg" /><media:content medium="image" url="https://pages.vision.ischool.illinois.edu/blog/assets/4director/fig1_teaser.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">How to Teach Large Multimodal Models New Skills?</title><link href="https://pages.vision.ischool.illinois.edu/blog/teach-lmm-new-skills/" rel="alternate" type="text/html" title="How to Teach Large Multimodal Models New Skills?" /><published>2026-09-08T00:00:00+00:00</published><updated>2026-09-08T00:00:00+00:00</updated><id>https://pages.vision.ischool.illinois.edu/blog/teach-lmm-new-skills</id><content type="html" xml:base="https://pages.vision.ischool.illinois.edu/blog/teach-lmm-new-skills/"><![CDATA[<p>Large multimodal models (LMMs) such as LLaVA and Qwen2.5-VL answer questions about images in natural language, and they are good at a remarkable range of things. But they are not good at everything. Ask one to count the people in a crowd, read an analog clock, or diagnose a pathology slide, and it will often fail. Training a new LMM from scratch costs millions of dollars and weeks of compute, so the practical route is to fine-tune the model you have. The problem is that naive fine-tuning on one narrow skill tends to erode the broad abilities the model came with, the familiar <strong>catastrophic forgetting</strong>. So: how do we teach an LMM something new without it forgetting the old?</p>

<p>Our ECCV 2026 paper takes this question literally. We build a controlled test bed, fine-tune three families of LMMs on a sequence of five new skills, and watch what happens to eight general benchmarks along the way. Two findings surprised us, and chasing down <em>why</em> led to two very simple training recipes that learn almost as well as full fine-tuning while forgetting almost nothing.</p>

<h2 id="two-surprises">Two surprises</h2>

<p>Our first experiments simply asked <em>where</em> in the model to put the gradient. An LMM has three parts: a vision encoder that turns the image into tokens, a projector that maps them into the language model’s width, and a decoder-only language model. Inside each transformer block of the language model there are, in turn, two sub-blocks: the self-attention projections (the query, key, value and output matrices, which we call <strong>SA Proj.</strong>) and the feed-forward <strong>MLP</strong>. We fine-tuned each of these in isolation, and the full model, on a sequence of five target tasks.</p>

<p>The first surprise is in the hero figure above. Tuning only the self-attention projections learns the target task nearly as well as tuning the whole language model, yet across the entire five-task sequence its held-out accuracy never moves. We were not aware of any other instance of strong learning without forgetting in the absence of replay, model expansion or heavy regularization.</p>

<p>The second surprise is the shape of the other curves. Held-out performance does not decline monotonically as more tasks are learned. It crashes after the counting task (PixmoCount) and then largely <em>recovers</em> after training on the next task, medical VQA, which has nothing to do with the benchmarks that were lost. With full-model tuning the held-out average falls from 76 to 63 and then climbs back to 74. Whatever was “forgotten” was evidently not gone.</p>

<h2 id="the-test-bed">The test bed</h2>

<p>To study this carefully we needed a protocol that measures learning and forgetting on equal footing. We chose five target skills that are common in real use and that LMMs typically struggle with, each with a different answer format: fine-grained bird classification (CUB-200), counting (PixmoCount), medical VQA (PathVQA), OCR reading (TextVQA) and reading analog clocks (TimeClock). Together they contain about 108K training samples. The model is fine-tuned on these one after another, with no rehearsal of earlier tasks, and after each stage we evaluate both the target tasks and eight held-out benchmarks that cover broad vision–language ability: AI2D, ChartQA, DocVQA, InfoVQA, RealWorldQA, SeedBench, ScienceQA and MMStar.</p>

<p>From this we compute four numbers per method, all as percentage-point changes from the base model: <strong>target learning</strong> (how much each task improves when it is the one being trained), <strong>target forgetting</strong> (how much earlier target tasks drop by the end of the sequence), <strong>target overall</strong> (the net change on all target tasks at the end) and <strong>held-out forgetting</strong> (the change in the held-out average at the end). Main results on LLaVA-OneVision are averaged over three different task orders.</p>

<h2 id="where-to-tune">Where to tune</h2>

<table>
  <thead>
    <tr>
      <th>Tuned component</th>
      <th>Target learning ↑</th>
      <th>Target forgetting ↑</th>
      <th>Target overall ↑</th>
      <th>Held-out forgetting ↑</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Full model</td>
      <td>+29.9</td>
      <td>−25.9</td>
      <td>+9.2</td>
      <td>−27.4</td>
    </tr>
    <tr>
      <td>Vision encoder</td>
      <td>+9.6</td>
      <td>−12.7</td>
      <td>−0.5</td>
      <td>−10.8</td>
    </tr>
    <tr>
      <td>Projector</td>
      <td>+2.3</td>
      <td>−0.8</td>
      <td>+1.7</td>
      <td>−1.3</td>
    </tr>
    <tr>
      <td>LLM</td>
      <td>+31.8</td>
      <td>−23.5</td>
      <td>+13.0</td>
      <td>−23.3</td>
    </tr>
    <tr>
      <td>SA Proj.</td>
      <td>+24.9</td>
      <td>−2.3</td>
      <td>+23.1</td>
      <td><strong>−0.6</strong></td>
    </tr>
    <tr>
      <td>SA Proj. (QKV only)</td>
      <td>+14.9</td>
      <td>−0.5</td>
      <td>+14.5</td>
      <td>+0.2</td>
    </tr>
    <tr>
      <td>MLP</td>
      <td>+31.1</td>
      <td>−19.5</td>
      <td>+15.5</td>
      <td>−15.7</td>
    </tr>
    <tr>
      <td><strong>MLP (Gate &amp; Up)</strong></td>
      <td>+30.5</td>
      <td>−4.2</td>
      <td><strong>+27.1</strong></td>
      <td>−2.1</td>
    </tr>
  </tbody>
</table>

<p class="table-note">LLaVA-OneVision, percentage-point change from the base model (43.9 on target tasks, 76.4 on held-out), averaged over three five-task orders.</p>

<p>A few patterns are clear. Learning lives in the language model: tuning the LLM gives the strongest gains, while tuning the vision encoder learns little and still damages general ability, and the projector barely moves anything. But full-LLM tuning also forgets the most. Opening up the LLM reveals the two sub-blocks behave very differently. Tuning the MLP learns about as much as the whole LLM and forgets somewhat less. Tuning the self-attention projections learns a bit less but forgets essentially nothing.</p>

<p>The last row is the one we ended up recommending most often. The MLP in these models computes <code class="language-plaintext highlighter-rouge">W_down(SiLU(x W_gate) ⊙ (x W_up))</code>. If we tune the gate and up matrices but freeze the down projection, we keep almost all of the learning of full-MLP tuning (+30.5 versus +31.1) while cutting held-out forgetting from −15.7 to −2.1, and we get the best end-of-sequence target score of any method.</p>

<h2 id="why-forgetting-as-output-distribution-shift">Why: forgetting as output distribution shift</h2>

<p>Why would the self-attention projections be so robust, and why would the down projection of the MLP matter so much? Mechanistic studies of transformers offer a division of labor. Attention heads mostly <em>route</em>: they decide where in the context to read from and mix the corresponding values. MLPs behave like key–value memories: the gate and up matrices detect input patterns, and the down projection writes features back into the residual stream that are aligned with groups of vocabulary tokens, directly shaping the next-token distribution.</p>

<p>That suggested a hypothesis. When you fine-tune on a narrow task whose answers all look alike (every counting answer is “There are N …”), the model has a shortcut available: instead of learning to count better, it can simply bias its output distribution toward numeric tokens. The write-back paths, the MLP down projection above all, are exactly where such a bias would be installed. If much of what we measure as forgetting is really this kind of response bias, then (a) restricting updates to routing (SA Proj.) or to feature detection (MLP gate and up) should limit it, and (b) training on a subsequent task with a different answer format should undo it, which is the recovery we saw.</p>

<p>To test this we designed a <strong>counting-bias probe</strong>. Take a fixed batch of ordinary captioning images, let the model generate captions with greedy decoding, and at every decoding position record the maximum probability the model assigns to any numeric token. Averaged over positions and images, this number tells us how strongly the model is being pulled toward saying numbers when it has no reason to.</p>

<figure class="full-width">
  <img src="/blog/assets/teach-lmm/fig3_probe.png" alt="Three plots over training steps on the counting task: PixmoCount accuracy rises for all tuning choices; held-out accuracy drops for LLM and MLP but not for SA Proj., MLP (Gate &amp; Up) or MLP with LwF; the likelihood of numeric tokens on captions surges for LLM and MLP and stays near baseline for the others." />
  <figcaption>Tuning LLaVA-OneVision for counting. Left: all variants learn to count. Middle: held-out accuracy collapses for full-LLM and full-MLP tuning, stays flat for SA Proj. and MLP (Gate &amp; Up), and is preserved by MLP tuning with a distillation loss (LwF). Right: the probability of emitting numeric tokens on a captioning set tracks the middle plot almost exactly.</figcaption>
</figure>

<p>The probe co-varies with forgetting remarkably well: across training steps and across methods the Spearman correlation between numeric-token likelihood and held-out drop is 0.84. Full-LLM and full-MLP tuning both learn to count and both start sprinkling numbers into their captions. SA Proj. and MLP (Gate &amp; Up) learn to count without doing so. A layer-wise analysis in the appendix attributes most of the induced logit shift to the MLP blocks, and extending the probe across three sequential stages shows the numeric bias spiking when counting is learned and receding when the next task is learned, in lockstep with the held-out accuracy falling and recovering.</p>

<p>The qualitative behavior is telling. After 1K steps of counting on PixmoCount, both the MLP-tuned and the SA Proj.-tuned model answer counting questions correctly. But ask the MLP-tuned model what is in a photo and it replies “There are 2 photos in the photo.” The visual understanding is intact, since it still recognizes the objects and relations, but the output has been reframed as a counting statement. The SA Proj.-tuned model still writes a detailed paragraph.</p>

<figure class="full-width">
  <img src="/blog/assets/teach-lmm/fig4_qualitative.png" alt="After tuning on counting, both the MLP-tuned and SA Proj.-tuned models count people correctly, but the MLP-tuned model describes a slide as 'There are 2 photos in the photo' and a greeting card as a list of counts, while the SA Proj.-tuned model gives full captions." />
  <figcaption>Outputs after tuning MLP (middle) or SA Proj. (right) on the counting task. Both count correctly. Only the SA Proj. model still captions.</figcaption>
</figure>

<h2 id="two-recipes">Two recipes</h2>

<p>This gives a clear answer to the title question. To teach an LMM a new skill without forgetting, restrict the update to components that do not directly write to the output distribution:</p>

<ul>
  <li><strong>SA Proj.</strong>: tune the query, key, value and output projections of every attention block, freeze everything else. Prefer this when stability on general benchmarks is paramount.</li>
  <li><strong>MLP (Gate &amp; Up)</strong>: tune the gate and up projections of every MLP block, freeze the down projections. Prefer this when you want near-maximal learning with limited forgetting.</li>
</ul>

<p>Either is a few lines of code on top of any existing training script:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">model</span><span class="p">.</span><span class="nf">parameters</span><span class="p">():</span>
    <span class="n">p</span><span class="p">.</span><span class="n">requires_grad</span> <span class="o">=</span> <span class="bp">False</span>

<span class="c1"># Recipe 1: self-attention projections only
</span><span class="n">sa_keys</span> <span class="o">=</span> <span class="p">(</span><span class="sh">"</span><span class="s">self_attn.q_proj</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">self_attn.k_proj</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">self_attn.v_proj</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">self_attn.o_proj</span><span class="sh">"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">model</span><span class="p">.</span><span class="nf">named_parameters</span><span class="p">():</span>
    <span class="k">if</span> <span class="nf">any</span><span class="p">(</span><span class="n">k</span> <span class="ow">in</span> <span class="n">name</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">sa_keys</span><span class="p">):</span>
        <span class="n">p</span><span class="p">.</span><span class="n">requires_grad</span> <span class="o">=</span> <span class="bp">True</span>

<span class="c1"># Recipe 2 (alternative): MLP gate &amp; up only, down projection stays frozen
</span><span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">model</span><span class="p">.</span><span class="nf">named_parameters</span><span class="p">():</span>
    <span class="k">if</span> <span class="sh">"</span><span class="s">mlp.gate_proj</span><span class="sh">"</span> <span class="ow">in</span> <span class="n">name</span> <span class="ow">or</span> <span class="sh">"</span><span class="s">mlp.up_proj</span><span class="sh">"</span> <span class="ow">in</span> <span class="n">name</span><span class="p">:</span>
        <span class="n">p</span><span class="p">.</span><span class="n">requires_grad</span> <span class="o">=</span> <span class="bp">True</span>
</code></pre></div></div>

<p>No extra modules, no replay buffer, no per-stage hyperparameters, and no extra forward passes. On four H100 GPUs, SA Proj. tuning runs at 1.46 samples per second, faster than LoRA (1.27) and much faster than a mixture-of-experts variant (0.44), because there are no adapters or routers in the compute graph.</p>

<h2 id="does-it-hold-beyond-one-model">Does it hold beyond one model?</h2>

<p>We repeated the five-task curriculum on LLaVA-NeXT (LLaMA-3 8B) and Qwen2.5-VL (7B). The numbers differ, but the picture does not.</p>

<table>
  <thead>
    <tr>
      <th>Tuned component</th>
      <th>LLaVA-NeXT: target overall ↑</th>
      <th>LLaVA-NeXT: held-out ↑</th>
      <th>Qwen2.5-VL: target overall ↑</th>
      <th>Qwen2.5-VL: held-out ↑</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Full model</td>
      <td>+15.4</td>
      <td>−32.0</td>
      <td>+13.1</td>
      <td>−17.5</td>
    </tr>
    <tr>
      <td>Vision + projector</td>
      <td>−1.3</td>
      <td>−13.4</td>
      <td>+4.9</td>
      <td>−6.2</td>
    </tr>
    <tr>
      <td>LLM</td>
      <td>+19.3</td>
      <td>−35.9</td>
      <td>+12.1</td>
      <td>−24.6</td>
    </tr>
    <tr>
      <td>SA Proj.</td>
      <td>+21.9</td>
      <td><strong>−7.7</strong></td>
      <td>+14.9</td>
      <td><strong>+0.6</strong></td>
    </tr>
    <tr>
      <td>MLP</td>
      <td><strong>+26.6</strong></td>
      <td>−16.3</td>
      <td>+13.9</td>
      <td>−10.9</td>
    </tr>
    <tr>
      <td>MLP (Gate &amp; Up)</td>
      <td>+20.9</td>
      <td>−8.7</td>
      <td><strong>+17.1</strong></td>
      <td>−4.6</td>
    </tr>
  </tbody>
</table>

<p class="table-note">Percentage-point change from each base model after the full five-task sequence.</p>

<p>LLaVA-NeXT is far more prone to forgetting in general, yet SA Proj. and MLP (Gate &amp; Up) still forget less than half as much as full-MLP tuning and less than a quarter as much as full-LLM tuning. On Qwen2.5-VL, SA Proj. actually shows a little positive transfer to the held-out suite, and MLP (Gate &amp; Up) gives the best overall target score with essentially zero target forgetting. Its absolute gains look smaller only because its base model starts much higher.</p>

<h2 id="comparison-to-the-usual-mitigations">Comparison to the usual mitigations</h2>

<p>Selective tuning is not the only way to limit forgetting, so we also benchmarked the standard pillars of continual learning, all applied to the MLP layers: Learning without Forgetting (a KL distillation loss against the previous-stage model), LoRA, a mixture-of-experts expansion, and WiSE-FT weight interpolation.</p>

<figure class="full-width">
  <img src="/blog/assets/teach-lmm/fig5_baselines.png" alt="Average target-task and held-out accuracy after each stage of the five-task sequence for SA Proj., MLP, MLP (Gate &amp; Up), and MLP with LwF, LoRA, MoE and WiSE-FT." />
  <figcaption>Sequential tuning on LLaVA-OneVision. Left: average accuracy on the target tasks after each stage. Right: average held-out accuracy. MLP (Gate &amp; Up) reaches the highest target accuracy while staying close to the baseline on held-out; SA Proj. is the flattest held-out line of all.</figcaption>
</figure>

<p>Distillation does curb forgetting reliably, which fits the output-shift story since it penalizes exactly that, but it can cost target performance. WiSE-FT preserves held-out accuracy well but needs task-dependent blending coefficients chosen per stage. LoRA learns noticeably less, and the mixture-of-experts neither learns nor retains as well as the simple recipes. SA Proj. and MLP (Gate &amp; Up) match or beat all of them on the learning–stability trade-off while being the simplest option on the table.</p>

<h2 id="limitations">Limitations</h2>

<p>Our study covers 7B to 8B open models. We have not tested whether the same recipes hold for much larger or commercial LMMs, or when the adaptation data scales to billions of samples. We did observe a little forgetting when we kept pushing with very large long-tailed tasks such as iNaturalist classification, so “no forgetting” should be read as “very little forgetting under realistic skill-adaptation workloads,” not as a guarantee.</p>

<h2 id="read-more">Read more</h2>

<p>The paper appears at ECCV 2026 as a spotlight. The <a href="https://github.com/jessemelpolio/LMM_CL">code</a> includes the full evaluation suite (five target tasks and eight held-out benchmarks via lmms-eval), the component-wise tuning scripts, the counting-bias probe, and reference implementations of the LwF, LoRA, MoE and WiSE-FT baselines.</p>

<div class="language-bibtex highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">zhu2026teach</span><span class="p">,</span>
  <span class="na">title</span>     <span class="p">=</span> <span class="s">{How To Teach Large Multimodal Models New Skills?}</span><span class="p">,</span>
  <span class="na">author</span>    <span class="p">=</span> <span class="s">{Zhen Zhu and Yiming Gong and Yao Xiao and Yaoyao Liu and Derek Hoiem}</span><span class="p">,</span>
  <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{European Conference on Computer Vision (ECCV)}</span><span class="p">,</span>
  <span class="na">year</span>      <span class="p">=</span> <span class="s">{2026}</span>
<span class="p">}</span>
</code></pre></div></div>

<p><em>This work is supported in part by ONR award N00014-23-1-2383 and the U.S. DARPA ECOLE program (HR00112390060). GPUs were provided by NCSA Delta through ACCESS allocation CIS230397.</em></p>]]></content><author><name>https://zzhu.vision/</name></author><category term="multimodal" /><category term="continual-learning" /><category term="fine-tuning" /><category term="eccv" /><summary type="html"><![CDATA[Fine-tuning an LMM on a new skill usually erodes what it already knew. We find that much of this 'forgetting' is a shift in the output token distribution, and that tuning only the self-attention projections, or only the MLP gate and up projections, learns almost as well as full tuning while barely forgetting.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://pages.vision.ischool.illinois.edu/blog/assets/teach-lmm/fig1_teaser.png" /><media:content medium="image" url="https://pages.vision.ischool.illinois.edu/blog/assets/teach-lmm/fig1_teaser.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">FreeOrbit4D: Replaying a Single Video from Any Camera, Without Training</title><link href="https://pages.vision.ischool.illinois.edu/blog/freeorbit4d/" rel="alternate" type="text/html" title="FreeOrbit4D: Replaying a Single Video from Any Camera, Without Training" /><published>2026-07-19T00:00:00+00:00</published><updated>2026-07-19T00:00:00+00:00</updated><id>https://pages.vision.ischool.illinois.edu/blog/freeorbit4d</id><content type="html" xml:base="https://pages.vision.ischool.illinois.edu/blog/freeorbit4d/"><![CDATA[<p>Every phone video is shot from exactly one viewpoint. Yet when we watch a clip of someone being interviewed, we can effortlessly imagine the same moment from the side, from behind, or from above: we build a rough 3D world in our heads and replay its motion from wherever we like. <strong>Camera redirection</strong> asks a machine to do the same, synthesizing a new video of a dynamic scene along a user-specified camera trajectory from a single monocular recording. Today, effects like bullet-time replays still require multi-camera rigs; doing it from one casual video would put that capability in everyone’s pocket.</p>

<p>We are excited to share <strong>FreeOrbit4D</strong>, our SIGGRAPH 2026 paper on this problem. FreeOrbit4D is <em>training-free</em>: it composes off-the-shelf models with no fine-tuning, and it handles the regime where existing methods break down, namely large-angle redirection, where the target camera swings 120° or even 180° away from where the video was shot. The key idea is to first recover a <strong>foreground-complete 4D proxy</strong>, an explicit point-cloud representation of the scene in which the moving object is reconstructed on <em>all</em> sides, including the sides the camera never saw, and then use that proxy as a geometric scaffold for a video diffusion model.</p>

<div class="grid grid-2">
  <figure><video src="/blog/assets/freeorbit4d/camel_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Input video</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/camel_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Redirected by FreeOrbit4D</figcaption></figure>
</div>

<h2 id="why-large-angle-redirection-is-hard">Why large-angle redirection is hard</h2>

<p>A monocular video is a very thin slice of a 4D world. It observes each surface from a single direction, and anything facing away from the camera is simply never recorded. Small viewpoint changes can be handled by reconstructing the visible surfaces and re-rendering them, as NeRF- and Gaussian-splatting-style methods do. But once the target camera moves far from the source trajectory, the unseen surfaces dominate the frame, and there is nothing in the input to anchor them.</p>

<p>Recent work leans on video diffusion models to fill this gap, and largely follows one of two recipes. <strong>Implicit control</strong> methods encode the target camera as a learned embedding or a text prompt and inject it into the generator. This gives only soft controllability: text cannot describe a precise orbit, learned conditions frequently ignore the requested path (a “turn to the back” instruction often does not turn to the back), and training these models needs paired multi-view video data that is expensive and scarce. <strong>Explicit warping</strong> methods instead estimate depth, warp the observed pixels to the target view, and let a generator inpaint the holes. Camera control is now precise, but the warped geometry only ever contains what was visible, so the object arrives at the new viewpoint with half of itself missing, and the generator has to hallucinate the rest frame by frame, with no 3D support to keep it consistent over time.</p>

<p>Neither recipe delivers both precise camera control and complete visibility of unseen regions, and both are needed for faithful large-angle redirection.</p>

<h2 id="a-foreground-complete-4d-proxy">A foreground-complete 4D proxy</h2>

<p>Our observation is that reconstructing a dynamic scene from monocular video and completing the occluded geometry of an object are fundamentally different tasks. The first needs temporally consistent, scene-level reasoning; the second needs multi-view understanding of object shape. So we handle them in separate coordinate spaces, with models that are good at each, and then unify the results.</p>

<figure class="full-width">
  <img src="/blog/assets/freeorbit4d/fig3_overview.jpg" alt="Overview of FreeOrbit4D: global scene reconstruction and canonical object completion are fused into a foreground-complete 4D proxy, rendered into depth maps that condition a video diffusion model." />
  <figcaption>Overview. Global scene reconstruction recovers the background and a partial foreground in world space; canonical object completion reconstructs the full foreground from synthesized multi-view images. After alignment, depth rendered from the unified 4D proxy along the target trajectory conditions a video diffusion model, together with a reference frame and a text prompt.</figcaption>
</figure>

<p><strong>Decoupled 4D reconstruction.</strong> In <em>global scene space</em>, a dynamic-aware feed-forward network (PAGE-4D, a temporally-aware extension of VGGT) lifts every frame of the source video into a point map registered in a single world coordinate system. SAM 2 masks, initiated by one click on the first frame or automatically from a detector, split these points into a static background and a per-frame foreground. This foreground is geometrically <em>incomplete</em>: it only contains surfaces the source camera saw. In parallel, in <em>canonical object space</em>, we cut the foreground out of each frame and feed the sequence to an object-centric multi-view video diffusion model (SV4D 2.0), which synthesizes four temporally synchronized novel views at 90° azimuth intervals. Running VGGT on the source view plus the four synthesized views gives a <em>complete</em> foreground point cloud for every frame, but one that knows nothing about where the object sits in the scene.</p>

<p><strong>Correspondence-aware alignment.</strong> The two foregrounds have to be merged, and the trick that makes this easy is that both were derived from the same source image. A pixel in frame <em>t</em> maps to a 3D point in the partial cloud <em>and</em> to a 3D point in the complete cloud, so we get dense, pixel-synchronized 3D–3D correspondences for free. Monocular lifting cannot pin down absolute depth scale, so we do not fit the two clouds point-by-point; instead we use the partial cloud only to determine where the object is and how big it is, estimating a per-frame scale and translation while keeping the complete cloud’s geometry intact. Because monocular depth also jitters from frame to frame, we smooth the aligned object’s centroid trajectory with a bidirectional Kalman filter under a constant-velocity model. The output is the foreground-complete 4D proxy: the static background plus a complete, correctly placed, temporally coherent foreground at every time step.</p>

<p><strong>Geometry-conditioned video synthesis.</strong> Finally, we render the proxy along the target camera trajectory into per-frame depth maps and hand them to a depth-conditioned video diffusion model (Wan2.2-VACE), along with the first source frame as an appearance reference and a text prompt. Depth is not full geometry, but it compactly encodes the 3D layout and visibility from the target viewpoint, which is exactly what the generator needs to follow the prescribed camera precisely and stay consistent across frames. The whole pipeline runs on a single NVIDIA A40 and processes a 45-frame clip at 832×480 in roughly 50 minutes.</p>

<h2 id="results">Results</h2>

<p>The interactive 4D point clouds for several scenes, including the robot demo below, can be rotated and scrubbed through time on the <a href="https://freeorbit4d.vision.ischool.illinois.edu/">project page</a>.</p>

<div class="grid grid-2">
  <figure><video src="/blog/assets/freeorbit4d/unitree_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Input video</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/unitree_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Redirected by FreeOrbit4D</figcaption></figure>
</div>

<p>Because the proxy is a real 3D representation, the same reconstruction can be replayed along any number of trajectories. Below, each input video is redirected to two different target cameras.</p>

<div class="grid grid-3">
  <figure><video src="/blog/assets/freeorbit4d/interview_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Input</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/interview_output_1.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Trajectory #1</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/interview_output_2.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Trajectory #2</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/robot_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Input</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/robot_output_1.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Trajectory #1</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/robot_output_2.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Trajectory #2</figcaption></figure>
</div>

<p>The scenes span articulated human motion, humanoid robots, animals and vehicles, with real-world footage from DAVIS and online videos as well as synthetic clips generated by Veo and Sora. For evaluation we deliberately choose extreme trajectories, with yaw or pitch rotations of 120° or 180° from the initial viewpoint.</p>

<div class="grid grid-2">
  <figure><video src="/blog/assets/freeorbit4d/breakdance_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Input video</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/breakdance_output.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Redirected by FreeOrbit4D</figcaption></figure>
</div>

<p>We compare against five recent camera-controlled video-to-video methods: ReCamMaster, TrajectoryCrafter, EX-4D, GEN3C and CogNVS. The “Swing” sequence below is a hard case, with fast foreground motion and thin swing ropes, viewed from a camera that orbits 120° around the scene. ReCamMaster and EX-4D lose the structure of the human body, producing blurred limbs and ghosting; TrajectoryCrafter, GEN3C and CogNVS drift geometrically and semantically as the camera rotates. FreeOrbit4D keeps sharp detail and stable geometry throughout, because the completed foreground tells the generator what should be there.</p>

<div class="grid grid-4">
  <figure><video src="/blog/assets/freeorbit4d/swing_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Input</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/swing_recammaster.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>ReCamMaster</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/swing_trajectorycrafter.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>TrajectoryCrafter</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/swing_ex4d.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>EX-4D</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/swing_gen3c.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>GEN3C</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/swing_cognvs.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>CogNVS</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/swing_ours.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>FreeOrbit4D (ours)</figcaption></figure>
</div>

<p>Quantitatively, FreeOrbit4D ranks first on five of the six VBench dimensions and achieves the best DINO-SIM, CLIP-SIM and FID-V among all methods. But we found that automatic metrics do not really measure whether a method <em>followed the camera</em>: ReCamMaster and TrajectoryCrafter score reasonably while visibly deviating from the requested path. So we also ran a user study with 20 participants across 10 sequences, rating overall preference, camera-motion accuracy and temporal stability on a 1–5 scale. There the gap is much larger.</p>

<table>
  <thead>
    <tr>
      <th>Method</th>
      <th>DINO-SIM ↑</th>
      <th>CLIP-SIM ↑</th>
      <th>FID-V ↓</th>
      <th>Overall ↑</th>
      <th>Motion ↑</th>
      <th>Stability ↑</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>ReCamMaster</td>
      <td>0.37</td>
      <td>0.75</td>
      <td>2.6</td>
      <td>2.0</td>
      <td>2.5</td>
      <td>2.0</td>
    </tr>
    <tr>
      <td>TrajectoryCrafter</td>
      <td>0.47</td>
      <td>0.79</td>
      <td>2.0</td>
      <td>2.8</td>
      <td>3.2</td>
      <td>2.9</td>
    </tr>
    <tr>
      <td>EX-4D</td>
      <td>0.28</td>
      <td>0.69</td>
      <td>3.2</td>
      <td>2.0</td>
      <td>2.5</td>
      <td>2.0</td>
    </tr>
    <tr>
      <td>GEN3C</td>
      <td>0.43</td>
      <td>0.75</td>
      <td>2.3</td>
      <td>2.4</td>
      <td>3.5</td>
      <td>2.3</td>
    </tr>
    <tr>
      <td>CogNVS</td>
      <td>0.34</td>
      <td>0.78</td>
      <td>3.0</td>
      <td>2.0</td>
      <td>3.2</td>
      <td>1.9</td>
    </tr>
    <tr>
      <td><strong>FreeOrbit4D</strong></td>
      <td><strong>0.65</strong></td>
      <td><strong>0.84</strong></td>
      <td><strong>1.7</strong></td>
      <td><strong>4.6</strong></td>
      <td><strong>4.5</strong></td>
      <td><strong>4.5</strong></td>
    </tr>
  </tbody>
</table>

<p class="table-note">FID-V is reported ×10². The three right-hand columns are user-study ratings (1–5). See the paper for the full VBench and FVD-V numbers.</p>

<p>Ablations confirm that both halves of the design matter. Naively feeding the synthesized multi-view images together with the source video into a single feed-forward reconstructor collapses correspondences across time and produces ghosting; removing multi-view generation degrades every metric; and removing the Kalman smoothing noticeably hurts coherence.</p>

<h2 id="beyond-redirection">Beyond redirection</h2>

<p>Because the proxy is an explicit, editable point cloud rather than a latent inside a network, it turns out to be useful for more than moving the camera.</p>

<p><strong>Appearance propagation.</strong> Edit a single reference frame with an image editor (we use Qwen-Image-Edit), say a zebra print on the car or an anime restyle, and the proxy’s depth scaffolds carry that edit consistently across every novel viewpoint of the redirected video.</p>

<div class="grid grid-2">
  <figure><video src="/blog/assets/freeorbit4d/app_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Original</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/app_color.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Appearance editing: color / pattern</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/app_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Original</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/app_style.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Appearance editing: style</figcaption></figure>
</div>

<p><strong>4D geometry manipulation.</strong> The point cloud can be edited directly in space-time: scale the foreground, or composite objects reconstructed from different videos into one shared scene, and then synthesize a plausible redirected video from the modified geometry. We think this points toward using the recovered proxy for 4D asset creation and synthetic data generation, where dense, consistent geometry and annotations come along for free.</p>

<div class="grid grid-2">
  <figure><video src="/blog/assets/freeorbit4d/app_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Original</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/app_shape.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Geometry editing: scaling</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/app_input.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Original</figcaption></figure>
  <figure><video src="/blog/assets/freeorbit4d/app_camel.mp4" autoplay="" muted="" loop="" playsinline=""></video><figcaption>Geometry editing: composition</figcaption></figure>
</div>

<h2 id="limitations">Limitations</h2>

<p>FreeOrbit4D assumes a single dominant foreground object and a mostly static background. It can handle several objects as long as each is observed well enough over time, but heavy mutual occlusion or dynamic backgrounds remain difficult for current object-centric models. As a modular system built from off-the-shelf parts, errors upstream in segmentation or multi-view synthesis propagate downstream, although that same modularity means each component can be swapped for a stronger one as it appears. And the multi-stage pipeline prioritizes quality over speed; at tens of minutes per clip, it is not yet interactive.</p>

<h2 id="read-more">Read more</h2>

<p>The paper appears in SIGGRAPH 2026 Conference Papers. The <a href="https://freeorbit4d.vision.ischool.illinois.edu/">project page</a> has interactive 4D reconstructions you can rotate and scrub, plus full comparisons on all sequences, and the <a href="https://github.com/VVeiCao/FreeOrbit4D">code</a> is on GitHub.</p>

<div class="language-bibtex highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">@inproceedings</span><span class="p">{</span><span class="nl">cao2026freeorbit4d</span><span class="p">,</span>
  <span class="na">title</span>     <span class="p">=</span> <span class="s">{{FreeOrbit4D}: Training-Free Arbitrary Camera Redirection for Monocular Videos via Foreground-Complete {4D} Reconstruction}</span><span class="p">,</span>
  <span class="na">author</span>    <span class="p">=</span> <span class="s">{Wei Cao and Hao Zhang and Fengrui Tian and Yulun Wu and Yingying Li and Shenlong Wang and Ning Yu and Yaoyao Liu}</span><span class="p">,</span>
  <span class="na">booktitle</span> <span class="p">=</span> <span class="s">{SIGGRAPH Conference Papers}</span><span class="p">,</span>
  <span class="na">year</span>      <span class="p">=</span> <span class="s">{2026}</span><span class="p">,</span>
  <span class="na">url</span>       <span class="p">=</span> <span class="s">{https://doi.org/10.1145/3799902.3811122}</span>
<span class="p">}</span>
</code></pre></div></div>

<p><em>This research is supported by the National Artificial Intelligence Research Resource (NAIRR) Pilot under award NAIRR250199, with computational resources from Delta and DeltaAI at NCSA through ACCESS allocations.</em></p>]]></content><author><name>https://vveicao.github.io/</name></author><category term="video-generation" /><category term="4d-reconstruction" /><category term="siggraph" /><summary type="html"><![CDATA[A training-free method that turns one monocular video into a foreground-complete 4D proxy, so a video diffusion model can re-shoot the scene from large-angle, even bullet-time, trajectories.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://pages.vision.ischool.illinois.edu/blog/assets/freeorbit4d/fig1_teaser.jpg" /><media:content medium="image" url="https://pages.vision.ischool.illinois.edu/blog/assets/freeorbit4d/fig1_teaser.jpg" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>