Joint work with Stability AI.

A film director stages a shot by deciding where the camera goes and where every actor goes, and leaves how an actor walks or how the light falls to the cast and crew. Video generation models are getting good at the second half: given a text prompt and an image, they produce footage with convincing motion, shading and detail. What they still lack is the first half. If you want the camel to turn around 180° while the camera arcs to the left, there is no reliable way to say so.

We are excited to share 4Director, a collaboration between our group and Stability AI. 4Director is a video world model that takes exactly the director’s inputs: a single image, a 3D trajectory for the camera, and a rigid 3D trajectory for each object, and generates a video that follows them. The key idea is to make the control signal an explicit 4D scene: every object is reconstructed once as a complete canonical mesh and then moved by one rigid transformation per frame, sharing a coordinate frame with the camera and a background point cloud. We render that scene as a depth video and train a Motion Adapter that turns the geometric scaffold into video, supplying the appearance, illumination and non-rigid dynamics the scaffold leaves out. Because the mesh has a surface on every side, geometry that a turn or a camera move reveals is rendered rather than hallucinated anew in each frame.

Input image of a hiker on a rocky mountain trail
Input image
Rigid 3D scene rendered to depth (the control)
4Director output

Why existing controls fall short

Controls for object motion in video generation fall into two families, and each misses something a director needs.

Image-plane cues, such as dragged paths, bounding boxes, masks and point tracks, state where an object should appear in each frame. But a single projection is consistent with many 3D motions: a box does not determine the object’s orientation, a shrinking box cannot distinguish an object moving away from one getting smaller, and a 2D path conflates camera parallax with object motion. 3D-aware controls fix position with rigid trajectories, 6-DoF poses or 3D boxes, or lift a proxy of the object from the input image: spheres on tracked parts (Perception-as-Control), 3D point tracks over a point cloud (SymphoMotion), or one 3D Gaussian blob per object (VerseCrafter). These remove the depth ambiguity, but their geometry is incomplete. A trajectory or box carries no surface at all, and a lifted proxy only captures the visible shell, so there is nothing to render when a turn brings the far side of the object into view. Point-based proxies are also driven by per-point trajectories, which are tedious to author.

A camel is asked to turn 180° in place. Under 2D box control it barely turns; under 3D point tracks and a 3D Gaussian blob it turns partway or deforms; under rigid 3D geometry control the depth video shows the full silhouette at every step and the generated camel completes the turn.
The same request, a 180° turn in place, under four control signals. A 2D bounding box, 3D point trajectories and a 3D Gaussian blob all lead the generator to keep roughly the original heading or distort the camel. Our rigid 3D geometry renders the complete mesh at every orientation, and the generated camel turns all the way round.

The table below summarizes what each representation can express. Only the rigid 3D geometry fixes camera trajectory, 3D position, orientation and complete geometry at once, and it is the only one that can insert a brand-new object into the scene.

Method Camera trajectory 3D object position Object orientation Complete geometry New object insertion
MotionCtrl ✓ partial ✗ ✗ –
Perception-as-Control ✓ ✓ ✗ ✗ –
SymphoMotion ✓ ✓ partial ✗ –
VerseCrafter ✓ ✓ partial ✗ –
4Director (ours) ✓ ✓ ✓ ✓ ✓

Rigid 3D geometry as the control

Overview of 4Director. Left: the input image is lifted into complete canonical meshes for each object and a background point cloud; user-defined object and camera trajectories move them; the scene is rendered to a depth video. Right: the depth video, input image and prompt are encoded and injected by the Motion Adapter's context blocks into a frozen Wan-DiT, which generates the video.
Overview. Left: each object is reconstructed once as a complete canonical mesh and moved by one rigid transformation per frame; meshes, background point cloud and camera share one coordinate frame, and the scene is rendered to a depth video. Right: the Motion Adapter, a trainable branch of context blocks, injects this rendering into a frozen pretrained video generator, which supplies appearance, illumination and non-rigid dynamics.

Lifting the image. Given one image, MoGe-2 estimates depth and camera intrinsics, and SAM 2 turns a few clicks on each object into a mask. The camera of the input image defines the reference frame. The background, everything outside the masks, is back-projected into a static point cloud. Each marked object is cut out and reconstructed by Pixal3D into a complete, textured canonical mesh, including the sides the photo does not show, and aligned to its mask and depth. Background, camera and objects now share one coordinate frame.

Directing the scene. In a 3D viewer, the user sets keyframes for the camera trajectory and for each object’s rigid trajectory, a rotation and translation per frame. Because a turn is literally a rotation and motion in depth is literally a displacement, there is no ambiguity for the generator to resolve. A new object can be inserted by reconstructing a mesh from a separate reference image, placing it in the scene and giving it its own trajectory; its textured mesh is rendered into the input image, which becomes the first frame. Altogether the rigid 3D scene is the background cloud, the set of meshes, their per-frame transformations, the intrinsics and the camera trajectory, of which only the last two vary in time.

Rendering the control. For each frame, the background and the placed meshes are projected into a depth map in which every pixel takes the depth of the nearest surface, so occlusion ordering is decided before generation. The resulting depth video is the entire control signal. It carries the viewpoint, the rigid motion of every object and their mutual occlusion, and deliberately nothing else: no appearance, no illumination, no limb motion, no background beyond what the image shows. Supplying all of that is the Motion Adapter’s job.

The Motion Adapter

We build on Wan2.1-VACE-14B, keeping its VAE, umT5 text encoder and DiT frozen as a generic video prior. The Motion Adapter follows VACE’s context-branch design: the VAE encodes the depth video and the input image into a context stream, which eight context blocks, one for every fifth DiT block, propagate with cross-attention to the prompt and inject into the DiT as linearly projected hints. The adapter has 3.0B parameters and is initialized from the released VACE branch.

The crucial point is what the adapter is trained to do. Every training pair comes from a real clip: the control is the depth video rendered from the clip’s recovered rigid 3D scene, and the target is the clip itself, with its first frame as the input image and its caption as the prompt. The control contains only the rigid part of each object’s motion. If a surface point of an object is observed at \(X_i^t = R_o^t Y_i + t_o^t + r_i^t\), the first two terms, the rigid motion of the mesh, are in the control, and the residual \(r_i^t\), a swinging limb or a turning head, is not. To reproduce the clip, the adapter must follow the rigid geometry exactly and let the generator fill in the rest. It is trained with the standard flow-matching objective, for three epochs on 24 GPUs at 832×480 and 81 frames. Because the adapter only ever sees the scene as a depth video, a hand-authored scene is treated exactly like a recovered one, so new trajectories and inserted objects need no extra training.

RealCOD-Rigid: 20,774 clips with rigid 3D scenes

No dataset pairs videos with complete object meshes and per-frame rigid transformations, and nobody can annotate that by hand at scale. So we built an automatic pipeline and ran it on RealCOD-25K, a collection of 81-frame stock and web clips that come with a text prompt and SAM3 masks for one or two objects. Four stages recover the rigid 3D scene of each clip: MegaSaM (with MoGe-2 as depth prior and UniDepthV2 as metric branch) estimates per-frame depth, intrinsics and the camera trajectory; Pixal3D reconstructs each object’s canonical mesh from the first frame and a robust similarity fit places it in the scene; TAPIP3D tracks 3D points on the object through the clip, and a robust alternating fit finds the object’s stable core, the tracks that move together rigidly, so that a swinging arm does not drag the whole-body pose with it; finally the placed meshes and background are rendered to the depth video. Clips in which any object fails a geometric or silhouette consistency gate are dropped, which leaves 20,774 training clips. Genuine occlusion is kept, so an object is allowed to disappear for some frames.

Measuring object control: Identity-Gated IoU

Camera control has a standard metric, the rotation and translation error between the requested trajectory and the one re-estimated from the generated video. Object control did not. Mask IoU against the source clip’s masks tells you whether something is where the trajectory says, but it happily rewards an object that is in the right place and has turned into a different object. We introduce Identity-Gated IoU (IG-IoU): on 16 frames spread uniformly over the clip, a vision–language judge (Qwen3-VL) compares the generated object crop to the reference crop and decides whether it is still intact and the same object. A frame that passes contributes its mask IoU; a frame that fails contributes zero but stays in the denominator. The fraction of frames that pass is reported separately as the recognition rate. Because the reference mask is the object’s silhouette at its true pose, an orientation error that changes the silhouette is penalized too.

Results

We compare against four public methods for joint camera and object control, MotionCtrl, Perception-as-Control (PaC), SymphoMotion and VerseCrafter, on 100 held-out clips of RealCOD-25K. To be fair, every method is given the same motion to follow: the camera and object trajectories recovered from each clip are converted into whatever control each baseline expects, from 2D trajectories to 3D Gaussians.

Method FID ↓ FVD ↓ CLIP-SIM ↑ RotErr (°) ↓ TransErr ↓ Recognition ↑ IG-IoU ↑
MotionCtrl 115.6 1653.3 27.5 4.74 0.231 32.6 18.8
Perception-as-Control 55.7 1454.6 30.2 7.00 0.228 80.5 40.8
VerseCrafter 55.0 484.7 31.7 6.16 0.174 94.8 54.8
SymphoMotion 45.8 405.4 30.9 3.89 0.132 90.2 52.9
4Director 44.1 370.4 31.8 3.65 0.122 94.9 60.4

Joint camera and object control on 100 held-out RealCOD-25K clips. Camera errors follow the CameraCtrl protocol; recognition rate and IG-IoU are in percent. See the paper for the eight VBench-I2V dimensions, on which 4Director also leads.

4Director is best on every metric, and the margin is widest where it should be: IG-IoU rises from 54.8 for VerseCrafter to 60.4 at the same recognition rate (94.9 versus 94.8). In other words, 4Director does not keep the object recognizable more often than VerseCrafter; it places it more accurately, which is what a complete mesh with a fixed orientation buys you over a Gaussian blob. MotionCtrl, driven by 2D trajectories, keeps the object recognizable in only a third of frames.

Automatic metrics compress a lot. In a user study, 20 participants rated the five methods on nine cases for visual quality, control accuracy and consistency on a 1–5 scale. 4Director scored 4.62, 4.82 and 4.69; the next best method on each criterion scored 3.69, 2.39 and 3.88. The gap on control accuracy, 4.82 against 2.39, is the whole story of the paper in one number.

Object rotation. The hardest case for the baselines is a turn. Below, a black swan and a car are each asked to rotate 180° in place with the camera held still. The baselines keep the original heading, turn partway, or lose the subject; 4Director completes the turn and shows the side that comes into view.

Input image of a black swan on a pond
Input
MotionCtrl
Perception-as-Control
SymphoMotion
VerseCrafter
4Director (ours)
Input image of a car at a roundabout
Input
MotionCtrl
Perception-as-Control
SymphoMotion
VerseCrafter
4Director (ours)

Leaving and re-entering the view. A representation with a trajectory defined at every frame handles a case that trips up proxies lifted from pixels. Here the camel is directed to walk out of the frame and come back while the camera stays fixed. In 4Director the mesh is simply not rendered while it is outside the view and rendered again from the same mesh when it returns, so the model shows an empty yard, fills in the ground the camel stood on, and brings the same camel back. MotionCtrl, SymphoMotion and VerseCrafter keep the camel in frame throughout; Perception-as-Control lets it leave but cannot preserve its shape when it returns.

Input image of a camel in an enclosure
Input
MotionCtrl
Perception-as-Control
SymphoMotion
VerseCrafter
4Director (ours)

What does the control need to encode? An ablation swaps the rigid 3D geometry for three simpler controls trained with the same recipe. A 2D box encodes neither depth nor orientation and scores lowest (IG-IoU 48.5). A 3D box keeps depth and orientation but has no surface (50.3). A 3D mesh that translates but does not rotate keeps the complete surface but loses orientation (53.0); in that setting the motorcycle in the test clip ends up facing the camera instead of turning sideways. Only the full rigid 3D geometry, with depth, rotation and a complete surface, reaches 60.4, and the four controls rank in the same order on FID and FVD. Notably, recognition rate stays between 91.9 and 95.0 for all four, so what the simpler controls lose is placement, not identity.

Composing shots

Since every object is a mesh in a shared 3D space, directing becomes compositional. The hiker sequence below starts from one photo. We first move the hiker along a path; then we insert a camel reconstructed from a separate reference image and give it its own trajectory; then we add a camera move on top. Each row shows the depth control the adapter receives and the video it generates. The adapter never saw an insertion pair during training; it only sees depth, so a composed scene looks to it like any other.

Input image of a hiker
Input image
Reference image of a camel used for object insertion
Reference image for the inserted camel
Hiker + inserted camel: depth control
Hiker + inserted camel: output
+ camera trajectory A: depth control
+ camera trajectory A: output
+ camera trajectory B: depth control
+ camera trajectory B: output

The same recipe works with less friendly co-stars. Below, a bear is first turned in place, then joined by a T-Rex inserted from a reference image, then filmed with a moving camera.

Input image of a brown bear among rocks
Input image
Reference image of a T-Rex model used for object insertion
Reference image for the inserted T-Rex
Bear rotation: depth control
Bear rotation: output
+ inserted T-Rex: depth control
+ inserted T-Rex: output
+ camera trajectory: depth control
+ camera trajectory: output

Beyond the controlled comparisons, the model carries over to subjects far from the training distribution, including game characters and fast vehicles, with the generator supplying the running, skiing and jumping that the rigid control leaves unspecified.

Skier
Sports car
Rooftop jump (game character)

Limitations

The representation controls motion at the level of a rigid body, which is both its strength and its boundary. An articulated or deforming object moves as one whole, so finer motion such as running, jumping or the swing of a limb cannot be prescribed; it is left to the generator. Usually the generator does the right thing, but in a failure case from the paper a breakdancer in a handstand is translated sideways as a rigid body, and the generated dancer simply holds the handstand throughout, faithfully following the rigid silhouette instead of dancing. Extending the rigid 3D scene with articulated parts, so that such motion can be prescribed too, is the natural next step. A lighter variant of the adapter, trained with rank-16 LoRA (15.3M parameters), still beats all four baselines on FID, FVD and IG-IoU, though the full 3.0B adapter leads it on every metric.

Read more

The paper is on arXiv (2610.02160). The project page has interactive point clouds you can rotate for every case, a qualitative comparison carousel against all four baselines, and the full set of authored trajectories and insertions. The code is on GitHub. 4Director builds on the same intuition as our earlier FreeOrbit4D: give the video generator complete geometry for what it cannot see, and it will stop making it up.

@article{cao20264director,
  title   = {{4Director}: Controlling Video World Models with Rigid {3D} Geometry},
  author  = {Wei Cao and Hao Zhang and Vikram Voleti and Yuqun Wu and Mallikarjun B R and Shimon Vainer and Mark Boss and Yaoyao Liu},
  journal = {arXiv preprint arXiv:2610.02160},
  year    = {2026}
}