Avatar V: Scaling Video-Reference Avatar Generation

Avatar V is built on a Diffusion Transformer with flow matching that conditions directly on the full token sequence of a user’s reference video—no bottleneck embeddings. Sparse Reference Attention keeps cost almost linear with reference length. A five-stage training curriculum progresses from general video pre-training through identity-preserving fine-tuning, distillation, and RLHF alignment.
Avatar V is the latest version of HeyGen’s avatar video generation system. It produces high-resolution avatar videos of arbitrary length from a single reference video and a driving audio signal. This post covers the model architecture and the five-stage training pipeline we built to get there.
We frame avatar generation as conditional video synthesis. Given a reference clip and an audio track, the model generates a talking-head video that preserves the speaker’s identity while following the rhythm and content of the audio. The core idea is video-reference conditioning: rather than compressing identity into low-dimensional embeddings or fixed-size feature vectors, the model conditions on the full token sequence of the user’s reference video at every transformer layer. This scales naturally with reference length—more context yields richer identity information—and requires no identity-specific fine-tuning at inference time.
What distinguishes Avatar V from existing systems is its ability to model both static and dynamic aspects of personal identity. Static features include fine-grained, time-invariant characteristics: dental structure, skin texture, facial geometry, hair, and accessories. Dynamic features encompass behavioral patterns: talking rhythm, habitual micro-expressions, and gestural tendencies during speech. Short references provide basic appearance information; longer references let the model observe and internalize the individual’s talking cadence and expression dynamics. The result is that generated videos are not merely facially similar to the target but are behaviorally recognizable.
Figure 1: Avatar V Architecture. Multi-modal inputs are patchified into a unified token sequence and processed through L transformer blocks with Sparse Reference Self-Attention, cross-attention modules, and motion injection.
Standard approaches either compress references into low-dimensional bottlenecks that lose fine-grained identity details, or concatenate all reference tokens with generation tokens at prohibitive quadratic cost. Sparse Reference Attention addresses this through a structured sparsity pattern that preserves full access to identity information while eliminating redundant computation among tokens that do not require mutual interaction. The resulting complexity scales almost linearly with reference length, enabling the model to condition on minutes-long reference footage.
Talking style—the characteristic temporal pattern of facial movements, mouth shapes, and head gestures during speech—is captured through a dedicated motion representation that serves as both a learning objective and a conditioning signal. Through joint optimization of these two roles, the model develops a unified understanding of the target speaker’s motion dynamics, producing generated videos that are behaviorally consistent with the reference speaker even for unseen speech content.
Before video generation begins, the Image Engine constructs a high-fidelity scene image of the speaker. Rather than relying on a single reference frame, the pipeline automatically selects a diverse set of frames spanning multiple viewpoints and expressions from the user’s input video. This multi-view sampling ensures robust identity representation, reproducing subtle cues like smile asymmetry and nasolabial fold characteristics across novel scenes.
Voice cloning runs on a separate Audio Engine built around an LLM backbone. Given as little as 10 seconds of audio, the engine generates natural speech with matched timbre, prosody, and accent through discrete audio token prediction, supporting multilingual output and emotion control. The audio output feeds directly into the motion encoder, closing the loop between voice and movement.
The base DiT generates video at low resolution, then a dedicated super-resolution refiner upscales to the final output. The refiner shares the same DiT backbone and inherits the full identity modeling apparatus—video reference conditioning, audio features, and motion representations—so facial identity is preserved through upsampling. Sparse temporal attention restricts each frame’s receptive field to a local neighborhood, and multi-stage distillation enables high-quality upsampling in very few denoising steps.
We train Avatar V in five progressive stages, each building on the last. Early stages use large, loosely-curated data to learn general video priors. Later stages use small, high-quality data to specialize the model for avatar generation and align it with human preferences.
Figure 2: Avatar V Training Pipeline. The five-stage curriculum progressively builds capabilities from general video generation through audio-driven synthesis and identity-preserving personality embedding to distillation and human feedback alignment.
The backbone starts as a general text-to-video model trained on internet-scale video data with progressive resolution and duration scaling. This stage jointly trains on text-to-video and image-to-video tasks, teaching the model coherent motion, camera handling, and temporal consistency before any avatar-specific data is introduced.
Starting from the T2V checkpoint, the model is adapted to accept a conditioning image and a driving audio track. This stage introduces audio cross-attention modules trained jointly with the visual backbone on a broad corpus of talking-head video spanning diverse speakers, languages, and speaking styles.
Supervised fine-tuning on curated same-identity-different-scene pairs. The model receives reference videos through Sparse Reference Attention and learns to extract identity-i