AI Research | HeyGen

Video transcript: Motion design isn't dead. In fact, we think it's just starting to expand and AI will play a huge part in unlocking the possibility for more people to create video. You see, every new model comes with a report card, a table of scores against benchmarks. These tests measure how well a model solves hard, checkable problems like code and agentic work or math and science. But many people are pushing these models somewhere else that is much harder to quantify. Creative work, UI design, writing, video. There's no answer key for those, so people improve by trial and error. Our team set out to change that for video generated from code, a quantifiable way to measure how well a model makes video. Introducing the Code2Video Benchmark. I'm sure you're asking, well, who says a video is good? We started with people, two generated videos side by side, followed by simple questions. Does it hold your attention? Does it match the brief? Thousands of judgments, all made by humans. Then we trained a model on those picks. Show it two videos and a question, and it tells you which one people would choose and why. This model flickers. That one ignores the brief. And because the judge can score any two videos in milliseconds, models can train against it, not just be ranked by it. Over time, this benchmark will help models produce better videos from code. This is just the beginning. HyperFrames. Go make something.
HeyGen let users choose between leading voice engines. By using real-world feedback and blind preference testing, HeyGen built a voice model that surpassed competitors in quality while reducing costs by millions.
An open beach. A moving car. A busy factory floor. HeyGen Voice brings your voice into the picture, shaping the delivery and sound to fit the scene.
Each speaker’s stretch of dialogue in a Video Podcast is rendered independently. The challenge is making those clips behave as though they came from the same recording.
HeyGen is an AI video generation platform with industry-leading avatar models. Avatar IV, the diffusion stack behind our talking-head videos, runs on more than 18B parameters and is available via Web and API. Working with the Google Cloud AI-Ninja team, we brought Avatar IV to an eight-chip Trillium (v6e) host and made it 1.86× faster than our first working version. It now matches the performance of our 8×H100 production setup while being up to 25% more cost-efficient per minute of generated video.
How we moved from a custom job system to durable execution across a multi-cloud GPU fleet.
Context engineering is very different when your agent's goal is multi-modal (e.g. videos). Here's what we learned shipping one.
The expensive part of multi-behavior avatar control is a dataset where every behavior co-occurs, labeled, at studio quality — it barely exists. We skip it: train each behavior as a separate LoRA on cheap single-attribute data, compose them at inference, and close the gap with a short fine-tune on only a little real overlap.
We synchronize AI avatar speech with live browser actions using per-word timestamps emitted by the Avatar Realtime API. By anchoring clicks, highlights, and scrolling to the exact word being spoken, the system keeps actions aligned with natural speech instead of unreliable timers. The approach powers interactive AI agents that can browse the web, teach, review code, and respond live while maintaining accurate audiovisual synchronization.
We handed Claude Code a benchmark, a tamper-proof quality gate, and one rule file, then walked away. On a single H100 it made the Wan 2.2 video decoder 3× faster on its own. The same speedup cost us 105 hand-steered experiments the first time; this run took 18, blind.
Copyright © 2026 HeyGen