Stormin' The Castle

voicevideoavatarwebrtc

Giving a Face to Your AI Assistant

by John Robinson @johnrobinsn

Why

Nuance Labs just unveiled their Nuplex model last week with an amazing online demo. In the demo, you have a two-way video chat with an AI agent, Emma. But not just with realtime speech, the model also generates a convincingly realistic video avatar for the agent so you can talk with it face-to-face. My experience took me a bit off guard, maybe because my expectations going in were a bit low, but I think that under the right circumstances this could pass for a real person. I'll have to try it again to see if my impressions hold up. Give it a try, I think you'll be pretty amazed as well.

Given this I was intriqued to see what I could cobble together to give my voice pipeline projects a realtime video avatar. I already have a voice pipeline, macvoice that I built for Apple Silicon that lets you have a two-way voice chat with any LLM or coding agent. You can read more about that project if you follow that link. It works pretty well, but it doesn't have a face.

I did this project using Linux+CUDA since I knew that I would have to throw a lot more compute to get a realtime video model working along with the rest of the voice pipleline. For my local setup, I'm using two NVIDIA RTX 6000 blackwell cards which gives me a fair bit of VRAM and compute. I can probably reduce the requirements to a single card but I was mostly focused on just getting things working given what I had available. The extra compute headroom definitely came in handy given the realtime demands of video chat and lip sync.

After doing a survey of the latest open weight video models, I selected AvatarForcing as the model. Here is their project page and paper, arXiv 2601.00664.

The AvatarForcing model and the cooresponding inference code is pretty well suited for this project. For this scenario, we need "endless video generation" which requires a model that can generate video autoregressively. Generating the video frames that come next given only context about the recent past. Also to make the generation efficient, we need a kv cache that supports rolling where as we generate new blocks, old blocks get evicted so the cache doesn't grow unbounded. Given that these requirements were well supported, I was able to focus largely on the inference pipeline and overall streaming infrastructure. My project is largely a webrtc video streaming application that coordinates the overall pacing of video generation along with the voice pipeline so that we can meet its realtime constraints such as preserving lipsync and quality over time.

The model starts its video generation from a single reference image and takes in realtime multimodal inputs from the user (video and audio) as well as realtime audio for the avatar. Given thse inputs, the model will generate video frames for the avatar's face that includes lip movement, nods, laughter etc driven by the inputs. The model is designed to generate video one block at a time. Each block consists of ten frames of video that represent 400ms of wallclock video at 25 fps. If we can generate 400ms of video in less than 400ms we can meet our realtime targets. Measured across 186 blocks of video here are the video generation times that I've observed on my RTX 6000 Pro MaxQ (blackwell) card.

Stat EWMA RTF
Min 347 ms 0.87x (better than realtime)
Mean 437 ms 1.09x
Max 492 ms 1.23x

On average we're very close to realtime. The spikes are usually right after a kv cache reanchor. As we are generating video, the kv cache will slowly accumulate error that will degrade its quality. In order to deal with this we need to periodically reanchor the generation. The approach taken for this project is to periodically (~45 seconds) flush the kv cache and reseed it with the reference image again. Visually this transition is masked with a cross fade between the last frame of the old generation to the first couple of blocks of the new generation. This reanchoring puts additional load on the overall pipeline causing the video generation to push out beyond realtime.

One interesting problem that I ran into is that when the voice pipeline was feeding silence (zeroes) into the model it would mouth words silently (hallucinating speech). I guessed that since the model was trained on speech input (with background noise from an open mic) that the zeroed out audio frames were likely out of distribution for the model. So instead of zeroed out audio frames, I generate low-energy (-50 dbFS) pink noise during periods of silence which effectively eliminated the problem but kept other behaviors like eyes blinking and subtle head movements intact.

You can find the code for my project AvatarVoice in my github repo.

Current state

Given that I only spent a couple of days on this I was happy to get everything working pretty well together, decent lip sync behavior and realtime generation. No doubt that there are a number of bugs and rough edges to push on but it was a fun project to see come together. Nowhere near what Nuance labs has pulled off but this project makes me appreciate what they have accomplished all the more.

Futures

Definitely a lot more improvement that can be made here are just a few that I'll likely prioritize.


Get more content like this. Occasional, low-cadence, real-news-only.


Follow along at storminthecastle.com — or reply if you're building in this space.


Share on Twitter |  Discuss on Twitter

John Robinson © 2022-2026