Limitless Video

What if AI video generation could be as fast as hitting "play"? #

Waiting minutes for AI video to render might become a thing of the past. By applying a groundbreaking training regime called Self Forcing to the popular WAN 2.1-T2V 1.3B parameter model researchers have pushed the boundary for what is possible with respect to real-time streaming video generation. While there is still a lot of room to improve quality and performance this advance shows the potential for AI video to become a genuinely interactive medium generating endless video from a prompt in near real-time on consumer grade hardware. These advances move us closer to having interactive video generation use cases— such as live streaming, gaming, and world simulation—where latency budgets are measured in milliseconds rather than minutes.

The Code #

As a companion to this article, I provide the full source for an application that can generate video from a text prompt at 12-15fps with a NVIDIA 5090 (or better). This application can be found at johnrobinsn/Limitless-Video.

But continue reading to find out more about how the WAN 2.1 1.3B parameter model was transformed into a low-latency, autoregressive model.

The Core Problem: Why Video Generation Used to Be Slow #

Traditional state-of-the-art video generation models, such as the original WAN 2.1, typically rely on Diffusion Transformers (DiT) that employ bidirectional attention. This design means that they denoise all frames for a fixed length video clip simultaneously, fundamentally limiting their applicability to real-time streaming applications where future information is unknown when generating the current frame.

In contrast, autoregressive (AR) models generate videos sequentially, aligning with the causal structure of temporal media and naturally reducing viewing latency. However, when AR models are trained with traditional methods like Teacher Forcing (TF) or Diffusion Forcing (DF), they often suffer from exposure bias. This occurs because the model is trained exclusively on perfect, "ground-truth" video frames but must rely on its own imperfect predictions during inference, causing small errors to accumulate and quality to degrade significantly over time. This degradation manifests as blurry visuals, oversaturated colors, and unnatural, repetitive motion patterns.

Self Forcing: Bridging the Gap for Real-Time Performance #

The solution is Self Forcing (SF), a novel training paradigm for autoregressive video diffusion models. Rather than introducing a new model, Self Forcing transforms the existing diffusion model (WAN 2.1-T2V-1.3B) into a model that can generate future video frames continuously.

1. Simulating Inference During Training #

The Problem: Models trained on perfect frames fail when using their own imperfect predictions.

The Solution: Self Forcing makes the model practice with its own mistakes:

• Generates video frame-by-frame during training
• Uses causal attention (only looking backward, not forward)
• Learns to recover from its own errors
• Eliminates the gap between training and real-world use

2. Efficiency Through Few-Step Diffusion #

To ensure that this sequential training remains computationally feasible, the Self Forcing implementation uses a few-step diffusion model to approximate the conditional distribution for each frame generation. The training employs a carefully designed gradient truncation strategy that limits backpropagation to only the final denoising step of each frame, addressing the challenge of excessive memory consumption that naive backpropagation through the entire AR diffusion process would cause. Surprisingly, Self Forcing achieves comparable per-iteration training time to parallel methods like Teacher Forcing and Diffusion Forcing, and achieves superior quality given the same wall-clock training budgets.

3. Inference Efficiency Using a Rolling KV-Cache #

Think of the KV-cache as the model's short-term memory—a sliding window that remembers the last few frames. As new frames arrive, the oldest memories are forgotten, keeping memory usage constant while generating unlimited video. This "conveyor belt" approach achieves O(TL) complexity instead of recomputing everything for each frame.

Note: T being the number of frames to generate and L being the size of the KV-cache

4. Stabilization By Incorporating V-Sink Into Training #

During training treat the initially generated sink_size frames as a "V-sink". This provides a larger memory context for the model, enabling the model to mitigate drift and improve overall prompt coherence during generation. This key insight was covered in the Infinite Forcing repository. For my demo application, I've used the model weights and KV-cache implementation from this project in my demo. Special thanks to the insights given in this project.

Conclusion: The Future is Live #

The combination of Self Forcing, Rolling KV-cache and V-sink stabilization transforms the WAN 2.1 from a batch-processing model into a real-time video streaming system. This isn't just a technical achievement—it's a paradigm shift that makes interactive AI video generation practical.

What's Next?

• Try the demo application yourself
• Quality and performance improvements are ongoing
• The techniques shown here could apply to other video models
• Adding additional prompting mechanisms such as explicit camera movement.
• Adding image to video generation in addition to text to video.

The gap between typing a prompt and seeing results is shrinking from minutes to milliseconds. That changes everything.

Additional Reading and References #

• Self Forcing
• Infinite Forcing
• Self Forcing: Endless
• Tiny AutoEncoder for Hunyuan Video
• Rolling Forcing

• Previous: Still Talking?