What is nava?

NAVA is an AI model that generates synchronized video and audio together from a text description. Rather than generating video and audio separately and then aligning them after the fact, NAVA produces both at the same time through a shared process that keeps them in step from the start. The research behind it was published by the Ernie Research team. The model can take a text prompt describing a scene and produce a video clip with matching sounds, speech, and background audio all generated together. It also supports starting from an image instead of just text, in which case it animates that image while producing the audio. A notable feature is voice control: you can supply a short audio clip of a specific person's voice, and the model will use that person's voice identity for any speech in the generated video. The model handles multiple speakers in the same clip by binding each reference voice to the correct speech segments. At 6.3 billion parameters, the model generates 720p video in roughly one minute when running across eight high-end GPUs. The README includes configuration options for machines with less video memory, trading generation speed for lower hardware requirements. The baseline setup needs about 80 GB of GPU memory, while an offloaded configuration can work with around 42 GB at the cost of slower generation. Running the model requires downloading the weights from Hugging Face, installing Python dependencies including PyTorch and Flash-Attention, and then running one of the provided shell scripts for the task you want. The README recommends putting prompts through a rewriting step first, because the model was trained on a particular style of detailed cinematic descriptions and performs better when inputs match that format. The full training code is included alongside the inference scripts. The project is released under the Apache 2.0 license.