Lost in translation no more: ElevenLabs takes AI dubbing to the next level

AI company ElevenLabs has released Dubbing v2, a new dubbing model that transfers a speaker’s emotional delivery and tone into more than 90 languages. Imogen Mulliner and Jakub Lichman write for ElevenLabs that the system conditions directly on the original audio performance rather than relying on a transcript alone.

Traditional AI dubbing tools work primarily from text transcripts. This approach tends to strip out the subtle qualities of human speech, including emotion, timing, and emphasis. Dubbing v2 aims to solve this by capturing the speaker’s intonation and pacing and carrying those qualities into the translated output.

Sync, scale, and accessibility

The model includes a synchronization feature that automatically aligns the start, stop, and pacing of translated speech with the original content. According to ElevenLabs, this reduces the need for manual editing after dubbing.

The company positions the tool as a lower-cost alternative to professional dubbing, which it says can cost hundreds of dollars per minute and typically involves translators, voice actors, and audio engineers.

Who can use it

• Content creators can localize YouTube videos and other content through ElevenLabs’ ElevenCreative platform. A Creator Dubbing Partner Program with discounted access is also launching alongside the model.
• Marketing teams can use it to adapt campaigns for international markets while maintaining the original tone and delivery style.
• Studios and broadcasters can access Dubbing v2 through ElevenProductions, which combines the AI model with human translators, voice casting, and professional audio mixing.

Dubbing v2 is available now in ElevenCreative and ElevenProductions. For a limited introductory period, ElevenLabs is offering free usage depending on subscription tier. API access is described as coming soon.