AI Video Dubbing

Tricks to Get Kids to Eat Healthier at Home

Tools Used: Speechify Studio, Camtasia, HandBrake Video Transcoder

Language: Mandarin Chinese

I undertook this project to prepare for a role that involved using AI speech tools to adapt content into multiple languages. I’d seen this particular video used in a nutrition course for Mandarin speakers and felt subtitles alone weren’t sufficient to engage viewers, especially given the text-heavy visual style. I thought that adding dubbed audio would make it more accessible for Mandarin speakers.

Challenges and Lessons Learned

Selecting an AI Tool

I explored several options for generating Mandarin speech. Surprisingly, some tools advertising Mandarin support actually lacked dedicated Mandarin voices. Instead, they used English voices to generate Mandarin with a heavy accent. 

I settled on Speechify, which offered about a dozen Mandarin voices and several regional dialects. The platform is well-known for quickly converting text to speech, allowing users to listen to books, documents, and web content. Speechify Studio is an extension that enables users to integrate AI speech into video projects.

Benefits and Shortcomings of AI Speech Tools

Speechify has its limitations, but so does collaborating with voice actors. For some projects, the advantages of working with experienced voice talent would outweigh the speed and convenience offered by AI tools. For this project, however, Speechify proved to be a quick way to improve accessibility

• A clear advantage of any AI speech tool is the ability to change and re-record content, a process that can otherwise be time-consuming.

• Adjusting tone is not currently possible for Mandarin in Speechify, but can be accomplished with a combination of pauses and speed adjustments. This sufficed for this particular video, which maintains a relatively consistent tone.

• Speechify is limited in its ability to adjust audio to fit within specific timeframes. This would be a dealbreaker for dubbing over an on-screen speaker, but is an acceptable limitation for voiceovers in which the speaker is not visible. I ultimately exported the video to finish edits in Camtasia.

Results and Conclusions

Using the dubbed video, I found that learners were able to restate key techniques more completely than those who had watched the subtitled version. Still, some learners were unable to explain how to implement these techniques at home, perhaps due to the video's lack of demonstration. With this in mind, I believe the content could be better presented in a different style, including more demonstrations and fewer screenshots of research studies.

While I'm interested in further exploring how AI speech tools can improve localization and accessibility, this project serves as a reminder that the process begins with quality content. Technology can help with accessibility, but can't replace the need for well-designed instructional materials.