DeepL Voice: Real-time voice translations for global collaboration

hey! I am building a translation app for a side project. Is there any chance you could share how you're building the voice-to-voice?

My strategy is to separate everything STT, translation, and TTS, instead of building 1 model (that constantly needs to be trained).

But the problem I am running into is there arent' any great STT or TTS models. Either they only support the top 10 languages, they are huge (whisper-v3-turbo), or non-commercial license (fb's tts/mms models).

Are you training your own models? Just targeting what languages you need? planning on running in the cloud?

My email is in my profile bio if you want to email/chat!

How well are you able to handle live speaker diarization? I've been tinkering with building similar solutions, but unless you have previously labeled speakers things tend to go haywire once you have multiple speakers + crosstalk.

Exactly. I just want to talk to my relatives in real time. How do I sign up? No idea.

The name is a bit misleading I think as it makes it sound like this is voicegen whereas it's actually just transcriptions with translations. The videos show two versions:

- real time transcription and translation in virtual meetings in a personal chat window

- real time transcription and translations for in-person conversations with side-by-side or face-to-face (orientation flipped across the middle of the screen) display of both languages on a tablet

Given what I've seen in transcriptions from MS Teams, I'm not sure I'd trust this.

Yes, where is the trial?

They do both show demos.

What's your use case?

I absolutely thought that is was dubbing after watching the video and not paying close attention. This seems like 2010s tech and my expectations in 2024 are so much higher.

Actually, the word “voice” makes one think of mouth and throat associations, so “DeepL Voice” makes a lot of Americans think dirty thoughts for a second as they try to make sense of what the L stands for.

I am sure the branding people have considered this?

I hate to have to agree. Also, I signed up for a Pro account but then found out that using the API is an entirely different service that seemed almost completely disconnected from the main one with separate billing. It was so confusing and felt like it was built by a company the size of SAP or something.

Financing. Germany doesn't have financiers with a "we'll figure out how to monetize it later" mindset. B2B is seen as safer that way than B2C.

Would it have mattered in this case? The advent of LLMs would have endangered their position just the same, no?

There's no VC money in Europe, you have to be profitable early on

Their translation is still substantially better than what most LLMs can offer. Certainly for languages where LLMs have very little training data for.

As an individual, I still go to DeepL while actively using LLMs. I think the main reason is that the translations are very good, with alternatives in one click, without having to type "translate xxxx into yyyy".

My previous company ran tests on translations, for their specific use case, DeepL API was overall better that OpenAI or Claude.

I created a web site that uses AI to translate books, and in my testing, Deepl was way better than other AIs I tried at translating.

It is still better than Google Translate like it always has been, almost as if Google did not care about their translator, especially "post-ChatGPT" it is not supposed to be this inaccurate.

I think they will just go the AI/LLM/ML route, too.

If OpenAI wrappers can flourish I'm sure DeepL can too. My guess is that they will have sales team that will reach out to customers and provide them with solutions to their specific needs and use cases.

You know how ChatGPT was supposed to be able to act like a translator in voice mode, they even had an amazing demo at the keynote? I tried to demonstrate it to my father who was supposed to speak in Bulgarian and I in Turkish and see how cool the feature looks like. It didn't work remotely as advertised, it instantly screwed up and we both knew it because we both speak these languages and proving that its not to be trusted. The tech behind it is amazing but the UX still needs to be crafted to be valuable for more than tech demos, so if DeepL has the tech and know the market they will be in much more advantageous position than OpenAI(has the tech, doesn't know the market) and others who might know the market but use OpenAI(their margins will be thinner if OpenAI can't reduce costs dramatically more than DeepL).

I agree, LLM translations are not only more convenient but also much more capable. I often find myself giving instructions on how to translate text, such as asking the LLM to use formal language in the target language or to apply specific gender-neutral wording. Additionally, it can translate text while preserving the structure (e.g. values in a JSON object) or even adapt to a new target structure. It's just so much more convenient.

They have a next-gen model that is an LLM. But it's only available for a few languages and only for paying users: https://support.deepl.com/hc/en-us/articles/14241705319580-N...

I’ve been wondering the same thing. As near as I can tell from the explanations on the website and in the videos, this DeepL Voice does not seem to be based on a multimodal large language model. Rather, it’s probably using text-to-speech and speech-to-text models linked through a translation engine. If that’s true, it means it can’t “hear” the tone of voice of the speakers. More important, it doesn’t seem to be promptable—that is, it cannot be told what the situation is, who the speakers are, or the type of translation desired. Those are all possible with OpenAI’s Advanced Voice mode, and they can make the difference between successful and unsuccessful interpretation.

That said, DeepL is focused on providing translation services, which the large LLM companies are not. Even if DeepL’s translation engines are not as powerful as the strongest commercial LLMs, they might be able to compete in other ways, such as security guarantees, on-device operation, and training and support.

Shouldn't it use much less compute than an LLM? Or is that not a significant percentage of the total cost?