What exactly is lost when TV shows and movies are subjugated or copied to a new language? It’s hard to answer, but for the Flawless team launching artificial intelligence, it may be one we don’t have to think about in the future. The company claims to have a solution to this particular language barrier; a technological innovation that can help TV shows and movies easily enter new markets around the world: deepfake dubs.
We often think of deep counterfeiting to manipulate the whole image of a person or scene, but the Flawless technique focuses on only one element: the mouth. Customers feed the company’s software with videos from a movie or TV show, as well as copied dialogue recorded by people. The flawless machine learning model then creates new lip movements that fit the translated speech and automatically attaches them to the actor’s head.
“When someone looks at this copied material, they are not shaken by a brazen word or the wrong mouth movement from the performance,” says Flawless founder Nick Lynes. Limit. “It’s about maintaining performance and maintaining the original style.”
The results – despite the company name – are not 100% flawless, but they are pretty good. You can see and hear what they look like in the demo roll below with the French dub of the classic 1992 legal drama A few good men, starring Jack Nicholson and Tom Cruise. We asked the native French speaker what they did with the picture, and they said it was off in a few places, but still much softer than traditional copying.
What makes Flawless technology particularly interesting is its potential for scale. The flawless tone is that deepfake dubs offer tremendous value for money: they are cheap and quick to create, especially when compared to the cost of complete rebuilds. And with global streaming platforms like Netflix, Disney Plus and Amazon Prime Video, accessing such copied content to international markets is easier than ever.
In a recent report in 2004 Wall Street Journal highlighted, demand for streaming services in the United States is saturated and companies are now looking for growth from abroad. For example, in the first quarter of 2021 89 percent of new Netflix users came from outside the United States and Canada, while the service’s most watched program, Lupine, is a Parisian thriller.
“You’re seeing more and more streaming services coming online, realizing that over time, most consumers will be coming outside the U.S.,” said Erik Barmack, the company’s former Netflix director of international production. WSJ. “The question is how your international content needs to be to succeed.”
As Barmack suggests, this demand can be met in different ways. You can create local flavors that still entertain domestic viewers. You can re-create local hits for new audiences. And you can distribute sub and dub songs. But Flawless bets that its technology offers a new option that is particularly appealing to filmmakers.
This is because the company’s deep-fake copies retain some of the original actor’s performance, Lynes says. Flawless technology is based on research by the Max Planck Institute for Information Technology was first published in 2019. As you can see from the showcase video below, the dub songs it produces are somewhat sensitive to the performers ’expressions, retaining their feelings and line-up.
Flawless has been developing these technologies over the past three years, Lynes says, by speeding up production time and reducing the amount of material fed. The results are still a balance with automatic copying and manual retouching (about 85-15 percent), but editing is fast. “If something comes up, we don’t particularly like doing a few repetitions; resubmit the exercise data in different formats and you’ll get another result,” Lynes says.
The company hopes that preserving the original performance will attract filmmakers who want to preserve the magic of their original actor. Lynes gives an example of a 2020 Oscar-winning Danish film Second floor, where Mads Mikkelsen belongs to a group of teachers who try a low level of alcoholism whether it will improve their lives. Following its success at home and at the International Awards, the film is set to be re-produced for an English-speaking audience starring Leonardo DiCaprio.
The news sparked debate the value of such renewals. Is the Danish beverage culture that forms the backbone of the film really so foreign to the American audience that a remake is required? Is Mikkelsen, an actor who has performed at such mainstream prices as Hannibal, Doctor Strangeand Rogue One, so unknown that he can not attract viewers to the United States? And whetherone inch barrier“Subtitles (quote Parasite director Bong Joon Ho) simply too much to win the audience?
From Lynes ’point of view, a deepfake dub would at least be a cheaper way to bring Second floor for English-speaking audiences while maintaining the original taste. “If we offer something that is two percent of the replacement cost, we only have to be half as attractive to offer 10 times better value,” he says.
Of course, those in charge of redirection have concerns other than money. No matter how loved Mikkelsen is, he is not as bank account as DiCaprio. But Lynes hopes that as deep-fake copies become more common, it will change the calculations for such reproductions in the future. Much more, he says, it can even shape the international film world, allowing Actors and Directors to reach new audiences with little effort.
“I think the appeal of players is changing globally as a result of this technology,” he says. “Different people’s performances and directors’ choices are better recognized because they can be seen by a wider audience.”
Maybe so, but right now, Flawless has to show that the audience really wants its technology. The company, which started earlier this month, says it already has its first contract with a customer who can’t be named, but there’s no timetable for when we can see its wares on a commercial TV show or movie, and that’s a real test. The proof is being copied.