Tavus, a four-year-old generative AI startup that helps companies create digital “replicas” of individuals for automated personalized video campaigns, has confirmed a fresh $18 million in funding and revealed that it’s opening its platform for third parties to integrate their software with the company’s technology.
Reports emerged back in August that Tavus had raised “about $18 million,” but details were scant. The company has now confirmed to TechCrunch that it has indeed raised $18 million in a Series A round led by Scale Venture Partners — an early-stage VC that has previously backed the likes of Box, HubSpot, and DocuSign. Other notable investors include Sequoia, which led Tavus’ $6.1 million seed round last year, which participated alongside Y Combinator (YC) and HubSpot.
Tavus taps generative AI to power personalized videos with voice and face cloning
Video takes center stage
The generative AI movement is best exemplified by text-based search engines like ChatGPT and text-to-image models such as DALL-E, which OpenAI is in the midst of combining into a single all-singing platform. But if the past few months have been anything to go by, generative AI could be on the cusp of another minor revolution, with video taking center stage.
OpenAI recently debuted Sora, a text-to-video model that could transform the creative industry as we know it. But it’s far from the only player in town, with tech giants such as Google working on similar tooling for several years, not to mention a slew of startups that have raised sizable chunks of VC change over the past year for various realizations of how generative AI might intersect with video.
Tavus, for its part, works with its clients to create replicas of individuals through voice and face cloning. The idea is that sales and marketing teams can use Tavus to send personalized videos to prospects at scale, or maybe a product team can create individualized walkthrough videos for onboarding new customers — all via simple text-based prompts that leverage the previously created digital replica. And by integrating Tavus with third-party systems such as Salesforce or Mailchimp, companies can automate much of this — for instance, a customer who completes an online form requesting further information on a product can be emailed a video instantly, with a sales rep addressing the prospect by name and explaining the next steps.
Tavus has managed to secure some fairly big-name customers in its short life so far, including Salesforce and Facebook’s parent Meta, which co-founder and CEO Hassaan Raza said are using the platform to upsell to their respective B2B customers through personalized demo videos.
Tavus as a platform
So far, Tavus has been served via a SaaS app, through which customers create their own AI video templates. The onboarding process requires an individual, such as the CEO or sales executive, to record a 15-minute video based on a script provided by Tavus.
This is then used to train the AI, after which the user goes to a web editor and selects which parts of the video they wish to personalize by defining the variables — such as location, executive name, company, or product. By tying Tavus into their CRM system, companies can tweak each of these variables to suit a particular customer segment, such as those who have expressed an interest in a particular product.
Companies can create hundreds of these replicas with different personnel involved, replete with different backgrounds for different target markets.
Through the in-app editor, it’s possible to generate any number of different scripts to attach to each use case — without having to re-record any of the original video.
While this core SaaS product isn’t going away, Tavus is today lifting the lid on a new turbo-charged version of its technology alongside the first installment of a suite of developer APIs that allow third parties to integrate Tavus into their own applications.
Replicate
The first facet of Tavus’ new developer platform to arrive is its “replica API,” which is all about creating “photo-realistic” digital replicas replete with text-to-video generation. With this, a company can replicate a person (e.g., head of marketing or CEO) using a new proprietary model created by Tavus dubbed “Phoenix,” which is based on a deep learning method called neural radiance field (NeRF). This can generate a 3D construct of a person from 2D images in just a couple of minutes.
“It essentially allows you to create entire videos with just two minutes of training data, which is a big leap forward from how we were previously doing the personalization at scale,” Raza told TechCrunch. “And so now all you have to do is record two minutes of training data, and it’ll create a full replica of you. And once you have replica, you can make as many videos as you want — from one, two, or a thousand scripts.”
The inaugural replica API leans on the entire functionality of the Phoenix model and captures an individual’s facial motion, including cheeks, nose, eyebrows, and lips.
“Moving your entire face drives realism, naturalness and quality — when you talk, your face expresses emotion beyond your lips moving,” Raza explained. “If you want to generate an entire video from a script — where you are speaking, one that looks natural and is incredibly high quality — you’d want to use the replica API.”
However, Tavus is also developing a number of additional APIs, including one specifically for lip-syncing, one for dubbing, and one for running mass, personalized video campaigns.
The lip-sync API will have a “lower entry cost,” according to Raza, and is better for situations where a “high degree of quality and realism is not necessary.”
The dubbing API, meanwhile, also uses the lip-sync model but includes multilanguage voice cloning, too, meaning a monolinguistic user can send out video campaigns in any number of languages using their own voice. In this instance, given that most of the video will remain the same, the API enables simple replacement of lip movements to align with the different sounds coming from the user’s mouth. This could prove useful for the creators of a video-editing software suite, for example, where they wish to enable their users to add lip-syncing, editing, and dubbing to their videos.
And then the video campaign API basically bundles the replica API alongside a swathe of additional tooling — such as hosting, variable mapping, thumbnails, and analytics — for those looking to launch large-scale video campaigns.
“We’re bringing the ability for any developer to provide an end-to-end video campaign experience out of the box, within their own solutions,” Raza said. “Whereas the replica and lip-sync APIs are more ‘model-as-a-service,’ the campaign API gives you tools to build an AI video campaign platform easily.”
Raza remained coy on who some of the early users of the Tavus platform are, but he did say that it’s “working with one of the largest video platforms” for customer engagement. “They’re looking to bring this to their millions of customers that are already using their platform to create video on a daily basis,” Raza said.
Deepfake dilemma
Instinctively, platforms such as Tavus are ripe for misuse — after all, what’s stopping anyone from uploading a preexisting video to create a digital replica? Deepfakes are indeed a growing concern in the burgeoning AI movement, but Raza says they have checks in place to avert chicanery. For instance, when a user submits their two minutes of training footage, they also have to submit a specific verbal consent statement, which is then aligned to the audio in the training footage to ensure there is a match.
“We run these checks automatically, and then do a human check for every replica that makes it through the automated checks to ensure safety,” Raza said.
It’s easy to see how that might work with Tavus as a stand-alone SaaS app, but now that it’s a platform accessed by any number of companies via an API, who is in control of verification then? Well, as it turns out, Tavus is — the company wants to keep its hands on the verification wheel, even when it’s merely providing the engine for third-party developers.
“We run the same checks, and assume responsibility for verifications with [the] API as well,” Raza continued.
Extending reality
While OpenAI has become almost the public face of generative AI, there is more than enough room for different players bringing something different to the mix. Indeed, while DALL-E and OpenAI’s recently released Sora model are mostly about helping people create visuals from text prompts, Raza says Tavus’ raison d’être is more about “extending” a person’s own reality.
“We see a future where everyone wants to have a digital replica of themselves; they control that and they have full authority over that,” Raza said. “And it’s gonna be important that it actually ends up capturing more and more of your personality, more and more of your gestures and traits. That’s how we see things going forward — there will be the models that create things that don’t exist, and then there’ll be the models that extend your reality.”
With $18 million in the bank, Raza said that the recent cash injection will be used to “fuel the fire that’s already burning” at Tavus towers.
“We’re an AI research company, so we want to be able to continue development on newer models like Phoenix,” Raza said. “But then also just sustain our growth, we’ve had a ton of demand continuously. And we want to be able to continuously hire on our machine learning and engineering teams to support our developer and SaaS customers.”
Comment