Karin Tsai joined Duolingo in 2012 and is now the director of engineering. Yesterday, at TechCrunch’s City Spotlight: Pittsburgh, she spoke on the company’s extensive development process and unique culture.
This interview was pre-recorded and aired a day after the company filed its SEC Form S-1 ahead of its initial public offering. As per SEC regulations, the company is now in a quiet period and it’s fair to say that Tsai gave the company’s last interview for quite a while.
This interview is the latest in TechCrunch’s deep dive into Pittsburgh’s Duolingo. Last month, we published a four-part series on the company that includes Duolingo’s origin story, it’s product-led growth strategy, monetization plan, and how develops and implements new initiatives and features.
And it all starts with an A/B test
Tsai works with the some 170 engineers at Duolingo, which make up the vast majority of the company. That composition has led to A/B testing turning into a key part of Duolingo’s strategy, as it works to create an app that has effective yet inclusive gamification. At any given time, there might be 100 different A/B tests running, which caused Duolingo’s analytics team to even build a dashboard to track impact of experiments. Tsai walked us through an early realization of the common test below:
[A/B testing] has been such a key lever in our success as a company. I think one of the things that was important for us was to keep an open mind to tests that fail. So I think one of the temptations we have is to come with a preconceived notion of what should win when we run the test. If it doesn’t win, we just assume [that] there was some bug or some error, and not really pause to think why did that experiment lose. So, I would recommend for any startups looking to really integrate A/B testing into the fabric of their decision making: keep an open mind on what you might be missing about what your users are telling you through the data. Also, always be open to improving what you actually measure. At the beginning, we just kind of flatly measured learner retention [and] how many daily active users we had. But after a while, we decided to be a little more fine-grained about what we measured. So we measure new user retention versus existing user retention to sort of see the difference between how our new users are perceiving Duolingo, etc. And over time, you add more and more of these metrics, you could get a fuller picture of what what your changes actually doing. (Timestamp: 2:53)
One quirk that came up during the Duolingo EC-1 is that Duolingo’s product team usually A/B tests new features in the French course for English speakers. It’s not because of any secret preferences, but because the course is so popular that it makes more statistical sense to test it out in this course before rolling it out to other languages. In contrast, if a team tested a feature in the Hindi course for English speakers, it would take a longer time to get useful results.
An A/B experiment that stands out in Tsai’s mind has to do with the subscription page, and saying no to increasing the bottom line.
On our subscription page, we offer a free trial of the subscription. Before, the button used to say ‘start your seven day free trial.’ We changed it to ‘start using Duolingo for free for a week’ just to play around with copy. And we didn’t really expect too much from that, we just wanted to shorten the text.
It turns out that this ended up making us a lot more revenue per day. However, learner retention took a hit, and especially new user retention. Our hypothesis was that actually changing from ‘start my free trial’ to ‘start my free week’ made people think that they had to pay to use Duolingo. So, a lot more people actually subscribed because they thought they needed to pay. But we lost some learners, because they thought the experience was just overall not free. So we actually reverted that change, even though it made us some money. We learned a lot about how precise copy really needs to be to convey meaning. So, it was a really great learning to find that actually being more transparent with how you monetize, actually led to more modern successful monetization. (Timestamp: 5:47)
- The product-led growth behind edtech’s most downloaded app
- The Morality Of A/B Testing
- Apple to introduce A/B testing and in-app events to the App Store
On prioritizing monetization
Duolingo notoriously launched with a bold promise: no advertisements, subscriptions or in-app purchases — approaches that now all exist on the platform. While the company appears to be doing just fine financially now, the choice to start charging was not easy, and felt directly in conflict with Duolingo’s mission to provide free education.
Tsai, who watched the company grow from 15 people to 400 plus, explained how she personally felt the company resolved that tension and ultimately made the move to charge some of its users, about 5%, money.
I think the way that we approach this is maybe idealistic. I can just speak for myself: oftentimes, it’s tempting for companies to either turn evil or feel like they’re turning evil once they start to monetize, right? But the reality is, hosting costs, salaries, etc, do cost a lot of money. If you want to continue hiring top talent, you need to make revenue and be able to support that. And the way that we approach it at Duolingo is that we know we can’t actually accomplish our mission without also being a sustainable company. If Duolingo continued being on VC funding the whole time, we would not be sustainable in the long term. If we do buy in so much, and love our products so much that we do want it to last hundreds of years. And to do that, we need to make sure we’re monetizing in a sustainable way. So I think that the way that we resolve tension in the company is that no matter what area you are working in, everyone has a common purpose to that end.
There’s a lot of benefit of the doubt that you kind of get for free by having a strong culture. (Timestamp: 8:21)
- Duolingo’s S-1 depicts heady growth, monetization, new focus on English certification
- How Duolingo became fluent in monetization
How debate can live harmoniously within a startup
The strong culture that Tsai is talking about was re-invented during the pandemic, when Duolingo along with many tech companies turned to distributed work.I wondered if remote work can make debate and disagreement, on something as small as an A/B test or as big as a monetization opportunity, difficult. Tsai mentioned a weekly e-mail from co-founder Luis von Ahn that made all the difference, as well as some transparency on failure.
Luis our CEO sent started sending us a weekly company update every Monday morning. One of the first updates he gave was just a reminder to have empathy for your fellow what we call Duo’s, or fellow Duolingo employees. He sent those reminders often. I think this culture of empathy and constant reminders says that in a remote world, it’s harder to read social cues, body cues and it’s easier to bring more to work than maybe you normally would. I think during those times where there was maybe debate, we always were just reminded to have empathy, to realize that there’s a lot going on in everyone’s lives at the moment, and that what is being communicated may not be the entire picture of what someone’s going through. (Timestamp: 11:45)
Another aspect that actually helps a lot is that we send out every single experiment result that gets launched or killed. So the entire company continues to know how we make decisions and which way they go. So by the time someone’s running their own experiments and making these decisions, they would have seen hundreds of examples of calls we made in the past. And we tend to emphasize the more controversial ones as well, so that the entire company could be in line with how we make decisions. (Timestamp: 13:11)
We ended up a conversation about opinions. From the jump, Duolingo has been an opinionated company – from where its based to how much it raised to how it wants to monetize (and differentiate from competitors). But that is easier said than done, especially in a polarized world.
I think it’s important to be strong about what you believe in. I think it’s not necessarily a bad thing to be opinionated, but I think it is important to also be open minded that not everyone will agree to be respectful of those positions. As a company, [your opinions] really needs to be [clear] from the beginning because it is always harder to add on. Suddenly, sudden opinions are in contrast with what a company has demonstrated through its actions. So I think avoiding hypocrisy is the most important thing when you do have strong opinions, and Duolingo has strong opinions about many things. I think [our opinions] are all consistent with how the company has operated from the beginning. And that has really helped us. And we tend to attract talent that believes in those missions. We’ve encoded those operating principles and actually written them down. And it is okay and reasonable for someone to disagree with this approach, but this is at least how we’ve decided to do it at Duolingo. And that makes it clear to people that we’ve intentionally made the decisions to go one way or the other on these things. So that has really helped, I think, unify everyone and understanding how that company operates. (Timestamp: 19:46)
Comment