Take a close look at any ambitious startup and you’ll find pugnacity nestled in its core. Stubbornness and a bullheaded belief in the worth of what a company wants to bring to fruition is often the biggest driver of its success, and the people at such companies also tend to share this quality.
So it wouldn’t be too far off the mark to say the people at Expensify are a stubborn lot — to the company’s ultimate benefit. This group of P2P pirates/hackers that set out to build an expense management app stuck to their gut, made their own rules. They asked questions few thought of, like: Why have lots of employees when you can find a way to get work done and reach impressive profitability with a few? Why work from an office in San Francisco when the internet lets you work from anywhere, even a sailboat in the Caribbean?
It makes sense in a way: If you’re a pirate, to hell with the rules, right? And even more so when nobody can explain the rules in the first place.
With that in mind, one could assume Expensify decided to ask itself: Why not build our own totally custom tech stack? Indeed, Expensify has made several tech decisions that were met with disbelief — from having an open-source frontend and cross-platform mobile development to hiring contractors to train its AI and recruiting open-source contributors — but its belief in its own choices has paid off over the years, and the company is ready to IPO any day now.
How much of a tech advantage Expensify enjoys owing to such choices is an open question, but one thing is clear: These choices are key to understanding Expensify and its roadmap. Let’s take a look.
Built on Bedrock
I think another question Expensify also decided to ask in its early days was something like: Why not have our database on top of a technology that’s built for small-scale application software?
It may sound incredible, but Expensify actually runs on a custom database built on top of SQLite. This is surprising, because despite being one of the most widely deployed database engines, SQLite is known for running on small, embedded systems like smartphones and web browsers, not powering enterprise-scale databases.
This custom database is called Bedrock, and its architecture is as unique as they come. Expensify explains it as an “RDBMS optimized for self-healing replication across relatively slow, relatively unreliable WAN (internet) connections, enabling extremely high availability/high performance multi-datacenter deployments without any single point of failure.” RDBMS means relational database management system, describing SQLite and other row-based databases where entries are interconnected with each other.
But why would Expensify build this instead of going for any number of widely available enterprise database solutions?
To answer that question, we need to go back to the early days of the company, which was originally a side project for its founder and CEO, David Barrett. His initial idea was to develop a prepaid card for the homeless, but this required putting a server on the Visa network, which brought several strict requirements and challenges. “I would say one of the most difficult [parts] was that I needed the ability to automatically replicate and failover,” Barrett told TechCrunch when we interviewed him a couple of months ago.
This was no easy feat in 2007, but Barrett was up for the challenge. “I just hit a moment where the technology available off the shelf just wasn’t that good. And I happened to be a peer-to-peer software developer who had tons of spare time and really wanted to build this thing to put on the Visa backend,” he said. The P2P aspect was important, as Barrett had the skills to make it work. His first hires for Expensify, P2P engineers he had worked with at Red Swoosh and Akamai, were also unusually suited for the job.
The initial idea blossomed into Bedrock. This relational database is best suited for a scenario in which it is distributed over three data centers, and redundant in each. In a 2017 episode of FLOSS Weekly, Barrett explained what happens when one of the servers fails: failover is automatic, and recovery is painless thanks in part to a distributed consensus algorithm that makes it easy to determine which server is the new leader and which are the followers.
This is also why Expensify describes Bedrock as being based on blockchain “before it was cool.” Some wondered if the claim was misleading, but “it depends on how strictly you want to interpret what blockchain means,” Expensify software engineer Tyler Karaszewski told TechCrunch. When talking about Bedrock, the company refers to its distributed general ledger and “Paxos-style algorithm,” which guarantee data integrity and constitute a private blockchain element.
And Expensify did all this when even SQLite’s creators didn’t think it was possible.
SQLite on steroids
Both parties now laugh about it, but when Barrett explained in 2012 how Expensify was using SQLite for its then 40GB database, SQLite’s primary author Richard Hipp tried to talk him out of it. “It sounds like you are pushing SQLite well beyond what it was intended to do. [ … ] SQLite was never designed or intended to replace an enterprise-level RDBMS. And I think you probably need an enterprise-level RDBMS at this point,” he wrote to Barrett via the SQLite mailing list.
But just because SQLite wasn’t designed for what Barrett intended didn’t mean the engine couldn’t do it; it just needed someone to try and make it happen. “It makes sense in hindsight, but we were so focused [on what we were doing] that we never stopped to think that it would be a good idea,” Hipp told TechCrunch.
He references a George Bernard Shaw quote when speaking about Barrett: “The reasonable man adapts himself to the world: The unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.” This clearly applies to Barrett, Hipp says. “We need people like David to move us forward.” The admiration is clearly mutual, as Barrett more than once called the SQLite team “some of the best engineers in the world.”
Expensify is also a long-time, top-tier sponsor of the SQLite Consortium, and ever since Barrett convinced them, Hipp and his team have been going out of their way to support the company. “Bedrock, as it is right now, wouldn’t have been possible without the contributions that the SQLite team made to their product so that we could use it in ours,” Karaszewski said.
In practical terms, the SQLite team added improvements designed for Bedrock and deployed on a separate branch because of the small overhead these add-ons brought. “1% extra CPU and memory might not sound like a lot, but if you multiply it across 5 billion cell phones and laptop computers, it can add up,” Hipp explained via email.
Expensify remains an outlier in the way it uses SQLite, but Hipp hopes others will follow. “It would be cool to see more web-facing apps use Bedrock or something architected like it, because the #1 vulnerability is to SQL injection attacks, and Bedrock makes it very unlikely. [ … ] We are amazed by what [Barrett] has accomplished.”
As odd as it seems, Expensify’s unique data architecture has so far withstood the test of time, open-source scrutiny and the company’s eventual scale. It has arguably also contributed to the company’s success.
“It’s crazy cool, and kind of amazing, frankly”
Expensify’s decision to open-source Bedrock in 2016 was singular, as was its motivation to do so. “The primary value of open-sourcing Bedrock wasn’t to get contributors, it was to hold ourselves to a higher standard and get external eyes on what we have,” Barrett told TechCrunch. External observers were sometimes harsh, but others understood why the company would expose itself to scrutiny: “As in all bold moves. I think that Expensify is taking the long view on this. When dust settles, they’ll be better off for it,” a commenter wrote on Twitter.
This is what seems to have happened, as the company received contributions, raised its own bar and kept on scaling. All in all, the process has reinforced Expensify’s trust in Bedrock, which Barrett sees as the company’s “most differentiated technology,” leading to “many of its key competitive advantages,” as First Round Review wrote in 2016.
Barrett is fully aware that MySQL is a stronger alternative than it was when he created Bedrock, which now comes with MySQL protocol support for users who might want to use Sequel Pro or Mode Analytics. Still, he believes that Bedrock is at an advantage because it has less baggage. “Bedrock is written for modern hardware with large SSD-backed RAID drives and generous RAM file caches, and thereby doesn’t mess with the zillion hacky tricks the other databases do to eke out high performance on largely obsolete hardware,” its website reads.
The hardware is definitely an important factor, as after some testing and calculations, Expensify figured out bare metal was the best option. “AWS must have fantastic marketing, because most of what people say about it is just objectively wrong,” Barrett told us. First, paying for hardware is less expensive than pre-paying for a server for a few years. “But secondarily,” he added, “the hardware you get through AWS is not the same as the hardware that you would buy yourself.” Without getting too technical, he meant that hyperthreading is not the same as additional cores, and added that Expensify would rather choose its own system settings for maximum performance.
Expensify does it this way because Bedrock is optimized to work on computers that are “just shy of supercomputer class computers,” according to Barrett. Bedrock is optimized for this not only for scale, but also because all of the company’s data sits in the same database — a little known fact that is often met with incredulity. As one Twitter user put it, “I knew Expensify used SQLite but I always assumed they were running thousands of tiny databases, one for each customer — I had no idea that [it] ran one giant 1TB of RAM monster database!”
Barrett says Bedrock really shines when aligned with Expensify’s business model. “By having basically a single conceptual giant database, that means we don’t have to partition our customers into different databases,” he explained. The latter is what an enterprise company with a top-down sales model would typically do with a multi-tenant architecture.
In contrast, Expensify is more interested in the viral aspect of letting its users send expense reports to their boss while also giving them the option to invite their friends and interact with other companies. “By having everyone swimming in the same pond, we can enable a more natural collaboration between any two users,” Barrett says.
How hard it would be to provide a similar user experience with a different architecture is up for debate, but one thing is for sure: Bedrock boosts Expensify’s allure among engineers. Jason Kruse, an ex-Expensify employee and CTO at Lithic (formerly Privacy.com), said, “It’s kind of crazy that they built their own database replication technology, but crazy cool, and kind of amazing, frankly.”
Kruse also praised the distributed general ledger and Bedrock’s resulting ability to verifiably log what happened first. Thanks to this, Bedrock can also act as a job management and distribution platform, and it uses these capabilities to power two initiatives: its customer support feature Concierge and “in-house Mechanical Turk” ExpensiWorks, both of which leverage AI.
The people behind the AI
Barrett is transparent about his take on artificial intelligence: “The reality of AI is that a lot of AI involves contractors.” This was at the center of a controversy in 2017, when people spotted receipts being shared by Expensify on Amazon Mechanical Turk. It turned out to be a mistake impacting “up to 208 receipts” during a beta test, and the company ultimately decided not to use Amazon’s platform. Now, it uses a human workforce to complement and train its AI.
That philosophy is at the core of SmartScan, its receipt-scanning product. It leverages AI to scan receipts and resolve errors, but escalates the hardest tasks to a human workforce. The company adopted this approach early on, when Barrett decided to stay away from only using optical character recognition (OCR) to scan receipts. “From the earliest days we’ve focused on total accuracy over just speed,” he says. His reasoning was that instant but less-than-perfect results were pointless, since they required checking and manually fixing the entries.
Barrett decided to provide a different user experience, which he describes as a “fire and forget” design. “It doesn’t ask you to confirm the results when you’re done. You just scan the receipt, put your phone away, and then everything else happens automatically,” he says.
Now, after the OCR process, human transcribers take over if required. These transcribers are vetted Expensify contractors and subcontractors under NDAs, not Turkers hired via Mechanical Turk. After the 2017 bad buzz around its test, Expensify decided to build an in-house platform to manage these tasks and workers: ExpensiWorks, which COO Anu Muralidharan was instrumental in creating.
“We didn’t really build anything from scratch, we just used everything that already existed,” she explained. “We built it in a way that it leveraged Bedrock, and also leveraged all of the existing expense infrastructure.”
A similar collaboration of humans and AI is also at the core of Concierge, Expensify’s customer support feature. Here, people do more than just step in. “We refer to our first responders as trainers, because their job is first and foremost to train the AI, and the byproduct of training the AI is also to support customers,” Barrett said. He describes the impact of Concierge as “completely transformative” for the company, and the reason why its revenue keeps going up when its headcount doesn’t.
Concierge’s natural language processing technology includes things like fuzzy matching, but you won’t find deep technical information about it. “Who has time to publish papers? No, we publish code and features,” Barrett says.
That statement reflects his take on real-world AI, and more generally, Expensify’s pragmatism when it comes to technology. “One of the strengths of Bedrock is that it’s not built in an academic environment: It’s not sort of a solution in seek of a problem; everything it does was built in response to real-world problems,” Barrett said in a 2017 interview.
His firm grounding in reality is also what explains how Expensify could support flatbed scanners at the same time as it was pioneering reporting expenses by using your phone camera.
Can’t ignore you if you’re everywhere
Being early to mobile was a boon for Expensify, but it wasn’t always easy. Early employee Tom Jacobs recalls, “With Android, iPhone OS, Blackberry and Palm, we had these four platforms that had completely different codebases; they all ran the same underlying business logic, but maintaining and keeping them updated was a problem. I used to upload a new version of the Blackberry app every week, and it was ridiculous.”
To solve the problem, Barrett suggested building an in-house cross-platform JavaScript framework: YAPL, which stands for “Yet Another Platform Layer” or “Yet Another Platform Library.” It was a pioneering step, as third-party alternatives like PhoneGap, Xamarin and Electron didn’t exist or were just getting started, and YAPL was going one step further by using a shared business logic on different platforms while still “taking advantage of native event handling and controls.”
Going cross-platform “was a hard call,” but it had benefits because it let Expensify stay relevant. Smartphones were not nearly as capable in 2012 as they are today, and mobile phones were different from cameraphones, because not all of them had usable cameras. Moreover, iPhones didn’t have auto-focus until 2009, so it was hard to predict which platform would matter the most for Expensify’s receipt-scanning feature, meaning that the company had better be present everywhere despite having limited staff.
“Being able to write code once and have it run across platforms is certainly the dream,” director of R&D Puneet Lath said in a recent interview. YAPL may have been a pioneer in that sense, but Expensify’s latest bet on React Native takes this further.
Looking at the big picture
Expensify’s newest project, Expensify.cash, a financial group chat service based on the premise that “chat and payments are the same thing”, is notable for many reasons, but what caught our eye is that it’s a complete rewrite of the Expensify frontend, built on a React Native platform using the same codebase across iOS, Android, web and desktop.
Several factors played a role in Expensify’s decision to go all in with React Native, but chief were the emergence of React Native for Web, and perhaps counterintuitively, Airbnb’s own decision to move off the framework. After looking into what happened there, Expensify’s team concluded that they shouldn’t follow their initial idea to slowly migrate to React Native since the result could be clunky. Instead, they decided to jump in head first.
Expensify.cash is going to be 100% open source, and it’s being built with an open-source community of contractors from Upwork. Anticipating likely questions about this outsourcing, the company explained the rationale:
The key to Expensify’s ability to remain so nimble at scale is keeping an extremely small core team of generalists who understand the ‘big picture’ — and then engaging a large ring of highly talented specialist contractors from around the world to execute that vision. [ … ] We believe there is an opportunity to execute that same playbook for our engineering team, pairing our core group of engineering generalists, with a global network of open-source engineering specialists.
Generalism is an important element of Expensify’s culture, and this has made it easier to adopt a plan involving React Native. Unlike at other companies, not many at Expensify are likely to feel threatened by Barrett’s argument that “the existence of React Native for Web obliterates the traditional hierarchy around engineering” and the specializations typically attached to it.
Another facet of Expensify.cash that could have scared other companies is that its frontend will be open source. But rather than worrying about proprietary code, the company sees benefits such as the ability to let customers customize the frontend. This could compensate for Expensify’s own lack of customization services for customers, which arises from its “compounding product management” policy.
“Compounding” has been a recurring concept at Expensify. “We have this idea of [a] compounding engineering philosophy, where essentially everything we do is about improving everything we’ve already done while laying a foundation for what comes next,” Barrett said. “And so I think that with this, every decision traces a very long lineage back to initial technology and legal decisions.”
That sentence has a ring of truth to it. In Expensify’s narrative, Barrett & Co’s initial decisions form the throughline that connects their early ideas to how its employees function today. It shows in how they approach problems and ideas, how they look at opportunities, and how they collaborate with the open-source community on the technology that makes it all happen, from Bedrock to Expensify.cash.
As we’ll find out in Part 5 of this EC-1, this long-term outlook sometimes put Expensify at odds with venture capitalists and brought a plethora of challenges, but the team’s pugnacity and belief in its business model eventually won through.
Expensify EC-1 Table of Contents
This EC-1 was serialized starting early May and continuing through early June.
- Introduction
- Part 1: Origin story
- Part 2: Culture
- Part 3: Expansion and remote work
- Part 4: Engineering and technology
- Part 5: Business model
Also check out other EC-1s on Extra Crunch.
Comment